2024-09-16T03:55:13.682 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:55:13.700 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:55:13.791 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_mon_workunits} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905963' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) mon_bind_msgr2: false sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECk5igt4xZa57MDBWGdZIJdDOVMPCSQbm20/OCYsqcqs24sJu0e5gOFYWZSzJrwZJ89yX4/CSm/Z0XxPbne6cM= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZqPdnqeT/Y0qONxHvb/cZtk+OnW4aylbI8kkqGwfu7bIQNPQnbr4BzHmzoW+YMHlaA/CA1xNmqOS7he87L6hI= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:55:13.791 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T03:55:13.792 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T03:55:13.792 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:55:13.794 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:55:13.816 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T03:55:13.816 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:55:13.816 INFO:teuthology.packaging:ref: None 2024-09-16T03:55:13.816 INFO:teuthology.packaging:tag: None 2024-09-16T03:55:13.816 INFO:teuthology.packaging:branch: squid 2024-09-16T03:55:13.816 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:55:13.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T03:55:14.037 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T03:55:14.039 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:55:14.047 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:55:14.048 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:55:14.093 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:55:14.107 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:55:14.128 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:55:14.154 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:50:02.460841', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECk5igt4xZa57MDBWGdZIJdDOVMPCSQbm20/OCYsqcqs24sJu0e5gOFYWZSzJrwZJ89yX4/CSm/Z0XxPbne6cM='} 2024-09-16T03:55:14.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:50:02.462273', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZqPdnqeT/Y0qONxHvb/cZtk+OnW4aylbI8kkqGwfu7bIQNPQnbr4BzHmzoW+YMHlaA/CA1xNmqOS7he87L6hI='} 2024-09-16T03:55:14.178 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:55:14.188 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T03:55:14.188 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T03:55:14.189 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:55:14.284 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdd779b3ac0>, signals=[15]) 2024-09-16T03:55:14.284 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:55:14.295 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:55:14.295 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T03:55:14.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:55:14.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T03:55:14.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:55:14.462 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:55:14.474 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-09-16T03:55:14.493 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-09-16T03:55:14.494 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="9" 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="9" 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:55:14.552 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:55:14.553 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:55:14.554 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-09-16T03:55:14.567 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-09-16T03:55:14.586 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-09-16T03:55:14.586 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-09-16T03:55:14.644 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="9" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="9" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:55:14.645 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:55:14.646 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:55:14.647 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:55:14.647 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-09-16T03:55:14.667 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:55:14.686 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:55:14.719 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:55:14.719 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:55:14.722 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:55:14.740 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:55:14.757 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:55:14.757 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:55:14.779 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:55:15.110 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:55:15.173 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:55:15.191 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:55:15.191 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:55:15.192 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:55:15.192 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:55:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:55:15.213 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-09-16T03:55:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:55:15.214 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-09-16T03:55:15.272 INFO:teuthology.orchestra.run.smithi171.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:55:15.273 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:55:15.273 INFO:teuthology.task.kernel:Running kernel on smithi171: 5.14.0-503.el9.x86_64 2024-09-16T03:55:15.273 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum install -y kernel 2024-09-16T03:55:15.273 INFO:teuthology.task.kernel:Running kernel on smithi148: 5.14.0-503.el9.x86_64 2024-09-16T03:55:15.273 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2024-09-16T03:55:17.657 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T03:55:18.104 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.2 MB 00:01 2024-09-16T03:55:19.516 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-16T03:55:20.030 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-16T03:55:24.239 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-16T03:55:25.731 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-16T03:55:25.925 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-16T03:55:26.810 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-16T03:55:27.309 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 100 kB/s | 19 kB 00:00 2024-09-16T03:55:28.556 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T03:55:32.191 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T03:55:34.013 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-16T03:55:34.181 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:55:34.181 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:55:34.276 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T03:55:34.286 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T03:55:34.286 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-09-16T03:55:34.286 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T03:55:34.286 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T03:55:34.286 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:55:34.286 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:Install 4 Packages 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 79 M 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 126 M 2024-09-16T03:55:34.287 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T03:55:34.451 INFO:teuthology.orchestra.run.smithi148.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 192 kB/s | 23 kB 00:00 2024-09-16T03:55:35.009 INFO:teuthology.orchestra.run.smithi148.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-16T03:55:35.501 INFO:teuthology.orchestra.run.smithi148.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-16T03:55:36.034 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:55:36.034 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:55:36.121 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T03:55:36.127 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T03:55:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T03:55:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:Install 4 Packages 2024-09-16T03:55:36.128 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T03:55:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 79 M 2024-09-16T03:55:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 126 M 2024-09-16T03:55:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T03:55:36.168 INFO:teuthology.orchestra.run.smithi148.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 17 MB/s | 29 MB 00:01 2024-09-16T03:55:36.168 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:55:36.168 INFO:teuthology.orchestra.run.smithi148.stdout:Total 42 MB/s | 79 MB 00:01 2024-09-16T03:55:36.545 INFO:teuthology.orchestra.run.smithi171.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 78 kB/s | 23 kB 00:00 2024-09-16T03:55:36.851 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T03:55:37.189 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T03:55:37.190 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T03:55:38.159 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T03:55:38.159 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T03:55:40.618 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T03:55:41.274 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:55:41.359 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:55:44.135 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:55:44.269 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:55:49.081 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:55:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:55:53.177 INFO:teuthology.orchestra.run.smithi171.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 947 kB/s | 16 MB 00:16 2024-09-16T03:55:54.029 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:10.517 INFO:teuthology.orchestra.run.smithi171.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 861 kB/s | 29 MB 00:33 2024-09-16T03:56:13.774 INFO:teuthology.orchestra.run.smithi171.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 943 kB/s | 35 MB 00:37 2024-09-16T03:56:13.775 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:56:13.775 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.1 MB/s | 79 MB 00:37 2024-09-16T03:56:14.452 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T03:56:14.826 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T03:56:14.827 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T03:56:15.791 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T03:56:15.791 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T03:56:18.228 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T03:56:18.858 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:56:18.952 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:56:21.760 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:56:21.885 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:56:26.709 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:56:26.794 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:31.631 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:32.192 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:32.232 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:32.580 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:32.580 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:56:32.580 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:56:32.580 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:56:33.433 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:56:33.433 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T03:56:33.433 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T03:56:33.434 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:56:33.434 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:56:33.434 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:56:33.434 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:56:33.434 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T03:56:33.434 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T03:56:33.800 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:56:33.881 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:56:33.882 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:56:33.882 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:56:33.882 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:56:33.882 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:56:33.883 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T03:56:33.883 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:56:33.883 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:56:33.943 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:56:33.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:56:33.944 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.06 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 82.el9 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Group : Unspecified 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 8274670 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:56:33.987 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : builder@centos.org 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:56:33.988 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2024-09-16T03:56:33.989 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T03:56:33.989 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:56:33.990 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:56:33.990 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:56:33.990 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:56:34.667 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-09-16T03:56:35.978 INFO:teuthology.orchestra.run.smithi148.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:56:36.010 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-09-16T03:56:36.012 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-09-16T03:56:36.028 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.kOZRGTRt9M 2024-09-16T03:56:36.029 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kOZRGTRt9M 2024-09-16T03:56:36.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.kOZRGTRt9M 2024-09-16T03:56:36.231 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.kOZRGTRt9M is 6KB 2024-09-16T03:56:36.282 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.kOZRGTRt9M 2024-09-16T03:56:36.297 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:56:36.360 INFO:teuthology.orchestra.run.smithi148.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:56:36.360 INFO:teuthology.orchestra.run.smithi148.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:56:36.360 INFO:teuthology.orchestra.run.smithi148.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:56:36.360 INFO:teuthology.orchestra.run.smithi148.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:56:36.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:56:36.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-09-16T03:57:06.499 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T03:57:06.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T03:57:06.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:09.554 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:09.906 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:09.906 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:57:09.906 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:57:09.906 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:57:10.817 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T03:57:10.818 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T03:57:11.171 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:57:11.262 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:57:11.262 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:57:11.262 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:57:11.263 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:57:11.263 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:57:11.263 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T03:57:11.263 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:57:11.263 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:57:11.322 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:57:11.322 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:57:11.322 DEBUG:teuthology.orchestra.run.smithi171:> sudo rpm -qi grub2-tools 2024-09-16T03:57:11.355 INFO:teuthology.orchestra.run.smithi171.stdout:Name : grub2-tools 2024-09-16T03:57:11.355 INFO:teuthology.orchestra.run.smithi171.stdout:Epoch : 1 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Version : 2.06 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Release : 82.el9 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: x86_64 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Group : Unspecified 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Size : 8274670 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:License : GPLv3+ 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:57:11.356 INFO:teuthology.orchestra.run.smithi171.stdout:Packager : builder@centos.org 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:Vendor : CentOS 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:Summary : Support tools for GRUB. 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:Description : 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:hardware devices. 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T03:57:11.357 INFO:teuthology.orchestra.run.smithi171.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:57:11.358 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:57:11.358 INFO:teuthology.task.kernel:Updating grub on smithi171 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:57:11.359 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:57:12.015 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2024-09-16T03:57:13.372 INFO:teuthology.orchestra.run.smithi171.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:57:13.400 INFO:teuthology.orchestra.run.smithi171.stderr:done 2024-09-16T03:57:13.403 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-09-16T03:57:13.418 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.ZjbVVeUIad 2024-09-16T03:57:13.419 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZjbVVeUIad 2024-09-16T03:57:13.485 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.ZjbVVeUIad 2024-09-16T03:57:13.621 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.ZjbVVeUIad is 6KB 2024-09-16T03:57:13.672 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.ZjbVVeUIad 2024-09-16T03:57:13.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:57:13.756 INFO:teuthology.orchestra.run.smithi171.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:57:13.757 INFO:teuthology.orchestra.run.smithi171.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:57:13.757 INFO:teuthology.orchestra.run.smithi171.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:57:13.757 INFO:teuthology.orchestra.run.smithi171.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:57:13.758 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:57:13.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2024-09-16T03:57:24.925 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-09-16T03:57:33.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T03:57:33.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:36.993 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-09-16T03:57:43.935 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T03:57:43.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T03:57:43.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:57:48.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T03:57:48.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:58:02.301 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-09-16T03:58:11.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T03:58:11.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:58:14.369 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-09-16T03:58:20.944 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-09-16T03:58:21.407 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T03:58:21.407 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:58:21.408 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-09-16T03:58:21.422 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:58:21.422 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:58:21.422 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:58:21.422 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:58:22.424 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T03:58:22.424 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:58:22.503 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-09-16T03:58:22.529 DEBUG:teuthology.parallel:result is None 2024-09-16T03:58:26.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T03:58:26.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:59:26.372 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:59:41.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T03:59:41.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:59:41.648 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-09-16T03:59:41.995 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T03:59:41.995 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:59:41.995 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-09-16T03:59:42.011 INFO:teuthology.orchestra.run.smithi171.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:59:42.011 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:59:42.011 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:59:42.011 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:59:43.012 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T03:59:43.013 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:59:43.388 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2024-09-16T03:59:43.416 DEBUG:teuthology.parallel:result is None 2024-09-16T03:59:43.416 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T03:59:43.423 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T03:59:43.424 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:59:43.427 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:59:43.446 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T03:59:43.473 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T03:59:43.523 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T03:59:43.524 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:59:43.526 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:59:43.805 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T03:59:43.812 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T03:59:43.812 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:59:43.815 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:59:43.848 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:59:43.860 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:59:43.862 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:59:43.874 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:59:43.876 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T03:59:43.883 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T03:59:43.883 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:59:43.904 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:59:43.932 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T03:59:43.942 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T03:59:43.942 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:59:43.973 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:59:43.991 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:59:44.074 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:59:44.124 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:59:44.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:59:44.248 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T03:59:44.248 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:59:44.314 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:59:44.358 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:59:44.416 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:59:44.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:59:44.545 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T03:59:44.545 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:59:44.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-09-16T03:59:44.616 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-09-16T03:59:44.673 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:59:44.707 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:59:45.012 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T03:59:45.020 INFO:teuthology.task.internal:Starting timer... 2024-09-16T03:59:45.020 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T03:59:45.070 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T03:59:45.117 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T03:59:45.118 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-09-16T03:59:45.155 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-09-16T03:59:45.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-09-16T03:59:45.227 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-09-16T03:59:45.229 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T03:59:45.229 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-09-16T03:59:45.248 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-09-16T03:59:45.249 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-09-16T03:59:45.274 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-09-16T03:59:45.274 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi148.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-09-16T03:59:45.274 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:59:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:45.316 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T03:59:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:59:45.344 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T03:59:45.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:59:45.386 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:59:45.423 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T03:59:45.432 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T03:59:45.443 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T03:59:45.444 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi148.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T04:03:34.007 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi148.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-09-16T04:03:34.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T04:03:34.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:03:34.089 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-09-16T04:03:34.180 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-16T04:03:34.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T04:03:34.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T04:03:34.257 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-09-16T04:03:34.333 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-09-16T04:03:34.333 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T04:03:34.343 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T04:03:34.344 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:03:34.344 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:03:34.346 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T04:03:34.346 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:03:34.373 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:03:34.388 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:03:34.403 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T04:03:34.412 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2024-09-16T04:03:34.418 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T04:03:34.423 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2024-09-16T04:03:34.438 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:03:34.443 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-09-16T04:03:34.452 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:03:34.454 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-09-16T04:03:34.468 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T04:03:34.482 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T04:03:34.512 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:03:34.515 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:03:34.515 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-09-16T04:03:34.516 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.516 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.516 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.516 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.547 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:03:34.550 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:03:34.550 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-09-16T04:03:34.550 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.551 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.551 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.551 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:03:34.552 INFO:teuthology.run_tasks:Running task install... 2024-09-16T04:03:34.560 DEBUG:teuthology.task.install:project ceph 2024-09-16T04:03:34.560 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T04:03:34.560 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T04:03:34.560 INFO:teuthology.task.install:Using flavor: default 2024-09-16T04:03:34.565 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T04:03:34.565 INFO:teuthology.task.install:extra packages: [] 2024-09-16T04:03:34.566 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:03:34.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:03:34.569 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T04:03:34.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:03:34.710 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:03:34.710 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:03:34.726 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T04:03:34.726 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T04:03:34.926 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:03:34.926 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:03:34.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:03:34.937 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T04:03:34.938 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:03:34.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T04:03:34.953 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:03:34.953 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:03:34.964 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T04:03:34.964 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T04:03:35.019 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:03:35.029 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T04:03:35.095 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:03:35.105 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T04:03:35.158 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2024-09-16T04:03:35.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2024-09-16T04:03:35.169 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-09-16T04:03:35.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-09-16T04:03:35.404 INFO:teuthology.orchestra.run.smithi171.stdout:45 files removed 2024-09-16T04:03:35.419 INFO:teuthology.orchestra.run.smithi148.stdout:45 files removed 2024-09-16T04:03:35.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw 2024-09-16T04:03:35.448 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw 2024-09-16T04:03:36.301 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 143 kB/s | 83 kB 00:00 2024-09-16T04:03:36.324 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 142 kB/s | 83 kB 00:00 2024-09-16T04:03:36.894 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 21 kB/s | 12 kB 00:00 2024-09-16T04:03:36.908 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 22 kB/s | 12 kB 00:00 2024-09-16T04:03:37.545 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-09-16T04:03:37.559 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-09-16T04:03:37.975 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 22 MB/s | 8.2 MB 00:00 2024-09-16T04:03:38.287 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T04:03:39.868 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-16T04:03:40.380 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:00 2024-09-16T04:03:44.177 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T04:03:44.715 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-16T04:03:45.645 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 19 kB 00:00 2024-09-16T04:03:46.483 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-16T04:03:46.640 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-16T04:03:47.513 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T04:03:52.172 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T04:03:53.052 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T04:03:54.523 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:03:54.584 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:03:54.584 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:03:54.584 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:03:54.585 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:03:54.586 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:03:54.587 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:03:54.588 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:03:54.588 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:03:54.588 INFO:teuthology.orchestra.run.smithi148.stdout:Install 30 Packages 2024-09-16T04:03:54.588 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 2 Packages 2024-09-16T04:03:54.588 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:03:54.588 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 60 M 2024-09-16T04:03:54.589 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:03:55.438 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T04:03:55.453 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T04:03:55.454 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T04:03:55.455 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout:Install 30 Packages 2024-09-16T04:03:55.456 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2024-09-16T04:03:55.457 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:03:55.457 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 60 M 2024-09-16T04:03:55.457 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:03:55.898 INFO:teuthology.orchestra.run.smithi148.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 4.6 MB/s | 5.4 MB 00:01 2024-09-16T04:03:55.956 INFO:teuthology.orchestra.run.smithi148.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 433 kB/s | 25 kB 00:00 2024-09-16T04:03:56.065 INFO:teuthology.orchestra.run.smithi148.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 6.8 MB/s | 751 kB 00:00 2024-09-16T04:03:56.148 INFO:teuthology.orchestra.run.smithi148.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 5.9 MB/s | 506 kB 00:00 2024-09-16T04:03:56.324 INFO:teuthology.orchestra.run.smithi148.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 6.6 MB/s | 11 MB 00:01 2024-09-16T04:03:56.374 INFO:teuthology.orchestra.run.smithi148.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 906 kB/s | 45 kB 00:00 2024-09-16T04:03:56.443 INFO:teuthology.orchestra.run.smithi148.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 1.9 MB/s | 136 kB 00:00 2024-09-16T04:03:56.510 INFO:teuthology.orchestra.run.smithi148.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.4 MB/s | 161 kB 00:00 2024-09-16T04:03:56.585 INFO:teuthology.orchestra.run.smithi148.stdout:(9/32): python3-rados-19.1.1-326.g8921c8ef.el9. 4.2 MB/s | 320 kB 00:00 2024-09-16T04:03:56.660 INFO:teuthology.orchestra.run.smithi148.stdout:(10/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 10 MB/s | 5.3 MB 00:00 2024-09-16T04:03:56.686 INFO:teuthology.orchestra.run.smithi148.stdout:(11/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 2.9 MB/s | 300 kB 00:00 2024-09-16T04:03:56.703 INFO:teuthology.orchestra.run.smithi148.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 2.3 MB/s | 99 kB 00:00 2024-09-16T04:03:56.879 INFO:teuthology.orchestra.run.smithi171.stdout:(1/32): ceph-base-19.1.1-326.g8921c8ef.el9.x86_ 4.3 MB/s | 5.4 MB 00:01 2024-09-16T04:03:56.911 INFO:teuthology.orchestra.run.smithi148.stdout:(13/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 9.7 MB/s | 21 MB 00:02 2024-09-16T04:03:56.929 INFO:teuthology.orchestra.run.smithi171.stdout:(2/32): ceph-selinux-19.1.1-326.g8921c8ef.el9.x 505 kB/s | 25 kB 00:00 2024-09-16T04:03:57.012 INFO:teuthology.orchestra.run.smithi148.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 328 kB/s | 107 kB 00:00 2024-09-16T04:03:57.028 INFO:teuthology.orchestra.run.smithi148.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 492 kB/s | 160 kB 00:00 2024-09-16T04:03:57.046 INFO:teuthology.orchestra.run.smithi171.stdout:(3/32): libcephfs2-19.1.1-326.g8921c8ef.el9.x86 6.3 MB/s | 751 kB 00:00 2024-09-16T04:03:57.053 INFO:teuthology.orchestra.run.smithi148.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-09-16T04:03:57.112 INFO:teuthology.orchestra.run.smithi148.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-09-16T04:03:57.130 INFO:teuthology.orchestra.run.smithi171.stdout:(4/32): libradosstriper1-19.1.1-326.g8921c8ef.e 5.9 MB/s | 506 kB 00:00 2024-09-16T04:03:57.146 INFO:teuthology.orchestra.run.smithi148.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-16T04:03:57.171 INFO:teuthology.orchestra.run.smithi148.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 360 kB/s | 42 kB 00:00 2024-09-16T04:03:57.196 INFO:teuthology.orchestra.run.smithi148.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T04:03:57.222 INFO:teuthology.orchestra.run.smithi148.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 474 kB/s | 36 kB 00:00 2024-09-16T04:03:57.248 INFO:teuthology.orchestra.run.smithi171.stdout:(5/32): ceph-radosgw-19.1.1-326.g8921c8ef.el9.x 6.5 MB/s | 11 MB 00:01 2024-09-16T04:03:57.264 INFO:teuthology.orchestra.run.smithi148.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-09-16T04:03:57.306 INFO:teuthology.orchestra.run.smithi148.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T04:03:57.306 INFO:teuthology.orchestra.run.smithi171.stdout:(6/32): python3-ceph-argparse-19.1.1-326.g8921c 774 kB/s | 45 kB 00:00 2024-09-16T04:03:57.331 INFO:teuthology.orchestra.run.smithi148.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-16T04:03:57.356 INFO:teuthology.orchestra.run.smithi148.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 289 kB/s | 26 kB 00:00 2024-09-16T04:03:57.364 INFO:teuthology.orchestra.run.smithi171.stdout:(7/32): python3-ceph-common-19.1.1-326.g8921c8e 2.3 MB/s | 136 kB 00:00 2024-09-16T04:03:57.381 INFO:teuthology.orchestra.run.smithi148.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T04:03:57.415 INFO:teuthology.orchestra.run.smithi148.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T04:03:57.433 INFO:teuthology.orchestra.run.smithi171.stdout:(8/32): python3-cephfs-19.1.1-326.g8921c8ef.el9 2.4 MB/s | 161 kB 00:00 2024-09-16T04:03:57.457 INFO:teuthology.orchestra.run.smithi148.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-16T04:03:57.498 INFO:teuthology.orchestra.run.smithi171.stdout:(9/32): python3-rados-19.1.1-326.g8921c8ef.el9. 4.7 MB/s | 320 kB 00:00 2024-09-16T04:03:57.565 INFO:teuthology.orchestra.run.smithi148.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-16T04:03:57.607 INFO:teuthology.orchestra.run.smithi148.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-09-16T04:03:57.732 INFO:teuthology.orchestra.run.smithi171.stdout:(10/32): ceph-common-19.1.1-326.g8921c8ef.el9.x 10 MB/s | 21 MB 00:02 2024-09-16T04:03:57.766 INFO:teuthology.orchestra.run.smithi148.stdout:(31/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 16 MB/s | 3.2 MB 00:00 2024-09-16T04:03:57.808 INFO:teuthology.orchestra.run.smithi171.stdout:(11/32): librgw2-19.1.1-326.g8921c8ef.el9.x86_6 7.9 MB/s | 5.3 MB 00:00 2024-09-16T04:03:57.816 INFO:teuthology.orchestra.run.smithi148.stdout:(32/32): librados2-19.1.1-326.g8921c8ef.el9.x86 9.5 MB/s | 3.4 MB 00:00 2024-09-16T04:03:57.821 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:03:57.821 INFO:teuthology.orchestra.run.smithi148.stdout:Total 19 MB/s | 60 MB 00:03 2024-09-16T04:03:57.833 INFO:teuthology.orchestra.run.smithi171.stdout:(12/32): python3-rgw-19.1.1-326.g8921c8ef.el9.x 991 kB/s | 99 kB 00:00 2024-09-16T04:03:57.858 INFO:teuthology.orchestra.run.smithi171.stdout:(13/32): python3-rbd-19.1.1-326.g8921c8ef.el9.x 835 kB/s | 300 kB 00:00 2024-09-16T04:03:57.923 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:03:57.951 INFO:teuthology.orchestra.run.smithi171.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 751 kB/s | 107 kB 00:00 2024-09-16T04:03:57.959 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:03:57.959 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:03:57.976 INFO:teuthology.orchestra.run.smithi171.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-16T04:03:58.010 INFO:teuthology.orchestra.run.smithi171.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 909 kB/s | 160 kB 00:00 2024-09-16T04:03:58.052 INFO:teuthology.orchestra.run.smithi171.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T04:03:58.077 INFO:teuthology.orchestra.run.smithi171.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-16T04:03:58.102 INFO:teuthology.orchestra.run.smithi171.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-16T04:03:58.127 INFO:teuthology.orchestra.run.smithi171.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-16T04:03:58.153 INFO:teuthology.orchestra.run.smithi171.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 715 kB/s | 36 kB 00:00 2024-09-16T04:03:58.178 INFO:teuthology.orchestra.run.smithi171.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-16T04:03:58.237 INFO:teuthology.orchestra.run.smithi171.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-16T04:03:58.262 INFO:teuthology.orchestra.run.smithi171.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-09-16T04:03:58.287 INFO:teuthology.orchestra.run.smithi171.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 966 kB/s | 49 kB 00:00 2024-09-16T04:03:58.313 INFO:teuthology.orchestra.run.smithi171.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T04:03:58.355 INFO:teuthology.orchestra.run.smithi171.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T04:03:58.389 INFO:teuthology.orchestra.run.smithi171.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-16T04:03:58.473 INFO:teuthology.orchestra.run.smithi171.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-16T04:03:58.598 INFO:teuthology.orchestra.run.smithi171.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-16T04:03:58.611 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:03:58.612 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:03:58.723 INFO:teuthology.orchestra.run.smithi171.stdout:(31/32): librados2-19.1.1-326.g8921c8ef.el9.x86 10 MB/s | 3.4 MB 00:00 2024-09-16T04:03:58.782 INFO:teuthology.orchestra.run.smithi171.stdout:(32/32): librbd1-19.1.1-326.g8921c8ef.el9.x86_6 10 MB/s | 3.2 MB 00:00 2024-09-16T04:03:58.787 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:03:58.787 INFO:teuthology.orchestra.run.smithi171.stdout:Total 18 MB/s | 60 MB 00:03 2024-09-16T04:03:58.886 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:03:58.923 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:03:58.923 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:03:59.608 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:03:59.609 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:04:00.099 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:04:00.180 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:04:00.213 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:04:00.246 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:04:00.269 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:00.293 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:00.487 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:00.540 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:00.925 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:04:00.973 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:01.048 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:01.083 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:04:01.135 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:04:01.158 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-16T04:04:01.165 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:01.196 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-16T04:04:01.221 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-16T04:04:01.228 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:01.253 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:01.257 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:04:01.275 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-16T04:04:01.306 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:04:01.377 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:01.401 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:01.465 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:01.473 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:01.500 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:04:01.536 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-16T04:04:01.616 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:04:01.651 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:04:01.692 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:04:01.724 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:04:01.762 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:04:01.797 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:04:01.916 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-16T04:04:02.017 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:02.115 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:04:02.126 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:02.157 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:02.215 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 8/34 2024-09-16T04:04:02.231 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:02.244 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:02.268 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:04:02.298 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/34 2024-09-16T04:04:02.306 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:04:02.336 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-16T04:04:02.380 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 11/34 2024-09-16T04:04:02.448 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:02.472 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:02.536 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:02.571 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-16T04:04:02.679 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-16T04:04:02.714 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-16T04:04:02.760 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-16T04:04:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:04:02.788 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-16T04:04:02.809 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:02.817 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-16T04:04:02.850 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-16T04:04:02.881 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:02.959 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:04:02.995 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:04:03.162 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-16T04:04:03.214 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:03.304 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 22/34 2024-09-16T04:04:03.344 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 23/34 2024-09-16T04:04:03.396 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-16T04:04:03.888 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-16T04:04:03.917 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:03.988 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 26/34 2024-09-16T04:04:04.066 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 27/34 2024-09-16T04:04:04.095 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 28/34 2024-09-16T04:04:05.265 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:05.308 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:05.884 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:05.920 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:05.995 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:05.996 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:04:05.996 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:04:05.996 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:06.021 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:06.434 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:06.476 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:07.065 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 29/34 2024-09-16T04:04:07.123 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:07.205 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 30/34 2024-09-16T04:04:07.205 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T04:04:07.206 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T04:04:07.206 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:07.241 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /root 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2024-09-16T04:04:16.135 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:17.146 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:04:17.174 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:04:17.174 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:17.174 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:04:17.174 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:04:17.174 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:04:17.174 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:17.176 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:04:17.273 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:04:17.274 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:04:17.634 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:17.635 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-09-16T04:04:17.635 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-09-16T04:04:17.635 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-09-16T04:04:17.635 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-09-16T04:04:17.635 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-09-16T04:04:17.635 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /root 2024-09-16T04:04:17.636 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-09-16T04:04:17.636 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:18.441 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:04:18.441 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:04:18.441 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:18.442 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:04:18.443 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:04:18.444 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:04:18.640 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:04:18.665 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 32/34 2024-09-16T04:04:18.665 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:18.665 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T04:04:18.665 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:04:18.666 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T04:04:18.666 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:18.668 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:04:18.760 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-16T04:04:18.760 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:04:19.530 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.531 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.532 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:19.533 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:19.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-test 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 1/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 3/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 4/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 5/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 6/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 7/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x 8/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86 9/34 2024-09-16T04:04:20.013 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 10/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 11/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 12/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 13/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-16T04:04:20.014 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 31/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-16T04:04:20.015 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 33/34 2024-09-16T04:04:20.459 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:20.600 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:20.605 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:20.605 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:04:20.605 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:20.605 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:04:20.605 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:04:20.606 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:04:20.606 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:04:20.606 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:04:20.606 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:04:20.606 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:20.607 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:04:20.607 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:20.607 INFO:teuthology.orchestra.run.smithi148.stdout:Install 4 Packages 2024-09-16T04:04:20.607 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:20.607 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 49 M 2024-09-16T04:04:20.607 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 209 M 2024-09-16T04:04:20.608 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.028 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T04:04:21.029 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T04:04:21.030 INFO:teuthology.orchestra.run.smithi171.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T04:04:21.031 INFO:teuthology.orchestra.run.smithi171.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T04:04:21.031 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:21.031 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:21.037 INFO:teuthology.orchestra.run.smithi148.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-16T04:04:21.079 INFO:teuthology.orchestra.run.smithi148.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T04:04:21.120 INFO:teuthology.orchestra.run.smithi148.stdout:(3/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 693 kB/s | 163 kB 00:00 2024-09-16T04:04:21.384 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test 2024-09-16T04:04:21.955 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:22.097 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T04:04:22.098 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:22.099 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:04:22.099 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:22.099 INFO:teuthology.orchestra.run.smithi171.stdout:Install 4 Packages 2024-09-16T04:04:22.099 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:22.099 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 49 M 2024-09-16T04:04:22.099 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 209 M 2024-09-16T04:04:22.100 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:04:22.286 INFO:teuthology.orchestra.run.smithi171.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-16T04:04:22.328 INFO:teuthology.orchestra.run.smithi171.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T04:04:22.378 INFO:teuthology.orchestra.run.smithi171.stdout:(3/4): libcephsqlite-19.1.1-326.g8921c8ef.el9.x 720 kB/s | 163 kB 00:00 2024-09-16T04:04:22.546 INFO:teuthology.orchestra.run.smithi148.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 29 MB/s | 49 MB 00:01 2024-09-16T04:04:22.548 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:22.548 INFO:teuthology.orchestra.run.smithi148.stdout:Total 25 MB/s | 49 MB 00:01 2024-09-16T04:04:22.576 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:04:22.596 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:04:22.596 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:04:22.989 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:04:22.990 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:04:23.527 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:04:23.608 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:04:23.653 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:04:23.681 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:04:23.720 INFO:teuthology.orchestra.run.smithi171.stdout:(4/4): ceph-test-19.1.1-326.g8921c8ef.el9.x86_6 31 MB/s | 49 MB 00:01 2024-09-16T04:04:23.723 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:23.723 INFO:teuthology.orchestra.run.smithi171.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-16T04:04:23.753 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:04:23.773 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:04:23.773 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:04:24.165 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:04:24.165 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:04:24.691 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:04:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T04:04:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T04:04:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:04:28.302 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:04:28.372 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:04:29.483 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 3/4 2024-09-16T04:04:29.563 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:04:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:04:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:04:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:04:30.836 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:04:31.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:31.153 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:31.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph 2024-09-16T04:04:31.984 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 4/4 2024-09-16T04:04:31.984 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 1/4 2024-09-16T04:04:31.984 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2/4 2024-09-16T04:04:31.984 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T04:04:32.009 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:32.157 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:04:32.160 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:04:32.161 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:04:32.162 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout:Install 37 Packages 2024-09-16T04:04:32.163 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:32.164 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 30 M 2024-09-16T04:04:32.164 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 108 M 2024-09-16T04:04:32.164 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:04:32.289 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T04:04:32.289 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:32.289 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:04:32.289 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:32.289 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:32.290 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T04:04:32.290 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T04:04:32.290 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:32.290 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:32.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph 2024-09-16T04:04:32.738 INFO:teuthology.orchestra.run.smithi148.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 45 kB/s | 6.5 kB 00:00 2024-09-16T04:04:32.896 INFO:teuthology.orchestra.run.smithi148.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 5.0 MB/s | 1.5 MB 00:00 2024-09-16T04:04:32.947 INFO:teuthology.orchestra.run.smithi148.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.8 MB/s | 2.4 MB 00:00 2024-09-16T04:04:33.022 INFO:teuthology.orchestra.run.smithi148.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 3.3 MB/s | 249 kB 00:00 2024-09-16T04:04:33.167 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:33.223 INFO:teuthology.orchestra.run.smithi148.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-16T04:04:33.304 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T04:04:33.307 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T04:04:33.308 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T04:04:33.309 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T04:04:33.310 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T04:04:33.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:33.311 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:04:33.311 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:33.311 INFO:teuthology.orchestra.run.smithi171.stdout:Install 37 Packages 2024-09-16T04:04:33.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:33.312 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 30 M 2024-09-16T04:04:33.312 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 108 M 2024-09-16T04:04:33.312 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:04:33.431 INFO:teuthology.orchestra.run.smithi148.stdout:(6/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_6 32 MB/s | 17 MB 00:00 2024-09-16T04:04:33.531 INFO:teuthology.orchestra.run.smithi148.stdout:(7/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 5.9 MB/s | 4.7 MB 00:00 2024-09-16T04:04:33.565 INFO:teuthology.orchestra.run.smithi148.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 3.6 MB/s | 1.2 MB 00:00 2024-09-16T04:04:33.590 INFO:teuthology.orchestra.run.smithi148.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-16T04:04:33.649 INFO:teuthology.orchestra.run.smithi148.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-16T04:04:33.683 INFO:teuthology.orchestra.run.smithi148.stdout:(11/37): python3-ply-3.11-14.el9.noarch.rpm 424 kB/s | 106 kB 00:00 2024-09-16T04:04:33.758 INFO:teuthology.orchestra.run.smithi148.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 648 kB/s | 125 kB 00:00 2024-09-16T04:04:33.803 INFO:teuthology.orchestra.run.smithi148.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-16T04:04:33.828 INFO:teuthology.orchestra.run.smithi148.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-16T04:04:33.854 INFO:teuthology.orchestra.run.smithi148.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T04:04:33.879 INFO:teuthology.orchestra.run.smithi148.stdout:(16/37): python3-markupsafe-1.1.1-12.el9.x86_64 287 kB/s | 35 kB 00:00 2024-09-16T04:04:33.904 INFO:teuthology.orchestra.run.smithi148.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-16T04:04:33.929 INFO:teuthology.orchestra.run.smithi148.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-16T04:04:33.979 INFO:teuthology.orchestra.run.smithi148.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-16T04:04:34.005 INFO:teuthology.orchestra.run.smithi148.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-16T04:04:34.030 INFO:teuthology.orchestra.run.smithi148.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-16T04:04:34.064 INFO:teuthology.orchestra.run.smithi148.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T04:04:34.089 INFO:teuthology.orchestra.run.smithi148.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 206 kB/s | 11 kB 00:00 2024-09-16T04:04:34.114 INFO:teuthology.orchestra.run.smithi148.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 363 kB/s | 18 kB 00:00 2024-09-16T04:04:34.140 INFO:teuthology.orchestra.run.smithi148.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-16T04:04:34.165 INFO:teuthology.orchestra.run.smithi148.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T04:04:34.169 INFO:teuthology.orchestra.run.smithi171.stdout:(1/37): ceph-19.1.1-326.g8921c8ef.el9.x86_64.rp 54 kB/s | 6.5 kB 00:00 2024-09-16T04:04:34.190 INFO:teuthology.orchestra.run.smithi148.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T04:04:34.216 INFO:teuthology.orchestra.run.smithi148.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-16T04:04:34.241 INFO:teuthology.orchestra.run.smithi148.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T04:04:34.266 INFO:teuthology.orchestra.run.smithi148.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-16T04:04:34.291 INFO:teuthology.orchestra.run.smithi148.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T04:04:34.308 INFO:teuthology.orchestra.run.smithi148.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-16T04:04:34.342 INFO:teuthology.orchestra.run.smithi148.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T04:04:34.361 INFO:teuthology.orchestra.run.smithi171.stdout:(2/37): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-16T04:04:34.376 INFO:teuthology.orchestra.run.smithi148.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-09-16T04:04:34.401 INFO:teuthology.orchestra.run.smithi148.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 808 kB/s | 20 kB 00:00 2024-09-16T04:04:34.420 INFO:teuthology.orchestra.run.smithi171.stdout:(3/37): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_6 6.4 MB/s | 2.4 MB 00:00 2024-09-16T04:04:34.426 INFO:teuthology.orchestra.run.smithi148.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T04:04:34.451 INFO:teuthology.orchestra.run.smithi148.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T04:04:34.456 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:34.456 INFO:teuthology.orchestra.run.smithi148.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-16T04:04:34.492 INFO:teuthology.orchestra.run.smithi171.stdout:(4/37): ceph-mgr-modules-core-19.1.1-326.g8921c 3.4 MB/s | 249 kB 00:00 2024-09-16T04:04:34.605 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:04:34.608 INFO:teuthology.orchestra.run.smithi171.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-16T04:04:34.644 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:04:34.644 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:04:34.709 INFO:teuthology.orchestra.run.smithi171.stdout:(6/37): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_6 8.7 MB/s | 4.7 MB 00:00 2024-09-16T04:04:34.776 INFO:teuthology.orchestra.run.smithi171.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 7.3 MB/s | 1.2 MB 00:00 2024-09-16T04:04:34.818 INFO:teuthology.orchestra.run.smithi171.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T04:04:34.843 INFO:teuthology.orchestra.run.smithi171.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 794 kB/s | 106 kB 00:00 2024-09-16T04:04:34.869 INFO:teuthology.orchestra.run.smithi171.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-16T04:04:35.061 INFO:teuthology.orchestra.run.smithi171.stdout:(11/37): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 24 MB/s | 17 MB 00:00 2024-09-16T04:04:35.086 INFO:teuthology.orchestra.run.smithi171.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 885 kB/s | 215 kB 00:00 2024-09-16T04:04:35.117 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:04:35.118 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:04:35.195 INFO:teuthology.orchestra.run.smithi171.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T04:04:35.220 INFO:teuthology.orchestra.run.smithi171.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 537 kB/s | 188 kB 00:00 2024-09-16T04:04:35.246 INFO:teuthology.orchestra.run.smithi171.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 933 kB/s | 172 kB 00:00 2024-09-16T04:04:35.263 INFO:teuthology.orchestra.run.smithi171.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-16T04:04:35.289 INFO:teuthology.orchestra.run.smithi171.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 829 kB/s | 77 kB 00:00 2024-09-16T04:04:35.314 INFO:teuthology.orchestra.run.smithi171.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 616 kB/s | 42 kB 00:00 2024-09-16T04:04:35.372 INFO:teuthology.orchestra.run.smithi171.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-16T04:04:35.398 INFO:teuthology.orchestra.run.smithi171.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-09-16T04:04:35.423 INFO:teuthology.orchestra.run.smithi171.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-16T04:04:35.449 INFO:teuthology.orchestra.run.smithi171.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-16T04:04:35.474 INFO:teuthology.orchestra.run.smithi171.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 709 kB/s | 18 kB 00:00 2024-09-16T04:04:35.499 INFO:teuthology.orchestra.run.smithi171.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-16T04:04:35.525 INFO:teuthology.orchestra.run.smithi171.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 458 kB/s | 23 kB 00:00 2024-09-16T04:04:35.558 INFO:teuthology.orchestra.run.smithi171.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2024-09-16T04:04:35.584 INFO:teuthology.orchestra.run.smithi171.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 380 kB/s | 20 kB 00:00 2024-09-16T04:04:35.609 INFO:teuthology.orchestra.run.smithi171.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-16T04:04:35.642 INFO:teuthology.orchestra.run.smithi171.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-16T04:04:35.669 INFO:teuthology.orchestra.run.smithi171.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 653 kB/s | 16 kB 00:00 2024-09-16T04:04:35.694 INFO:teuthology.orchestra.run.smithi171.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-16T04:04:35.719 INFO:teuthology.orchestra.run.smithi171.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T04:04:35.744 INFO:teuthology.orchestra.run.smithi171.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-16T04:04:35.778 INFO:teuthology.orchestra.run.smithi171.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-16T04:04:35.803 INFO:teuthology.orchestra.run.smithi171.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 795 kB/s | 20 kB 00:00 2024-09-16T04:04:35.828 INFO:teuthology.orchestra.run.smithi171.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-16T04:04:35.862 INFO:teuthology.orchestra.run.smithi171.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-09-16T04:04:35.866 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:35.866 INFO:teuthology.orchestra.run.smithi171.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-16T04:04:36.012 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:04:36.052 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:04:36.052 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:04:36.489 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:04:36.545 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:04:36.554 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:04:36.554 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:04:36.594 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:04:36.623 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:04:36.695 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:04:36.760 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:04:36.788 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:04:36.881 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:04:36.960 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:04:36.998 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:04:37.034 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:04:37.064 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:04:37.098 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:04:37.132 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:04:37.173 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:04:37.207 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:04:37.242 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:04:37.290 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:04:37.323 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:04:37.396 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:04:37.535 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:04:37.614 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:04:37.666 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:04:37.705 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:04:37.754 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:04:37.809 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:04:37.911 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:04:37.966 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T04:04:37.994 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:04:38.008 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T04:04:38.043 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:04:38.043 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T04:04:38.122 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:04:38.146 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T04:04:38.193 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T04:04:38.209 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:04:38.226 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T04:04:38.258 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:04:38.327 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T04:04:38.374 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:04:38.413 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T04:04:38.455 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T04:04:38.490 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T04:04:38.520 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T04:04:38.531 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:04:38.555 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T04:04:38.560 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:04:38.587 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:04:38.588 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:38.588 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:04:38.588 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:04:38.588 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:04:38.588 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:38.588 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T04:04:38.637 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T04:04:38.681 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T04:04:38.716 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T04:04:38.764 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T04:04:38.797 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T04:04:38.878 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T04:04:39.026 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T04:04:39.106 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T04:04:39.159 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T04:04:39.198 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T04:04:39.255 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T04:04:39.311 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T04:04:39.498 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T04:04:39.545 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T04:04:39.624 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T04:04:39.712 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T04:04:39.766 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T04:04:39.887 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T04:04:40.028 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:04:40.049 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 32/37 2024-09-16T04:04:40.055 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:04:40.055 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:40.055 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:04:40.055 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:04:40.056 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:04:40.056 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:40.081 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:04:40.107 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 33/37 2024-09-16T04:04:40.107 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:40.108 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T04:04:40.108 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:04:40.108 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T04:04:40.108 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:40.503 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:04:40.527 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:04:40.527 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:40.528 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:04:40.528 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:04:40.528 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:04:40.528 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:40.762 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:04:40.786 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:04:40.786 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:40.787 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:04:40.787 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:04:40.787 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:04:40.787 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:40.851 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:04:41.541 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:04:41.568 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 34/37 2024-09-16T04:04:41.569 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:41.569 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T04:04:41.569 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:04:41.569 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T04:04:41.569 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:42.009 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 35/37 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T04:04:42.033 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:42.268 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:04:42.292 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 36/37 2024-09-16T04:04:42.292 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:42.292 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T04:04:42.292 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:04:42.292 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T04:04:42.292 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:42.357 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:04:42.670 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:04:42.670 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:04:42.670 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:04:42.670 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:04:42.670 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:04:42.670 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:04:42.671 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:04:42.672 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:04:42.673 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:04:42.674 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:04:43.646 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:04:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:43.647 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:04:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:43.647 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:04:43.648 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:04:43.649 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:04:43.650 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:43.651 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:43.966 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-base 2024-09-16T04:04:44.189 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 37/37 2024-09-16T04:04:44.189 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/37 2024-09-16T04:04:44.189 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2/37 2024-09-16T04:04:44.189 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 3/37 2024-09-16T04:04:44.189 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 4/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 5/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.n 6/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T04:04:44.190 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T04:04:44.191 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T04:04:44.192 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T04:04:44.192 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T04:04:44.192 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T04:04:44.192 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T04:04:44.192 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T04:04:44.192 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T04:04:44.536 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:04:44.668 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:44.669 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:04:44.669 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:44.731 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install cephadm 2024-09-16T04:04:45.232 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T04:04:45.233 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T04:04:45.234 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T04:04:45.235 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T04:04:45.236 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T04:04:45.236 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:45.236 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:45.295 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:45.427 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:45.428 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 765 k 2024-09-16T04:04:45.429 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 771 k 2024-09-16T04:04:45.429 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:04:45.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base 2024-09-16T04:04:45.686 INFO:teuthology.orchestra.run.smithi148.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-16T04:04:45.686 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:45.686 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-16T04:04:45.687 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:04:45.692 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:04:45.692 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:04:45.697 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:04:45.698 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:04:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:04:46.174 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:46.247 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:46.263 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:04:46.305 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:46.306 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:04:46.306 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:46.347 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:46.368 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install cephadm 2024-09-16T04:04:46.723 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:46.930 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:46.970 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:46.970 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:46.971 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:04:46.971 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:46.971 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:46.971 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:47.064 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:47.064 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 765 k 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 771 k 2024-09-16T04:04:47.065 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:04:47.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:04:47.440 INFO:teuthology.orchestra.run.smithi171.stdout:cephadm-19.1.1-326.g8921c8ef.el9.noarch.rpm 2.0 MB/s | 765 kB 00:00 2024-09-16T04:04:47.440 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:47.440 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.0 MB/s | 765 kB 00:00 2024-09-16T04:04:47.441 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:04:47.445 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:04:47.446 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:04:47.451 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:04:47.452 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:04:47.525 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:04:47.731 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:47.867 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:47.868 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:47.868 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2024-09-16T04:04:47.868 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:47.868 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:04:47.868 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:04:47.868 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 145 k 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 435 k 2024-09-16T04:04:47.869 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:04:48.003 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:48.066 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 734 kB/s | 145 kB 00:00 2024-09-16T04:04:48.067 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:48.067 INFO:teuthology.orchestra.run.smithi148.stdout:Total 727 kB/s | 145 kB 00:00 2024-09-16T04:04:48.068 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:04:48.075 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:04:48.075 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:04:48.086 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:48.132 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:04:48.133 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:04:48.289 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:04:48.367 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:04:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:04:48.393 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:48.393 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:04:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:48.437 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:04:48.720 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:48.720 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:04:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:48.721 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:48.721 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:48.953 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T04:04:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:04:49.300 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:49.300 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:04:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:49.301 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:49.301 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:49.530 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr 2024-09-16T04:04:49.532 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:49.667 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:49.668 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:49.668 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-09-16T04:04:49.668 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:49.668 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:04:49.668 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 145 k 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 435 k 2024-09-16T04:04:49.669 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:04:49.885 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-immutable-object-cache-19.1.1-326.g8921c8e 675 kB/s | 145 kB 00:00 2024-09-16T04:04:49.886 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:49.886 INFO:teuthology.orchestra.run.smithi171.stdout:Total 669 kB/s | 145 kB 00:00 2024-09-16T04:04:49.886 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:04:49.893 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:04:49.894 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:04:49.952 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:04:49.952 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:04:50.103 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:50.105 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:04:50.186 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:04:50.198 INFO:teuthology.orchestra.run.smithi148.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:04:50.211 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:04:50.211 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:04:50.211 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T04:04:50.211 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:50.240 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:50.241 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:04:50.241 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:50.299 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:04:50.865 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:50.999 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:51.000 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:51.000 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:04:51.000 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:51.000 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:04:51.000 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:04:51.000 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:04:51.001 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:04:51.002 INFO:teuthology.orchestra.run.smithi148.stdout:Install 12 Packages 2024-09-16T04:04:51.002 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:51.002 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 8.7 M 2024-09-16T04:04:51.002 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 104 M 2024-09-16T04:04:51.002 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:04:51.078 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.e 1/1 2024-09-16T04:04:51.079 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:51.079 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:04:51.079 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:04:51.079 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:51.079 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:51.297 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr 2024-09-16T04:04:51.613 INFO:teuthology.orchestra.run.smithi148.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 119 kB/s | 32 kB 00:00 2024-09-16T04:04:51.638 INFO:teuthology.orchestra.run.smithi148.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 56 kB/s | 16 kB 00:00 2024-09-16T04:04:51.871 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:51.964 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:04:52.007 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:52.008 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:04:52.008 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:52.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T04:04:52.414 INFO:teuthology.orchestra.run.smithi148.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 3.3 MB/s | 3.6 MB 00:01 2024-09-16T04:04:52.439 INFO:teuthology.orchestra.run.smithi148.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 336 kB/s | 269 kB 00:00 2024-09-16T04:04:52.497 INFO:teuthology.orchestra.run.smithi148.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.3 MB/s | 548 kB 00:00 2024-09-16T04:04:52.522 INFO:teuthology.orchestra.run.smithi148.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-16T04:04:52.557 INFO:teuthology.orchestra.run.smithi148.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-16T04:04:52.615 INFO:teuthology.orchestra.run.smithi148.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-16T04:04:52.631 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:04:52.641 INFO:teuthology.orchestra.run.smithi148.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T04:04:52.674 INFO:teuthology.orchestra.run.smithi148.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-16T04:04:52.732 INFO:teuthology.orchestra.run.smithi148.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.7 MB/s | 2.0 MB 00:00 2024-09-16T04:04:52.764 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:04:52.765 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:52.765 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:04:52.765 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:52.765 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T04:04:52.766 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T04:04:52.767 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:52.767 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:04:52.767 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:04:52.767 INFO:teuthology.orchestra.run.smithi171.stdout:Install 12 Packages 2024-09-16T04:04:52.767 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:52.767 INFO:teuthology.orchestra.run.smithi148.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.4 MB/s | 863 kB 00:00 2024-09-16T04:04:52.768 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 8.7 M 2024-09-16T04:04:52.768 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 104 M 2024-09-16T04:04:52.768 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:04:52.770 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:52.770 INFO:teuthology.orchestra.run.smithi148.stdout:Total 4.9 MB/s | 8.7 MB 00:01 2024-09-16T04:04:52.819 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:04:52.836 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:04:52.836 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:04:53.036 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:04:53.037 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:04:53.434 INFO:teuthology.orchestra.run.smithi171.stdout:(1/12): ceph-grafana-dashboards-19.1.1-326.g892 208 kB/s | 32 kB 00:00 2024-09-16T04:04:53.459 INFO:teuthology.orchestra.run.smithi171.stdout:(2/12): ceph-prometheus-alerts-19.1.1-326.g8921 92 kB/s | 16 kB 00:00 2024-09-16T04:04:53.580 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:04:53.650 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:04:53.677 INFO:teuthology.orchestra.run.smithi171.stdout:(3/12): ceph-mgr-dashboard-19.1.1-326.g8921c8ef 9.0 MB/s | 3.6 MB 00:00 2024-09-16T04:04:53.685 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:04:53.702 INFO:teuthology.orchestra.run.smithi171.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-16T04:04:53.732 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:04:53.761 INFO:teuthology.orchestra.run.smithi171.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T04:04:53.762 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:04:53.794 INFO:teuthology.orchestra.run.smithi171.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-16T04:04:53.820 INFO:teuthology.orchestra.run.smithi171.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 331 kB/s | 19 kB 00:00 2024-09-16T04:04:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:04:53.862 INFO:teuthology.orchestra.run.smithi171.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.1 MB/s | 144 kB 00:00 2024-09-16T04:04:53.879 INFO:teuthology.orchestra.run.smithi171.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 2.0 MB/s | 31 kB 00:00 2024-09-16T04:04:53.913 INFO:teuthology.orchestra.run.smithi171.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-09-16T04:04:53.988 INFO:teuthology.orchestra.run.smithi171.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-16T04:04:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:04:54.082 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:04:54.134 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:04:54.395 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:04:54.427 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:04:54.738 INFO:teuthology.orchestra.run.smithi171.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 814 kB/s | 863 kB 00:01 2024-09-16T04:04:54.743 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:04:54.743 INFO:teuthology.orchestra.run.smithi171.stdout:Total 4.4 MB/s | 8.7 MB 00:01 2024-09-16T04:04:54.792 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:04:54.810 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:04:54.810 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:04:55.022 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:04:55.023 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:04:55.484 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:04:55.519 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:04:55.567 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:04:55.656 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T04:04:55.699 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T04:04:55.745 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T04:04:55.785 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T04:04:55.877 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T04:04:56.012 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T04:04:56.099 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T04:04:56.150 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T04:04:56.196 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:04:56.236 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 10/12 2024-09-16T04:04:56.723 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:04:56.723 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:04:56.723 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:04:56.723 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:04:56.723 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:04:56.723 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:04:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:04:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:04:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:04:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:04:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:04:56.724 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:04:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:04:57.052 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:04:57.053 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:04:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:04:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:04:57.054 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:57.054 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:04:57.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:04:57.302 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 11/12 2024-09-16T04:04:57.354 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:04:57.846 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:04:57.981 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:04:57.981 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:04:57.982 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout:Install 12 Packages 2024-09-16T04:04:57.983 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:04:57.984 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 38 M 2024-09-16T04:04:57.984 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 212 M 2024-09-16T04:04:57.984 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:04:58.415 INFO:teuthology.orchestra.run.smithi148.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T04:04:58.448 INFO:teuthology.orchestra.run.smithi148.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-16T04:04:58.549 INFO:teuthology.orchestra.run.smithi148.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-16T04:04:58.599 INFO:teuthology.orchestra.run.smithi148.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-09-16T04:04:58.649 INFO:teuthology.orchestra.run.smithi148.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-16T04:04:58.775 INFO:teuthology.orchestra.run.smithi148.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-32 15 MB/s | 7.4 MB 00:00 2024-09-16T04:04:58.776 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 12/12 2024-09-16T04:04:58.776 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9 1/12 2024-09-16T04:04:58.776 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noar 2/12 2024-09-16T04:04:58.776 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9. 3/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T04:04:58.777 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T04:04:59.008 INFO:teuthology.orchestra.run.smithi148.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:04:59.113 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:04:59.114 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:04:59.384 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T04:04:59.475 INFO:teuthology.orchestra.run.smithi148.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-09-16T04:04:59.625 INFO:teuthology.orchestra.run.smithi148.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 250 kB 00:00 2024-09-16T04:04:59.955 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:06 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:00.091 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T04:05:00.092 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:00.093 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:00.094 INFO:teuthology.orchestra.run.smithi171.stdout:====================================================================================== 2024-09-16T04:05:00.094 INFO:teuthology.orchestra.run.smithi171.stdout:Install 12 Packages 2024-09-16T04:05:00.094 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:00.094 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 38 M 2024-09-16T04:05:00.094 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 212 M 2024-09-16T04:05:00.094 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:00.643 INFO:teuthology.orchestra.run.smithi171.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T04:05:00.685 INFO:teuthology.orchestra.run.smithi171.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-16T04:05:00.692 INFO:teuthology.orchestra.run.smithi148.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.4 MB/s | 4.9 MB 00:02 2024-09-16T04:05:00.786 INFO:teuthology.orchestra.run.smithi171.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-16T04:05:00.836 INFO:teuthology.orchestra.run.smithi171.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-16T04:05:00.911 INFO:teuthology.orchestra.run.smithi171.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 512 kB/s | 38 kB 00:00 2024-09-16T04:05:00.926 INFO:teuthology.orchestra.run.smithi148.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.7 MB/s | 5.1 MB 00:01 2024-09-16T04:05:01.129 INFO:teuthology.orchestra.run.smithi171.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-32 12 MB/s | 7.4 MB 00:00 2024-09-16T04:05:01.329 INFO:teuthology.orchestra.run.smithi171.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-09-16T04:05:01.371 INFO:teuthology.orchestra.run.smithi171.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-16T04:05:01.554 INFO:teuthology.orchestra.run.smithi171.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 250 kB 00:00 2024-09-16T04:05:02.113 INFO:teuthology.orchestra.run.smithi171.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-09-16T04:05:02.347 INFO:teuthology.orchestra.run.smithi171.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.0 MB/s | 5.1 MB 00:01 2024-09-16T04:05:03.509 INFO:teuthology.orchestra.run.smithi148.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.2 MB/s | 16 MB 00:03 2024-09-16T04:05:03.510 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:03.510 INFO:teuthology.orchestra.run.smithi148.stdout:Total 6.9 MB/s | 38 MB 00:05 2024-09-16T04:05:03.871 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:03.896 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:03.897 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:04.330 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:04.330 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:05.124 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:05:05.216 INFO:teuthology.orchestra.run.smithi171.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.4 MB/s | 16 MB 00:03 2024-09-16T04:05:05.217 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:05.217 INFO:teuthology.orchestra.run.smithi171.stdout:Total 7.5 MB/s | 38 MB 00:05 2024-09-16T04:05:05.244 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:05.339 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:05.392 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:05:05.576 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:05:05.578 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:05.604 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:05.604 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:05.606 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:05:05.683 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:05:06.047 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:06.048 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:06.322 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:05:06.436 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:05:06.839 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:06.903 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T04:05:06.951 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:07.055 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:07.097 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T04:05:07.284 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T04:05:07.313 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T04:05:07.390 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T04:05:07.661 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:08.054 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T04:05:08.187 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T04:05:08.692 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:05:08.731 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:10.495 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T04:05:10.528 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:11.299 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:11.300 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:05:11.687 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:11.688 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:12.053 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:05:12.629 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:12.762 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:12.763 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:12.763 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:05:12.763 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:12.763 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:12.763 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:05:12.763 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:05:12.764 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout:Install 13 Packages 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 2.2 M 2024-09-16T04:05:12.765 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 26 M 2024-09-16T04:05:12.766 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:13.218 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 12/12 2024-09-16T04:05:13.219 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8 1/12 2024-09-16T04:05:13.219 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T04:05:13.219 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T04:05:13.219 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T04:05:13.219 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T04:05:13.220 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T04:05:13.220 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T04:05:13.220 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T04:05:13.220 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T04:05:13.220 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T04:05:13.220 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T04:05:13.306 INFO:teuthology.orchestra.run.smithi148.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 154 kB/s | 19 kB 00:00 2024-09-16T04:05:13.331 INFO:teuthology.orchestra.run.smithi148.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 172 kB/s | 26 kB 00:00 2024-09-16T04:05:13.357 INFO:teuthology.orchestra.run.smithi148.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 282 kB/s | 49 kB 00:00 2024-09-16T04:05:13.490 INFO:teuthology.orchestra.run.smithi148.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-16T04:05:13.516 INFO:teuthology.orchestra.run.smithi148.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-09-16T04:05:13.549 INFO:teuthology.orchestra.run.smithi148.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-09-16T04:05:13.591 INFO:teuthology.orchestra.run.smithi148.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-16T04:05:13.616 INFO:teuthology.orchestra.run.smithi148.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-16T04:05:13.650 INFO:teuthology.orchestra.run.smithi148.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 143 kB/s | 14 kB 00:00 2024-09-16T04:05:13.675 INFO:teuthology.orchestra.run.smithi148.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-16T04:05:13.701 INFO:teuthology.orchestra.run.smithi148.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-16T04:05:13.703 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T04:05:13.703 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:13.703 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:13.703 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:13.703 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:13.704 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:13.734 INFO:teuthology.orchestra.run.smithi148.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-16T04:05:13.767 INFO:teuthology.orchestra.run.smithi148.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-09-16T04:05:13.770 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:13.770 INFO:teuthology.orchestra.run.smithi148.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-09-16T04:05:13.834 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:13.937 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:13.938 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:14.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-rook 2024-09-16T04:05:14.441 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:14.613 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:05:14.667 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:14.677 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:05:14.741 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:05:14.778 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:05:14.801 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:14.802 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:14.802 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:05:14.802 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:14.802 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:14.802 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T04:05:14.803 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 2.2 M 2024-09-16T04:05:14.804 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 26 M 2024-09-16T04:05:14.805 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:14.819 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:05:14.929 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:05:15.030 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:05:15.087 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:05:15.100 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 296 kB/s | 26 kB 00:00 2024-09-16T04:05:15.125 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 172 kB/s | 19 kB 00:00 2024-09-16T04:05:15.176 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): ceph-mgr-rook-19.1.1-326.g8921c8ef.el9. 304 kB/s | 49 kB 00:00 2024-09-16T04:05:15.234 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T04:05:15.260 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-16T04:05:15.296 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 910 kB/s | 54 kB 00:00 2024-09-16T04:05:15.327 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 478 kB/s | 32 kB 00:00 2024-09-16T04:05:15.352 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-16T04:05:15.378 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-16T04:05:15.404 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-16T04:05:15.437 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.3 MB/s | 90 kB 00:00 2024-09-16T04:05:15.462 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 702 kB/s | 59 kB 00:00 2024-09-16T04:05:15.496 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-09-16T04:05:15.499 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:15.499 INFO:teuthology.orchestra.run.smithi171.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-16T04:05:15.563 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:15.578 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:15.578 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:15.669 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:15.669 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:15.797 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:05:15.852 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:05:15.885 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:05:15.955 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:05:16.190 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:16.361 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T04:05:16.415 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T04:05:16.470 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T04:05:16.508 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T04:05:16.549 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T04:05:16.651 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:05:16.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:05:16.727 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:05:16.727 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:05:16.727 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:05:16.754 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T04:05:16.810 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T04:05:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:05:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:17.074 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:17.074 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:17.075 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:17.349 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:05:17.543 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T04:05:17.592 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T04:05:17.626 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T04:05:17.674 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T04:05:17.687 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:05:17.921 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout:Install 6 Packages 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 7.0 M 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 33 M 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:18.437 INFO:teuthology.orchestra.run.smithi148.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T04:05:18.462 INFO:teuthology.orchestra.run.smithi148.stdout:(2/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 610 kB/s | 146 kB 00:00 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 13/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 1/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T04:05:18.496 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T04:05:18.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T04:05:18.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T04:05:18.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T04:05:18.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T04:05:18.497 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T04:05:18.520 INFO:teuthology.orchestra.run.smithi148.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T04:05:18.546 INFO:teuthology.orchestra.run.smithi148.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 690 kB/s | 58 kB 00:00 2024-09-16T04:05:18.571 INFO:teuthology.orchestra.run.smithi148.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-16T04:05:18.842 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T04:05:18.842 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:18.842 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:18.842 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T04:05:18.843 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T04:05:18.844 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T04:05:18.844 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T04:05:18.844 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T04:05:18.844 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T04:05:18.844 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:18.844 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:19.101 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T04:05:19.682 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:19.763 INFO:teuthology.orchestra.run.smithi148.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-16T04:05:19.766 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:19.766 INFO:teuthology.orchestra.run.smithi148.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-16T04:05:19.817 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:19.968 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T04:05:19.969 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 7.0 M 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 33 M 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:19.970 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:20.079 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-16T04:05:20.146 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-09-16T04:05:20.171 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-16T04:05:20.197 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T04:05:20.289 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): ceph-mgr-cephadm-19.1.1-326.g8921c8ef.el 416 kB/s | 146 kB 00:00 2024-09-16T04:05:20.296 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:20.356 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-16T04:05:20.357 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:20.358 INFO:teuthology.orchestra.run.smithi171.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-16T04:05:20.430 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:20.441 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:20.441 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:20.466 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:05:20.524 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:05:20.541 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:20.541 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:20.849 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:21.023 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T04:05:21.054 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:05:21.083 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T04:05:21.142 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:05:21.208 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:05:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:05:21.610 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T04:05:21.695 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T04:05:21.759 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T04:05:21.788 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:05:21.978 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:05:21.978 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:05:21.978 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:05:21.978 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:05:21.978 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:05:21.978 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:05:22.250 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:22.251 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:22.501 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-fuse 2024-09-16T04:05:22.524 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 6/6 2024-09-16T04:05:22.524 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 1/6 2024-09-16T04:05:22.524 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T04:05:22.524 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T04:05:22.524 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T04:05:22.525 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T04:05:22.807 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T04:05:22.807 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:22.808 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:23.058 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse 2024-09-16T04:05:23.067 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:23.201 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-16T04:05:23.202 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:Install 2 Packages 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 976 k 2024-09-16T04:05:23.203 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 2.9 M 2024-09-16T04:05:23.204 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:23.523 INFO:teuthology.orchestra.run.smithi148.stdout:(1/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 3.3 MB/s | 896 kB 00:00 2024-09-16T04:05:23.625 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:23.760 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:23.760 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-09-16T04:05:23.761 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:23.762 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 976 k 2024-09-16T04:05:23.762 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.9 M 2024-09-16T04:05:23.762 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:23.948 INFO:teuthology.orchestra.run.smithi148.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 116 kB/s | 80 kB 00:00 2024-09-16T04:05:23.949 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:23.949 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.3 MB/s | 976 kB 00:00 2024-09-16T04:05:23.961 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:23.983 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:23.983 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:24.003 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 636 kB/s | 80 kB 00:00 2024-09-16T04:05:24.063 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:24.064 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:24.253 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:24.402 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:05:24.445 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86_6 1.5 MB/s | 896 kB 00:00 2024-09-16T04:05:24.446 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:24.446 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.4 MB/s | 976 kB 00:00 2024-09-16T04:05:24.457 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:24.472 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:05:24.479 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:24.479 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:24.560 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:24.561 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:24.743 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T04:05:24.961 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:05:25.209 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:05:25.209 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:05:25.475 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:25.475 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:25.475 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:05:25.475 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:25.475 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:25.642 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2/2 2024-09-16T04:05:25.642 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/2 2024-09-16T04:05:25.737 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-volume 2024-09-16T04:05:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T04:05:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:25.933 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T04:05:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:25.934 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:26.209 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-volume 2024-09-16T04:05:26.330 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:26.466 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:26.466 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:26.466 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 285 k 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 1.3 M 2024-09-16T04:05:26.467 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:26.694 INFO:teuthology.orchestra.run.smithi148.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-16T04:05:26.694 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:26.695 INFO:teuthology.orchestra.run.smithi148.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T04:05:26.695 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:26.703 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:26.703 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:26.747 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:26.747 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:26.797 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:26.937 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:26.938 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:05:26.939 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:26.939 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 285 k 2024-09-16T04:05:26.939 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 1.3 M 2024-09-16T04:05:26.939 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:26.965 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:27.046 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:27.061 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:27.061 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:27.061 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:05:27.061 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:27.163 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-volume-19.1.1-326.g8921c8ef.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-16T04:05:27.163 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:27.164 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T04:05:27.164 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:27.172 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:27.172 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:27.204 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:27.204 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:27.437 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:27.507 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:27.523 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:27.524 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:27.524 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T04:05:27.524 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:28.065 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:28.065 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:28.065 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:28.065 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:28.065 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:28.065 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:28.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados-devel 2024-09-16T04:05:28.542 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 1/1 2024-09-16T04:05:28.542 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:28.542 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:28.542 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T04:05:28.542 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:28.542 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:28.863 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel 2024-09-16T04:05:28.944 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:29.082 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:29.083 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 127 k 2024-09-16T04:05:29.084 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 456 k 2024-09-16T04:05:29.084 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:29.315 INFO:teuthology.orchestra.run.smithi148.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 548 kB/s | 127 kB 00:00 2024-09-16T04:05:29.316 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:29.317 INFO:teuthology.orchestra.run.smithi148.stdout:Total 544 kB/s | 127 kB 00:00 2024-09-16T04:05:29.317 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:29.322 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:29.322 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:29.391 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:29.391 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:29.451 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:29.546 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:29.592 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:29.592 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:29.592 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 127 k 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 456 k 2024-09-16T04:05:29.593 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:29.656 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:29.781 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-19.1.1-326.g8921c8ef.el9.x86_64. 676 kB/s | 127 kB 00:00 2024-09-16T04:05:29.781 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:29.782 INFO:teuthology.orchestra.run.smithi171.stdout:Total 671 kB/s | 127 kB 00:00 2024-09-16T04:05:29.782 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:29.787 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:29.787 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:29.857 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:29.857 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:30.020 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:30.096 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:30.346 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:30.346 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:30.346 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:30.346 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.346 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:30.346 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:30.531 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:30.629 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs2 2024-09-16T04:05:30.799 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:30.799 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:30.799 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:30.799 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:30.799 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:30.799 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:31.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2 2024-09-16T04:05:31.208 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:31.301 INFO:teuthology.orchestra.run.smithi148.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:31.345 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:31.346 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:31.346 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:31.408 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install libcephfs-devel 2024-09-16T04:05:31.634 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:31.727 INFO:teuthology.orchestra.run.smithi171.stdout:Package libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:31.775 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:31.775 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:31.775 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:31.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs-devel 2024-09-16T04:05:31.981 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:32.117 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:32.118 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 32 k 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 142 k 2024-09-16T04:05:32.119 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:32.327 INFO:teuthology.orchestra.run.smithi148.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 153 kB/s | 32 kB 00:00 2024-09-16T04:05:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:Total 152 kB/s | 32 kB 00:00 2024-09-16T04:05:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:32.331 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:32.331 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:32.344 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:32.345 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:32.414 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:32.435 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:32.550 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:32.550 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:32.551 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:32.552 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:05:32.552 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:32.552 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 32 k 2024-09-16T04:05:32.552 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 142 k 2024-09-16T04:05:32.552 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:32.839 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs-devel-19.1.1-326.g8921c8ef.el9.x86_64 110 kB/s | 32 kB 00:00 2024-09-16T04:05:32.839 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:32.839 INFO:teuthology.orchestra.run.smithi171.stdout:Total 110 kB/s | 32 kB 00:00 2024-09-16T04:05:32.839 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:32.843 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:32.843 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:32.864 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:32.864 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:32.957 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:32.958 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:33.205 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:33.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:33.206 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:33.206 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:33.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:33.206 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:33.379 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librados2 2024-09-16T04:05:33.446 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:33.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2 2024-09-16T04:05:33.948 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:34.039 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:34.082 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:34.083 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:34.083 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:34.146 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install librbd1 2024-09-16T04:05:34.461 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:34.553 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:34.596 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:34.597 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:34.597 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:34.667 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1 2024-09-16T04:05:34.719 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:34.811 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:34.854 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:34.855 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:34.855 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:34.916 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rados 2024-09-16T04:05:35.233 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:35.324 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:35.367 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:35.368 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:35.368 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:35.427 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rados 2024-09-16T04:05:35.484 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:35.576 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:35.620 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:35.621 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:35.621 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:35.680 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rgw 2024-09-16T04:05:35.993 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:36.084 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:36.127 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:36.128 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:36.192 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rgw 2024-09-16T04:05:36.247 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:36.337 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:36.381 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:36.383 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:36.383 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:36.449 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-cephfs 2024-09-16T04:05:36.762 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:36.853 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:36.896 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:36.897 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:36.897 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:36.959 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-cephfs 2024-09-16T04:05:37.019 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:37.111 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:37.154 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:37.155 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:37.155 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:37.214 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install python3-rbd 2024-09-16T04:05:37.525 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:37.617 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:37.660 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:37.661 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:37.661 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:37.730 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rbd 2024-09-16T04:05:37.777 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:37.870 INFO:teuthology.orchestra.run.smithi148.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:37.913 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:37.914 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2024-09-16T04:05:37.914 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:37.975 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-fuse 2024-09-16T04:05:38.324 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:38.422 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 is already installed. 2024-09-16T04:05:38.468 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:38.469 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-09-16T04:05:38.469 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:38.542 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:38.558 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse 2024-09-16T04:05:38.677 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:38.678 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:38.679 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:05:38.679 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:38.679 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 85 k 2024-09-16T04:05:38.679 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 231 k 2024-09-16T04:05:38.679 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:38.880 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 425 kB/s | 85 kB 00:00 2024-09-16T04:05:38.880 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:38.880 INFO:teuthology.orchestra.run.smithi148.stdout:Total 422 kB/s | 85 kB 00:00 2024-09-16T04:05:38.881 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:38.887 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:38.888 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:38.951 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:38.951 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:39.096 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:39.146 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:39.202 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:39.282 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:39.283 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:39.283 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-09-16T04:05:39.283 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:39.283 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 85 k 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 231 k 2024-09-16T04:05:39.284 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:39.461 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-19.1.1-326.g8921c8ef.el9.x86_64.rpm 481 kB/s | 85 kB 00:00 2024-09-16T04:05:39.462 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:39.462 INFO:teuthology.orchestra.run.smithi171.stdout:Total 476 kB/s | 85 kB 00:00 2024-09-16T04:05:39.463 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:39.470 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:39.470 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:39.534 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:39.534 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:39.601 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:39.677 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:39.800 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:39.865 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:39.865 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:39.866 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:39.866 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:40.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-mirror 2024-09-16T04:05:40.185 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:40.457 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:40.457 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:40.457 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:40.457 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:40.457 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:40.457 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:40.721 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror 2024-09-16T04:05:40.758 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:40.898 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repo Size 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:40.899 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 3.1 M 2024-09-16T04:05:40.900 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 13 M 2024-09-16T04:05:40.900 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:41.234 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-09-16T04:05:41.235 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:41.235 INFO:teuthology.orchestra.run.smithi148.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-16T04:05:41.235 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:41.244 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:41.244 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:41.302 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:41.325 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:41.325 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:41.439 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:41.440 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:41.440 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-09-16T04:05:41.440 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.1 M 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 13 M 2024-09-16T04:05:41.441 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:41.770 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:41.781 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-19.1.1-326.g8921c8ef.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-16T04:05:41.782 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:41.782 INFO:teuthology.orchestra.run.smithi171.stdout:Total 9.1 MB/s | 3.1 MB 00:00 2024-09-16T04:05:41.782 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:41.791 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:41.791 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:41.852 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:41.852 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:41.852 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:05:41.852 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:05:41.853 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:05:41.853 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:41.871 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:41.872 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:42.319 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:42.382 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:42.407 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:42.407 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T04:05:42.407 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T04:05:42.407 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:05:42.408 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T04:05:42.408 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:42.919 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:42.919 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:42.920 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:42.920 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:42.920 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:42.920 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:43.163 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install rbd-nbd 2024-09-16T04:05:43.538 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:43.538 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:43.538 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:43.538 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:43.538 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:43.539 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:43.747 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 04:03:52 AM UTC. 2024-09-16T04:05:43.775 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd 2024-09-16T04:05:43.881 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-16T04:05:43.882 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:43.882 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-09-16T04:05:43.882 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:43.882 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-16T04:05:43.882 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:05:43.882 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout:Install 1 Package 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 170 k 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 494 k 2024-09-16T04:05:43.883 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-16T04:05:44.062 INFO:teuthology.orchestra.run.smithi148.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 951 kB/s | 170 kB 00:00 2024-09-16T04:05:44.063 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:44.063 INFO:teuthology.orchestra.run.smithi148.stdout:Total 944 kB/s | 170 kB 00:00 2024-09-16T04:05:44.063 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-16T04:05:44.071 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-16T04:05:44.071 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-16T04:05:44.134 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-16T04:05:44.135 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-16T04:05:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-16T04:05:44.365 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 04:03:53 AM UTC. 2024-09-16T04:05:44.393 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:44.503 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-09-16T04:05:44.504 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:44.504 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-09-16T04:05:44.504 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:44.504 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-09-16T04:05:44.504 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T04:05:44.504 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 170 k 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 494 k 2024-09-16T04:05:44.505 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-09-16T04:05:44.709 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_64.rpm 833 kB/s | 170 kB 00:00 2024-09-16T04:05:44.710 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-09-16T04:05:44.710 INFO:teuthology.orchestra.run.smithi171.stdout:Total 828 kB/s | 170 kB 00:00 2024-09-16T04:05:44.710 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-09-16T04:05:44.718 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-09-16T04:05:44.718 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-09-16T04:05:44.783 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-09-16T04:05:44.784 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-09-16T04:05:44.804 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:44.937 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-09-16T04:05:45.032 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:45.080 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:45.080 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:45.080 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-16T04:05:45.080 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:45.081 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:45.081 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-16T04:05:45.318 DEBUG:teuthology.parallel:result is None 2024-09-16T04:05:45.476 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:45.743 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 1/1 2024-09-16T04:05:45.743 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:45.743 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-09-16T04:05:45.743 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T04:05:45.743 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:45.743 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-09-16T04:05:45.988 DEBUG:teuthology.parallel:result is None 2024-09-16T04:05:45.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:05:46.131 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:05:46.153 INFO:teuthology.orchestra.run.smithi148.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:05:46.153 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:05:46.153 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:05:46.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T04:05:46.291 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T04:05:46.313 INFO:teuthology.orchestra.run.smithi171.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T04:05:46.314 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T04:05:46.314 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T04:05:46.315 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T04:05:46.316 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:46.316 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:05:46.342 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:46.342 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T04:05:46.380 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T04:05:46.380 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:46.380 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:05:46.407 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:05:46.472 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:46.472 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T04:05:46.497 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T04:05:46.562 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T04:05:46.562 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:46.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:05:46.587 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:05:46.652 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:46.653 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T04:05:46.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T04:05:46.746 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T04:05:46.746 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:46.746 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:05:46.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:05:46.835 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:46.835 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T04:05:46.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T04:05:46.925 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T04:05:47.006 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T04:05:47.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2024-09-16T04:05:47.009 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /var/log/ceph 2024-09-16T04:05:47.034 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T04:05:47.034 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:05:47.074 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T04:05:47.099 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T04:05:47.099 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:05:47.139 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T04:05:47.172 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T04:05:47.173 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T04:05:47.173 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_mon_workunits}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905963', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'mon_bind_msgr2': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi148.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECk5igt4xZa57MDBWGdZIJdDOVMPCSQbm20/OCYsqcqs24sJu0e5gOFYWZSzJrwZJ89yX4/CSm/Z0XxPbne6cM=', 'smithi171.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZqPdnqeT/Y0qONxHvb/cZtk+OnW4aylbI8kkqGwfu7bIQNPQnbr4BzHmzoW+YMHlaA/CA1xNmqOS7he87L6hI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T04:05:47.173 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:05:47.204 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T04:05:47.220 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:05:47.260 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T04:05:47.284 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:47.285 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:05:47.338 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:05:47.339 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 04:05:42.088511502 +0000 2024-09-16T04:05:47.394 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 04:03:14.404034159 +0000 2024-09-16T04:05:47.395 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 04:03:14.404034159 +0000 2024-09-16T04:05:47.395 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 04:03:14.404034159 +0000 2024-09-16T04:05:47.395 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:05:47.458 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T04:05:47.458 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T04:05:47.458 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000266389 s, 1.9 MB/s 2024-09-16T04:05:47.459 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:05:47.516 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-09-16T04:05:47.571 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 04:05:42.088511502 +0000 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 04:03:14.736025838 +0000 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 04:03:14.736025838 +0000 2024-09-16T04:05:47.572 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 04:03:14.736025838 +0000 2024-09-16T04:05:47.573 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:05:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T04:05:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T04:05:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000245034 s, 2.1 MB/s 2024-09-16T04:05:47.636 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:05:47.693 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 04:05:42.089511478 +0000 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 04:03:15.057017793 +0000 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 04:03:15.057017793 +0000 2024-09-16T04:05:47.748 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 04:03:15.057017793 +0000 2024-09-16T04:05:47.749 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:05:47.811 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T04:05:47.811 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T04:05:47.811 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000207003 s, 2.5 MB/s 2024-09-16T04:05:47.812 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:05:47.870 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-09-16T04:05:47.925 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:05:47.925 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:47.925 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T04:05:47.925 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:47.925 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:47.925 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-16 04:05:42.089511478 +0000 2024-09-16T04:05:47.926 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-16 04:03:15.392009398 +0000 2024-09-16T04:05:47.926 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-16 04:03:15.392009398 +0000 2024-09-16T04:05:47.926 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-16 04:03:15.392009398 +0000 2024-09-16T04:05:47.926 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:05:47.989 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-16T04:05:47.989 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-16T04:05:47.989 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.00019362 s, 2.6 MB/s 2024-09-16T04:05:47.990 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:05:48.047 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:05:48.047 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:48.047 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T04:05:48.065 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T04:05:48.065 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-09-16T04:05:48.121 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T04:05:48.121 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-09-16 04:05:42.663366271 +0000 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-09-16 04:03:12.838732367 +0000 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-09-16 04:03:12.838732367 +0000 2024-09-16T04:05:48.122 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-09-16 04:03:12.838732367 +0000 2024-09-16T04:05:48.122 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T04:05:48.187 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-09-16T04:05:48.187 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-09-16T04:05:48.187 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000234666 s, 2.2 MB/s 2024-09-16T04:05:48.188 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T04:05:48.246 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-09-16T04:05:48.303 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T04:05:48.303 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:48.303 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T04:05:48.303 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:48.303 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:48.304 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-09-16 04:05:42.664366252 +0000 2024-09-16T04:05:48.304 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-09-16 04:03:13.172725869 +0000 2024-09-16T04:05:48.304 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-09-16 04:03:13.172725869 +0000 2024-09-16T04:05:48.304 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-09-16 04:03:13.172725869 +0000 2024-09-16T04:05:48.304 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T04:05:48.368 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-09-16T04:05:48.368 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-09-16T04:05:48.368 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000220008 s, 2.3 MB/s 2024-09-16T04:05:48.369 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T04:05:48.426 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-09-16T04:05:48.482 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T04:05:48.482 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-09-16 04:05:42.664366252 +0000 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-09-16 04:03:13.510719294 +0000 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-09-16 04:03:13.510719294 +0000 2024-09-16T04:05:48.483 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-09-16 04:03:13.510719294 +0000 2024-09-16T04:05:48.483 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T04:05:48.546 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-09-16T04:05:48.547 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-09-16T04:05:48.547 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000213412 s, 2.4 MB/s 2024-09-16T04:05:48.548 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T04:05:48.605 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-09-16T04:05:48.661 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-09-16 04:05:42.664366252 +0000 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-09-16 04:03:13.836712951 +0000 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-09-16 04:03:13.836712951 +0000 2024-09-16T04:05:48.662 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-09-16 04:03:13.836712951 +0000 2024-09-16T04:05:48.663 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T04:05:48.725 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-09-16T04:05:48.725 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-09-16T04:05:48.725 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000190136 s, 2.7 MB/s 2024-09-16T04:05:48.726 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T04:05:48.783 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:05:48.783 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T04:05:48.783 INFO:tasks.ceph:Generating config... 2024-09-16T04:05:48.784 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T04:05:48.784 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T04:05:48.784 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T04:05:48.784 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T04:05:48.785 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T04:05:48.785 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T04:05:48.785 INFO:tasks.ceph:[global] ms type = async 2024-09-16T04:05:48.785 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T04:05:48.785 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T04:05:48.785 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T04:05:48.785 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T04:05:48.785 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T04:05:48.785 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T04:05:48.786 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-16T04:05:48.786 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T04:05:48.786 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-16T04:05:48.786 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T04:05:48.786 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T04:05:48.787 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T04:05:48.788 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T04:05:48.788 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T04:05:48.788 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T04:05:48.788 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T04:05:48.788 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T04:05:48.840 INFO:teuthology.orchestra.run.smithi148.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T04:05:48.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T04:05:48.886 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:05:48.952 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.148:6789]'), ('mon.c', '[v1:172.21.15.148:6790]'), ('mon.b', '[v1:172.21.15.171:6789]')] 2024-09-16T04:05:48.953 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.148:6789],[v1:172.21.15.148:6790],[v1:172.21.15.171:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T04:05:48.954 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:48.954 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:05:49.008 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.148:6789]' --addv c '[v1:172.21.15.148:6790]' --addv b '[v1:172.21.15.171:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:49.095 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: generated fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:setting min_mon_release = quincy 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:epoch 0 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:min_mon_release 17 (quincy) 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:election_strategy: 1 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:05:49.096 INFO:teuthology.orchestra.run.smithi148.stdout:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:05:49.097 INFO:teuthology.orchestra.run.smithi148.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T04:05:49.098 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T04:05:49.114 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bc50e21b-589f-47d0-85d5-a865de7d0d68... 2024-09-16T04:05:49.115 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:05:49.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T04:05:49.196 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2024-09-16T04:05:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2024-09-16T04:05:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:05:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2024-09-16T04:05:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.196 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2024-09-16T04:05:49.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:05:49.198 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2024-09-16T04:05:49.199 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = "[v1:172.21.15.148:6789],[v1:172.21.15.148:6790],[v1:172.21.15.171:6789]" 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: mon client directed command retry = 5 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: mon election default strategy = 1 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: ms bind msgr2 = False 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: ms inject socket failures = 5000 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: ms type = async 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.200 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug trim objects = true 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.201 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.202 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore block size = 96636764160 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression algorithm = zlib 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression mode = aggressive 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore fsck on mount = True 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore zero block detection = True 2024-09-16T04:05:49.203 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluefs = 20 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluestore = 20 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 20 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: debug rocksdb = 10 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd full ratio = 0.9 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify cached snaps = True 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify missing on start = True 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock override recovery settings = True 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: osd objectstore = bluestore 2024-09-16T04:05:49.204 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue = debug_random 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: osd op queue cut off = debug_random 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2024-09-16T04:05:49.205 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:05:49.206 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: # 1m isn't quite enough 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon min osdmap epochs = 25 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon osdmap full prune interval = 2 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon osdmap full prune min = 15 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon osdmap full prune txsize = 2 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T04:05:49.207 INFO:teuthology.orchestra.run.smithi148.stdout: mon scrub inject missing keys = 0.05 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: mon scrub interval = 300 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: paxos min = 10 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: paxos service trim min = 5 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: paxos trim min = 10 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: debug monc = 20 2024-09-16T04:05:49.208 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.c] 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.b] 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi171.stdout:[global] 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi171.stdout: chdir = "" 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi171.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T04:05:49.209 INFO:teuthology.orchestra.run.smithi171.stdout: auth supported = cephx 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: filestore xattr use omap = true 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: mon clock drift allowed = 1.000 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: osd crush chooseleaf type = 0 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: auth debug = true 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on old message = true 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: ms die on bug = true 2024-09-16T04:05:49.210 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon pg warn max object skew = 0 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default size = 2 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow primary affinity = true 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd allow pg remap = true 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on legacy crush tunables = false 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on crush straw calc version zero = false 2024-09-16T04:05:49.211 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on no sortbitwise = false 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on osd down out interval zero = false 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon warn on too few osds = false 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon_allow_pool_size_one = true 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: osd default data pool replay window = 5 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon allow pool delete = true 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.212 INFO:teuthology.orchestra.run.smithi171.stdout: mon cluster log file level = debug 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: debug asserts on shutdown = true 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: mon health detail to clog = false 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: mon host = "[v1:172.21.15.148:6789],[v1:172.21.15.148:6790],[v1:172.21.15.171:6789]" 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: mon client directed command retry = 5 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: mon election default strategy = 1 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: ms bind msgr2 = False 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: ms inject socket failures = 5000 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: ms type = async 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: fsid = bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout:[osd] 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: osd journal size = 100 2024-09-16T04:05:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub load threshold = 5.0 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd scrub max interval = 600 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock profile = high_recovery_ops 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd recover clone overlap = true 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd recovery max chunk = 1048576 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug shutdown = true 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug op order = true 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify stray on activate = true 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug trim objects = true 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: osd open classes on start = true 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug pg log writeout = true 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: osd deep scrub update digest min age = 30 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: osd map max advance = 10 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: journal zero on create = true 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: filestore ondisk finisher threads = 3 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: filestore apply finisher threads = 3 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: bdev debug aio = true 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug misdirected ops = true 2024-09-16T04:05:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore block size = 96636764160 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression algorithm = zlib 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore compression mode = aggressive 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore fsck on mount = True 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: bluestore zero block detection = True 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluefs = 20 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: debug bluestore = 20 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: debug osd = 20 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: debug rocksdb = 10 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd full ratio = 0.9 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd nearfull ratio = 0.8 2024-09-16T04:05:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify cached snaps = True 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: osd debug verify missing on start = True 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: osd failsafe full ratio = 0.95 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: osd mclock override recovery settings = True 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: osd objectstore = bluestore 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: osd op queue = debug_random 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: osd op queue cut off = debug_random 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr] 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: debug mgr = 20 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:05:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: mgr/telemetry/nag = false 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout:[mon] 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: debug mon = 20 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: debug paxos = 20 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: debug auth = 20 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: mon data avail warn = 5 2024-09-16T04:05:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: mon mgr mkfs grace = 240 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min pgs per osd = 4 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd reporter subtree level = osd 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: mon osd prime pg temp = true 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: mon reweight min bytes per osd = 10 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T04:05:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: # 1m isn't quite enough 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon_down_mkfs_grace = 2m 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon_warn_on_filestore_osds = false 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon min osdmap epochs = 25 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune interval = 2 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune min = 15 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon osdmap full prune txsize = 2 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon scrub inject missing keys = 0.05 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: mon scrub interval = 300 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: paxos min = 10 2024-09-16T04:05:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: paxos service trim min = 5 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: paxos trim min = 10 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout:[client] 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: rgw cache enabled = true 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable ops log = true 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: rgw enable usage log = true 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: debug monc = 20 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: debug ms = 1 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.a] 2024-09-16T04:05:49.221 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.c] 2024-09-16T04:05:49.222 INFO:teuthology.orchestra.run.smithi171.stdout:[mon.b] 2024-09-16T04:05:49.223 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T04:05:49.223 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T04:05:49.317 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T04:05:49.317 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:49.317 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T04:05:49.333 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:49.333 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T04:05:49.388 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:05:49.388 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:49.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:05:49.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:05:49.462 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:05:49.462 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:49.518 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:05:49.518 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:49.518 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T04:05:49.518 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T04:05:49.554 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:05:49.554 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:49.609 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T04:05:49.609 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T04:05:49.609 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:05:49.707 INFO:teuthology.orchestra.run.smithi171.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T04:05:49.709 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T04:05:49.710 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T04:05:49.710 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T04:05:49.750 INFO:teuthology.orchestra.run.smithi171.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T04:05:49.763 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T04:05:49.763 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi171.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T04:05:49.764 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T04:05:49.789 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:05:49.789 INFO:tasks.ceph:role: osd.0 2024-09-16T04:05:49.789 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:05:49.789 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:05:49.861 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:05:49.862 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:05:49.862 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:05:49.862 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:05:50.381 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-09-16T04:05:50.383 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-09-16T04:05:50.383 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T04:05:50.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T04:05:50.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T04:05:50.673 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:05:50.673 INFO:tasks.ceph:role: osd.1 2024-09-16T04:05:50.673 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:05:50.673 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:05:50.745 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:05:50.746 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:05:51.201 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-09-16T04:05:51.203 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-09-16T04:05:51.203 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T04:05:51.293 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T04:05:51.322 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T04:05:51.388 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T04:05:51.388 INFO:tasks.ceph:role: osd.2 2024-09-16T04:05:51.388 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:05:51.389 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:05:51.461 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:05:51.461 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:05:51.461 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:05:51.461 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:05:51.462 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:05:51.462 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:05:51.462 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:05:51.462 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:05:51.462 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:05:51.462 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:05:51.943 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-09-16T04:05:51.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-09-16T04:05:51.945 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T04:05:52.080 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T04:05:52.113 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:52.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:52.207+0000 7f7f7d820640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T04:05:52.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:52.212+0000 7f7f7d820640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T04:05:52.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:52.212+0000 7f7f7d820640 -1 bdev(0x5617d437d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T04:05:52.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:52.212+0000 7f7f7d820640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T04:05:55.369 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T04:05:55.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:55.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:55.487+0000 7fb15831e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T04:05:55.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:55.487+0000 7fb15831e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T04:05:55.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:55.487+0000 7fb15831e640 -1 bdev(0x55e204195800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T04:05:55.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:55.487+0000 7fb15831e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T04:05:58.645 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T04:05:58.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:05:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:58.763+0000 7f4e11460640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T04:05:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:58.763+0000 7f4e11460640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T04:05:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:58.763+0000 7f4e11460640 -1 bdev(0x563ea984f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T04:05:58.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:05:58.763+0000 7f4e11460640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T04:06:01.924 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T04:06:01.952 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:01.983 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:06:01.984 INFO:tasks.ceph:role: osd.3 2024-09-16T04:06:01.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:06:01.984 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T04:06:02.061 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:02.061 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:02.062 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:02.570 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-09-16T04:06:02.573 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-09-16T04:06:02.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:02.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:02.718 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:02.786 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:06:02.786 INFO:tasks.ceph:role: osd.4 2024-09-16T04:06:02.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:06:02.787 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T04:06:02.860 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:02.861 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:03.329 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-09-16T04:06:03.331 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-09-16T04:06:03.332 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:03.416 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:03.445 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:03.510 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T04:06:03.510 INFO:tasks.ceph:role: osd.5 2024-09-16T04:06:03.510 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:06:03.510 DEBUG:teuthology.orchestra.run.smithi171:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi171.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi171.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi171.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi171.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi171.stdout: = sunit=0 swidth=0 blks 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi171.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi171.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi171.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi171.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T04:06:04.069 INFO:teuthology.orchestra.run.smithi171.stdout:Discarding blocks...Done. 2024-09-16T04:06:04.071 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi171.front.sepia.ceph.com -o noatime 2024-09-16T04:06:04.071 DEBUG:teuthology.orchestra.run.smithi171:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:04.194 DEBUG:teuthology.orchestra.run.smithi171:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:04.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:04.324 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:04.321+0000 7f5255264640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T04:06:04.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:04.327+0000 7f5255264640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T04:06:04.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:04.327+0000 7f5255264640 -1 bdev(0x5601e1b51800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:04.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:04.327+0000 7f5255264640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T04:06:07.738 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T04:06:07.769 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:07.858 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:07.855+0000 7fe191a18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T04:06:07.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:07.856+0000 7fe191a18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T04:06:07.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:07.856+0000 7fe191a18640 -1 bdev(0x55edcae9d800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:07.859 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:07.856+0000 7fe191a18640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T04:06:10.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T04:06:10.801 DEBUG:teuthology.orchestra.run.smithi171:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:10.891 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:10.888+0000 7f3f857db640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T04:06:10.891 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:10.888+0000 7f3f857db640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T04:06:10.891 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:10.888+0000 7f3f857db640 -1 bdev(0x55d1c461f800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T04:06:10.891 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:10.888+0000 7f3f857db640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T04:06:14.297 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T04:06:14.324 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T04:06:14.324 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:06:14.324 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T04:06:14.354 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:06:14.354 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T04:06:14.423 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:06:14.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T04:06:14.492 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:14.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T04:06:14.524 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:14.524 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T04:06:14.590 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:14.590 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T04:06:14.655 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:14.655 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T04:06:14.720 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:14.720 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T04:06:14.776 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T04:06:14.776 DEBUG:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:06:14.779 DEBUG:teuthology.orchestra.run.smithi171:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCgrudmXAdlDBAAC7TaTEprkwaRrcV4iTy0Tw== 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCjrudm4m8ZHRAAg/nV9FyeKHfr2kL2k8cRzA== 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCmrudmmEiMLRAAxNidzU9a5Y638I1NmyQLdQ== 2024-09-16T04:06:14.820 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCdrudmIj0WKhAAxMe2wcLApk5jDOeDTd3rJw== 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.3] 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCsrudmn01HExAAwl2xYR41aphSBp1IRlLZPQ== 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.4] 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCvrudm+7IcMxAANeQrvz5N+5lH6kWgZxGctg== 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.5] 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCyrudmmJQLNRAAhpsaN8XtatbGdz3PFVZ30A== 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2024-09-16T04:06:14.821 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQCdrudmprKwLBAAb1Ftx7JbyZanG/iHRVBuog== 2024-09-16T04:06:14.839 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.0] 2024-09-16T04:06:14.839 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCgrudmXAdlDBAAC7TaTEprkwaRrcV4iTy0Tw== 2024-09-16T04:06:14.839 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.1] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCjrudm4m8ZHRAAg/nV9FyeKHfr2kL2k8cRzA== 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.2] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCmrudmmEiMLRAAxNidzU9a5Y638I1NmyQLdQ== 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout:[mgr.x] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCdrudmIj0WKhAAxMe2wcLApk5jDOeDTd3rJw== 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.3] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCsrudmn01HExAAwl2xYR41aphSBp1IRlLZPQ== 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.4] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCvrudm+7IcMxAANeQrvz5N+5lH6kWgZxGctg== 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout:[osd.5] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCyrudmmJQLNRAAhpsaN8XtatbGdz3PFVZ30A== 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout:[client.0] 2024-09-16T04:06:14.840 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCdrudmprKwLBAAb1Ftx7JbyZanG/iHRVBuog== 2024-09-16T04:06:14.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:14.865 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:14.932 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:14.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:14.988 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:14.990 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.041 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:06:15.044 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T04:06:15.095 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.097 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.148 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.201 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.204 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T04:06:15.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:06:15.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T04:06:15.310 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T04:06:15.310 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T04:06:15.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:06:15.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T04:06:15.455 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T04:06:15.521 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:06:15.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T04:06:15.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T04:06:15.667 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T04:06:15.764 DEBUG:teuthology.orchestra.run.smithi171:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T04:06:15.788 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:15.791 DEBUG:teuthology.orchestra.run.smithi171:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T04:06:15.843 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T04:06:15.844 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:06:15.844 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:06:15.846 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:06:15.846 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:06:15.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:06:15.848 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:06:15.848 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:06:15.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:06:15.886 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:06:15.886 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T04:06:15.886 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T04:06:15.886 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T04:06:15.888 INFO:tasks.ceph.mgr.x:Started 2024-09-16T04:06:15.889 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T04:06:15.889 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T04:06:15.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.987+0000 7f0110b45640 10 monclient: get_monmap_and_config 2024-09-16T04:06:15.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.987+0000 7f0110b45640 10 monclient: build_initial_monmap 2024-09-16T04:06:15.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.987+0000 7f0110b45640 10 monclient: monmap: 2024-09-16T04:06:15.992 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:15.992 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:15.992 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:15.988086+0000 2024-09-16T04:06:15.992 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:15.988086+0000 2024-09-16T04:06:15.993 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:15.993 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:15.993 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:15.993 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:15.993 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:15.993 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:15.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.987+0000 7f0110b45640 1 Processor -- start 2024-09-16T04:06:15.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.987+0000 7f0110b45640 1 -- start start 2024-09-16T04:06:15.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.987+0000 7f0110b45640 10 monclient: init 2024-09-16T04:06:15.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:15.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:15.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient(hunting): picked mon.noname-c con 0x7f010c0a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:15.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient(hunting): picked mon.noname-b con 0x7f010c0a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:15.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient(hunting): picked mon.noname-a con 0x7f010c13d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:15.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0a0590 con 0x7f010c13d3c0 2024-09-16T04:06:15.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0a1270 con 0x7f010c0a3a90 2024-09-16T04:06:15.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0a4fd0 con 0x7f010c0a4580 2024-09-16T04:06:15.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:15.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f0110b45640 10 monclient(hunting): authenticate will time out at 796.613493s 2024-09-16T04:06:15.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f010ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f010c13d3c0 0x7f010c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48500/0 (socket says 172.21.15.148:48500) 2024-09-16T04:06:15.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:15.988+0000 7f010ad9d640 1 -- 172.21.15.148:0/2787982048 learned_addr learned my addr 172.21.15.148:0/2787982048 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): tick 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 monclient(hunting): continuing hunt 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 -- 172.21.15.148:0/2787982048 >> v1:172.21.15.148:6790/0 conn(0x7f010c0a4580 legacy=0x7f010c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 -- 172.21.15.148:0/2787982048 >> v1:172.21.15.171:6789/0 conn(0x7f010c0a3a90 legacy=0x7f010c0a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:18.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 -- 172.21.15.148:0/2787982048 >> v1:172.21.15.148:6789/0 conn(0x7f010c13d3c0 legacy=0x7f010c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f00ec0036e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f00ec007370 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f010c0a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 -- 172.21.15.148:0/2787982048 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c06e170 con 0x7f00ec0036e0 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 -- 172.21.15.148:0/2787982048 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00ec00f890 con 0x7f010c0a3a90 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 1 -- 172.21.15.148:0/2787982048 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00ec010ac0 con 0x7f00ec007370 2024-09-16T04:06:18.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:18.988+0000 7f00fbfff640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:21.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.160+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954105874 0 0) 0x7f010c06e170 con 0x7f00ec0036e0 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.160+0000 7f010959a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.160+0000 7f010959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.160+0000 7f010959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.160+0000 7f010959a640 10 monclient(hunting): my global_id is 4098 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.160+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00f4003520 con 0x7f00ec0036e0 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836023198 0 0) 0x7f00ec010ac0 con 0x7f00ec007370 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient(hunting): my global_id is 4103 2024-09-16T04:06:21.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c06e170 con 0x7f00ec007370 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 454280700 0 0) 0x7f00f4003520 con 0x7f00ec0036e0 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 >> v1:172.21.15.148:6790/0 conn(0x7f00ec007370 legacy=0x7f00ec009800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 >> v1:172.21.15.171:6789/0 conn(0x7f010c0a3a90 legacy=0x7f00ec00d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: _finish_hunting 0 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 monclient: found mon.noname-a 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ec011d50 con 0x7f00ec0036e0 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: _finish_auth 0 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: _check_auth_tickets 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f00fc001650 con 0x7f00ec0036e0 2024-09-16T04:06:21.166 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: dump: 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.167 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f0110b45640 5 monclient: authenticate success, global_id 4098 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f0110b45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f00fc0018f0 con 0x7f00ec0036e0 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 1 -- 172.21.15.148:0/2787982048 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f00fc001d00 con 0x7f00ec0036e0 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f0110b45640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f0110b45640 4 set_mon_vals no callback set 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.161+0000 7f010959a640 10 monclient: dump: 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.168 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 10 monclient: shutdown 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 1 -- 172.21.15.148:0/2787982048 >> v1:172.21.15.148:6789/0 conn(0x7f00ec0036e0 legacy=0x7f00ec005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 1 -- 172.21.15.148:0/2787982048 shutdown_connections 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 1 -- 172.21.15.148:0/2787982048 >> 172.21.15.148:0/2787982048 conn(0x7f010c0a9a40 msgr2=0x7f010c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 1 -- 172.21.15.148:0/2787982048 shutdown_connections 2024-09-16T04:06:21.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.162+0000 7f0110b45640 1 -- 172.21.15.148:0/2787982048 wait complete. 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.163+0000 7f0110b45640 10 monclient: build_initial_monmap 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.163+0000 7f0110b45640 10 monclient: monmap: 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:21.174 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 1 Processor -- start 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 1 -- start start 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 10 monclient: init 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 10 monclient(hunting): picked mon.noname-b con 0x7f00ec0036e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 10 monclient(hunting): picked mon.noname-c con 0x7f00ec007370 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 10 monclient(hunting): picked mon.noname-a con 0x7f010c0a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:21.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c135120 con 0x7f010c0a3a90 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.164+0000 7f0110b45640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c13b780 con 0x7f00ec0036e0 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.165+0000 7f0110b45640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0a7f10 con 0x7f00ec007370 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.165+0000 7f0109d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f00ec007370 0x7f010c1d0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:58854/0 (socket says 172.21.15.148:58854) 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.165+0000 7f0109d9b640 1 -- 172.21.15.148:0/684894309 learned_addr learned my addr 172.21.15.148:0/684894309 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.165+0000 7f0110b45640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.165+0000 7f0110b45640 10 monclient(hunting): authenticate will time out at 801.790142s 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862877303 0 0) 0x7f010c0a7f10 con 0x7f00ec007370 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): my global_id is 4109 2024-09-16T04:06:21.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00ec005310 con 0x7f00ec007370 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 577711603 0 0) 0x7f00ec005310 con 0x7f00ec007370 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f010c0a7f10 con 0x7f00ec007370 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01000018b0 con 0x7f00ec007370 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): dump: 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:21.177 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1403257554 0 0) 0x7f010c135120 con 0x7f010c0a3a90 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): my global_id is 4107 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00ec005310 con 0x7f010c0a3a90 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1965790808 0 0) 0x7f00ec005310 con 0x7f010c0a3a90 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f010c135120 con 0x7f010c0a3a90 2024-09-16T04:06:21.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0104001a20 con 0x7f010c0a3a90 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.166+0000 7f00faffd640 10 monclient(hunting): dump: 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:21.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2850848779 0 0) 0x7f010c135120 con 0x7f010c0a3a90 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 >> v1:172.21.15.148:6790/0 conn(0x7f00ec007370 legacy=0x7f010c1d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 >> v1:172.21.15.171:6789/0 conn(0x7f00ec0036e0 legacy=0x7f010c134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 monclient: found mon.a 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c0a0280 con 0x7f010c0a3a90 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 5 monclient: authenticate success, global_id 4107 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 10 monclient: _renew_subs 2024-09-16T04:06:21.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f010c1203b0 con 0x7f010c0a3a90 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 10 monclient: _renew_subs 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f010c1cc4b0 con 0x7f010c0a3a90 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0104001f30 con 0x7f010c0a3a90 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0104004d00 con 0x7f010c0a3a90 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 10 monclient: dump: 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.181 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f01040052b0 con 0x7f010c0a3a90 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f0108d99640 4 set_mon_vals no callback set 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.167+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 458503526 0 0) 0x7f01040058e0 con 0x7f010c0a3a90 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.168+0000 7f0110b45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.168+0000 7f0110b45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.168+0000 7f0110b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.168+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c09c8e0 con 0x7f010c0a3a90 2024-09-16T04:06:21.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.170+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f010c0a0590 con 0x7f010c0a3a90 2024-09-16T04:06:21.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.170+0000 7f00faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.170+0000 7f00faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:21.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.236+0000 7f0110b45640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T04:06:21.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.236+0000 7f0110b45640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T04:06:21.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.236+0000 7f0110b45640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.236+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f010c062790 con 0x7f010c0a3a90 2024-09-16T04:06:21.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.239+0000 7f00faffd640 1 -- 172.21.15.148:0/684894309 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f0104005df0 con 0x7f010c0a3a90 2024-09-16T04:06:21.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.239+0000 7f00faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.239+0000 7f00faffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:adjusted tunables profile to default 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 10 monclient: shutdown 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 >> v1:172.21.15.148:6789/0 conn(0x7f010c0a3a90 legacy=0x7f010c1d4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 shutdown_connections 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 >> 172.21.15.148:0/684894309 conn(0x7f010c0a9a40 msgr2=0x7f010c09f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 shutdown_connections 2024-09-16T04:06:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:21.240+0000 7f0110b45640 1 -- 172.21.15.148:0/684894309 wait complete. 2024-09-16T04:06:21.350 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:21.269+0000 7fd78a236040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T04:06:21.350 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:21.310+0000 7fd78a236040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T04:06:21.351 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T04:06:21.351 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T04:06:21.351 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:06:21.352 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T04:06:21.377 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:06:21.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T04:06:21.442 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-16T04:06:21.442 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T04:06:21.455 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:21.454+0000 7fd78a236040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T04:06:21.505 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:21.504+0000 7fd78a236040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T04:06:21.506 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:21.507 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T04:06:21.532 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:21.532 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T04:06:21.596 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:06:21.596 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T04:06:21.662 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 30d6186b-ce3c-4385-a2da-04f1171e31c7 0 2024-09-16T04:06:21.676 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:21.674+0000 7fd78a236040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:06:21.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.772+0000 7f8ebabbd640 10 monclient: get_monmap_and_config 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.772+0000 7f8ebabbd640 10 monclient: build_initial_monmap 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.772+0000 7f8ebabbd640 10 monclient: monmap: 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:06:21.772930+0000 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:06:21.772930+0000 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.774 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:21.775 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:21.775 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:21.775 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.788 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.772+0000 7f8ebabbd640 1 Processor -- start 2024-09-16T04:06:21.788 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.772+0000 7f8ebabbd640 1 -- start start 2024-09-16T04:06:21.788 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.772+0000 7f8ebabbd640 10 monclient: init 2024-09-16T04:06:21.788 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:21.788 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:21.788 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb40a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb40a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb40a0e00 con 0x7f8eb40a1b10 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb40a10e0 con 0x7f8eb413d060 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb40a4970 con 0x7f8eb40a2410 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8ebabbd640 10 monclient(hunting): authenticate will time out at 757.278403s 2024-09-16T04:06:21.789 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb915a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8eb413d060 0x7f8eb413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45570/0 (socket says 172.21.15.171:45570) 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb915a640 1 -- 172.21.15.171:0/2633925137 learned_addr learned my addr 172.21.15.171:0/2633925137 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364413087 0 0) 0x7f8eb40a10e0 con 0x7f8eb413d060 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): my global_id is 4099 2024-09-16T04:06:21.790 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ea4003520 con 0x7f8eb413d060 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896709291 0 0) 0x7f8eb40a0e00 con 0x7f8eb40a1b10 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): my global_id is 4110 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb40a10e0 con 0x7f8eb40a1b10 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837575842 0 0) 0x7f8eb40a4970 con 0x7f8eb40a2410 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.791 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 10 monclient(hunting): my global_id is 4112 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.773+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb40a0e00 con 0x7f8eb40a2410 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1798056502 0 0) 0x7f8ea4003520 con 0x7f8eb413d060 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 >> v1:172.21.15.148:6790/0 conn(0x7f8eb40a2410 legacy=0x7f8eb40a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 >> v1:172.21.15.148:6789/0 conn(0x7f8eb40a1b10 legacy=0x7f8eb413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 monclient: found mon.noname-c 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb406dfd0 con 0x7f8eb413d060 2024-09-16T04:06:21.792 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 5 monclient: authenticate success, global_id 4099 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8eac001990 con 0x7f8eb413d060 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: dump: 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.793 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8eac001cb0 con 0x7f8eb413d060 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 1 -- 172.21.15.171:0/2633925137 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8eac002080 con 0x7f8eb413d060 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 4 set_mon_vals no callback set 2024-09-16T04:06:21.794 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8eb37fe640 10 monclient: dump: 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.795 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 10 monclient: shutdown 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2633925137 >> v1:172.21.15.171:6789/0 conn(0x7f8eb413d060 legacy=0x7f8eb413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2633925137 shutdown_connections 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2633925137 >> 172.21.15.171:0/2633925137 conn(0x7f8eb40a97b0 msgr2=0x7f8eb40a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2633925137 shutdown_connections 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.774+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2633925137 wait complete. 2024-09-16T04:06:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.775+0000 7f8ebabbd640 10 monclient: build_initial_monmap 2024-09-16T04:06:21.797 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.775+0000 7f8ebabbd640 10 monclient: monmap: 2024-09-16T04:06:21.797 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:21.797 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:21.797 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-09-16T04:06:21.797 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-09-16T04:06:21.797 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:21.798 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.798 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:21.798 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:21.798 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:21.798 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.798 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.775+0000 7f8ebabbd640 1 Processor -- start 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.775+0000 7f8ebabbd640 1 -- start start 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.775+0000 7f8ebabbd640 10 monclient: init 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb40a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb40a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:21.799 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb41cc910 con 0x7f8eb40a1b10 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb413cd40 con 0x7f8eb40a2410 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb413c7f0 con 0x7f8eb413d060 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8ebabbd640 10 monclient(hunting): authenticate will time out at 757.281379s 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8eb40a2410 0x7f8eb41cc200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45574/0 (socket says 172.21.15.171:45574) 2024-09-16T04:06:21.800 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb3fff640 1 -- 172.21.15.171:0/2677590215 learned_addr learned my addr 172.21.15.171:0/2677590215 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb915a640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.148:6790/0 conn(0x7f8eb413d060 legacy=0x7f8eb41d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb915a640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.148:6790/0 conn(0x7f8eb413d060 legacy=0x7f8eb41d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb915a640 1 --1- 172.21.15.171:0/2677590215 >> v1:172.21.15.148:6790/0 conn(0x7f8eb413d060 0x7f8eb41d3d50 :-1 s=OPENED pgs=16 cs=1 l=1).handle_message read tag failed 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb915a640 1 --1- 172.21.15.171:0/2677590215 >> v1:172.21.15.148:6790/0 conn(0x7f8eb413d060 0x7f8eb41d3d50 :-1 s=OPENED pgs=16 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6790/0 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338393850 0 0) 0x7f8eb413cd40 con 0x7f8eb40a2410 2024-09-16T04:06:21.801 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): my global_id is 4102 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e98003520 con 0x7f8eb40a2410 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1558372313 0 0) 0x7f8eb41cc910 con 0x7f8eb40a1b10 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): my global_id is 4113 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8eb413cd40 con 0x7f8eb40a1b10 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb3fff640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.171:6789/0 conn(0x7f8eb40a2410 legacy=0x7f8eb41cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb3fff640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.171:6789/0 conn(0x7f8eb40a2410 legacy=0x7f8eb41cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb3fff640 1 --1- 172.21.15.171:0/2677590215 >> v1:172.21.15.171:6789/0 conn(0x7f8eb40a2410 0x7f8eb41cc200 :-1 s=OPENED pgs=15 cs=1 l=1).handle_message read tag failed 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb3fff640 1 --1- 172.21.15.171:0/2677590215 >> v1:172.21.15.171:6789/0 conn(0x7f8eb40a2410 0x7f8eb41cc200 :-1 s=OPENED pgs=15 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.171:6789/0 2024-09-16T04:06:21.803 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1672584993 0 0) 0x7f8eb413cd40 con 0x7f8eb40a1b10 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8eb41cc910 con 0x7f8eb40a1b10 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ea80018b0 con 0x7f8eb40a1b10 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): dump: 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.804 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.805 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.776+0000 7f8eb17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2880653332 0 0) 0x7f8eb41cc910 con 0x7f8eb40a1b10 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.148:6790/0 conn(0x7f8eb413d060 legacy=0x7f8eb41d3d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.171:6789/0 conn(0x7f8eb40a2410 legacy=0x7f8eb41cc200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 monclient: found mon.a 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:21.806 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb40a0130 con 0x7f8eb40a1b10 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 5 monclient: authenticate success, global_id 4113 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 10 monclient: _renew_subs 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb41235b0 con 0x7f8eb40a1b10 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 10 monclient: _renew_subs 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8eb4122160 con 0x7f8eb40a1b10 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ea8003610 con 0x7f8eb40a1b10 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ea8003960 con 0x7f8eb40a1b10 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb2ffd640 4 set_mon_vals no callback set 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 10 monclient: dump: 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.808 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8ea8003e90 con 0x7f8eb40a1b10 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.777+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 2062578416 0 0) 0x7f8ea80054e0 con 0x7f8eb40a1b10 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.778+0000 7f8ebabbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.778+0000 7f8ebabbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.778+0000 7f8ebabbd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.778+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e94005080 con 0x7f8eb40a1b10 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.780+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f8ea8004360 con 0x7f8eb40a1b10 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.780+0000 7f8eb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.809 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.780+0000 7f8eb17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:21.810 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:21.791+0000 7fd78a236040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T04:06:21.851 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.849+0000 7f8ebabbd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "30d6186b-ce3c-4385-a2da-04f1171e31c7", "id": 0}] 2024-09-16T04:06:21.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.849+0000 7f8ebabbd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "30d6186b-ce3c-4385-a2da-04f1171e31c7", "id": 0}] 2024-09-16T04:06:21.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.849+0000 7f8ebabbd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:21.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.849+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd new", "uuid": "30d6186b-ce3c-4385-a2da-04f1171e31c7", "id": 0} v 0) -- 0x7f8e94005370 con 0x7f8eb40a1b10 2024-09-16T04:06:21.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.851+0000 7f8eb17fa640 1 -- 172.21.15.171:0/2677590215 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "30d6186b-ce3c-4385-a2da-04f1171e31c7", "id": 0}]=0 v3) ==== 112+0+2 (unknown 1165318761 0 732707515) 0x7f8ea8004710 con 0x7f8eb40a1b10 2024-09-16T04:06:21.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.851+0000 7f8eb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "30d6186b-ce3c-4385-a2da-04f1171e31c7", "id": 0}] 2024-09-16T04:06:21.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.851+0000 7f8eb17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:21.853 INFO:teuthology.orchestra.run.smithi171.stdout:0 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 10 monclient: shutdown 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 >> v1:172.21.15.148:6789/0 conn(0x7f8eb40a1b10 legacy=0x7f8eb41cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 shutdown_connections 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 >> 172.21.15.171:0/2677590215 conn(0x7f8eb40a97b0 msgr2=0x7f8eb41401d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 shutdown_connections 2024-09-16T04:06:21.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.852+0000 7f8ebabbd640 1 -- 172.21.15.171:0/2677590215 wait complete. 2024-09-16T04:06:21.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new d04305dd-0099-4412-a6cd-7aae325a600d 1 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.960+0000 7f3f8a918640 10 monclient: get_monmap_and_config 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.960+0000 7f3f8a918640 10 monclient: build_initial_monmap 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.960+0000 7f3f8a918640 10 monclient: monmap: 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:06:21.960925+0000 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:06:21.960925+0000 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:21.962 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:21.963 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.972 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.960+0000 7f3f8a918640 1 Processor -- start 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.960+0000 7f3f8a918640 1 -- start start 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.960+0000 7f3f8a918640 10 monclient: init 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient(hunting): picked mon.noname-b con 0x7f3f840a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient(hunting): picked mon.noname-a con 0x7f3f840a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f840a0590 con 0x7f3f840a4580 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f840a1270 con 0x7f3f8413d3c0 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f840a4fd0 con 0x7f3f840a3a90 2024-09-16T04:06:21.973 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f8a918640 10 monclient(hunting): authenticate will time out at 757.466509s 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f88eb5640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3f8413d3c0 0x7f3f8413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45584/0 (socket says 172.21.15.171:45584) 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f88eb5640 1 -- 172.21.15.171:0/1025575030 learned_addr learned my addr 172.21.15.171:0/1025575030 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368295997 0 0) 0x7f3f840a1270 con 0x7f3f8413d3c0 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f82ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f82ffd640 10 monclient(hunting): my global_id is 4105 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.961+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6c003520 con 0x7f3f8413d3c0 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3697385173 0 0) 0x7f3f840a4fd0 con 0x7f3f840a3a90 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.974 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): my global_id is 4115 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f840a1270 con 0x7f3f840a3a90 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4237179462 0 0) 0x7f3f78001110 con 0x7f3f840a4580 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient(hunting): my global_id is 4116 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f840a4fd0 con 0x7f3f840a4580 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1810867304 0 0) 0x7f3f6c003520 con 0x7f3f8413d3c0 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 >> v1:172.21.15.148:6790/0 conn(0x7f3f840a3a90 legacy=0x7f3f840a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 >> v1:172.21.15.148:6789/0 conn(0x7f3f840a4580 legacy=0x7f3f8413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:21.975 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 monclient: found mon.noname-c 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8406e170 con 0x7f3f8413d3c0 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 5 monclient: authenticate success, global_id 4105 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f7c001910 con 0x7f3f8413d3c0 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: dump: 2024-09-16T04:06:21.976 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f7c001b70 con 0x7f3f8413d3c0 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:21.977 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 1 -- 172.21.15.171:0/1025575030 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f7c001f40 con 0x7f3f8413d3c0 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 4 set_mon_vals no callback set 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f82ffd640 10 monclient: dump: 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.978 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 10 monclient: shutdown 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 1 -- 172.21.15.171:0/1025575030 >> v1:172.21.15.171:6789/0 conn(0x7f3f8413d3c0 legacy=0x7f3f8413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 1 -- 172.21.15.171:0/1025575030 shutdown_connections 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 1 -- 172.21.15.171:0/1025575030 >> 172.21.15.171:0/1025575030 conn(0x7f3f840a9a40 msgr2=0x7f3f840a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 1 -- 172.21.15.171:0/1025575030 shutdown_connections 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.962+0000 7f3f8a918640 1 -- 172.21.15.171:0/1025575030 wait complete. 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.963+0000 7f3f8a918640 10 monclient: build_initial_monmap 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.963+0000 7f3f8a918640 10 monclient: monmap: 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:21.979 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.963+0000 7f3f8a918640 1 Processor -- start 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.963+0000 7f3f8a918640 1 -- start start 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.963+0000 7f3f8a918640 10 monclient: init 2024-09-16T04:06:21.980 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient(hunting): picked mon.noname-b con 0x7f3f840a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient(hunting): picked mon.noname-a con 0x7f3f840a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8413d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8409f130 con 0x7f3f840a4580 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8409da00 con 0x7f3f840a3a90 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f841ccbe0 con 0x7f3f8413d3c0 2024-09-16T04:06:21.981 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:21.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f8a918640 10 monclient(hunting): authenticate will time out at 757.469532s 2024-09-16T04:06:21.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f83fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3f840a3a90 0x7f3f841cbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45600/0 (socket says 172.21.15.171:45600) 2024-09-16T04:06:21.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f83fff640 1 -- 172.21.15.171:0/3925957424 learned_addr learned my addr 172.21.15.171:0/3925957424 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:21.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170655221 0 0) 0x7f3f8409da00 con 0x7f3f840a3a90 2024-09-16T04:06:21.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): my global_id is 4108 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f70003520 con 0x7f3f840a3a90 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110491862 0 0) 0x7f3f8409f130 con 0x7f3f840a4580 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): my global_id is 4119 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f8409da00 con 0x7f3f840a4580 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787892228 0 0) 0x7f3f841ccbe0 con 0x7f3f8413d3c0 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:21.983 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 10 monclient(hunting): my global_id is 4118 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f8409f130 con 0x7f3f8413d3c0 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.964+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 769841341 0 0) 0x7f3f70003520 con 0x7f3f840a3a90 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f841ccbe0 con 0x7f3f840a3a90 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f74001970 con 0x7f3f840a3a90 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.984 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 725089418 0 0) 0x7f3f8409da00 con 0x7f3f840a4580 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f70003520 con 0x7f3f840a4580 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f78001a00 con 0x7f3f840a4580 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:21.985 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3165252821 0 0) 0x7f3f8409f130 con 0x7f3f8413d3c0 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f8409da00 con 0x7f3f8413d3c0 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f7c001ec0 con 0x7f3f8413d3c0 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:21.986 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 672826986 0 0) 0x7f3f841ccbe0 con 0x7f3f840a3a90 2024-09-16T04:06:21.987 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 >> v1:172.21.15.148:6790/0 conn(0x7f3f8413d3c0 legacy=0x7f3f841d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 >> v1:172.21.15.148:6789/0 conn(0x7f3f840a4580 legacy=0x7f3f841cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 monclient: found mon.b 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f841409b0 con 0x7f3f840a3a90 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: _finish_auth 0 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 5 monclient: authenticate success, global_id 4108 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 10 monclient: _renew_subs 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8413d140 con 0x7f3f840a3a90 2024-09-16T04:06:21.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 10 monclient: _renew_subs 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f8409ee50 con 0x7f3f840a3a90 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f740038c0 con 0x7f3f840a3a90 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f827fc640 4 set_mon_vals no callback set 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f74003a70 con 0x7f3f840a3a90 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 10 monclient: dump: 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:21.989 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3f740040a0 con 0x7f3f840a3a90 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.965+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 3600568970 0 0) 0x7f3f740067e0 con 0x7f3f840a3a90 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.966+0000 7f3f8a918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.966+0000 7f3f8a918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.966+0000 7f3f8a918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:21.990 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.966+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f64005080 con 0x7f3f840a3a90 2024-09-16T04:06:21.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.968+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f3f74006a80 con 0x7f3f840a3a90 2024-09-16T04:06:21.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.968+0000 7f3f80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:21.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:21.968+0000 7f3f80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:22.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.038+0000 7f3f8a918640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d04305dd-0099-4412-a6cd-7aae325a600d", "id": 1}] 2024-09-16T04:06:22.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.038+0000 7f3f8a918640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d04305dd-0099-4412-a6cd-7aae325a600d", "id": 1}] 2024-09-16T04:06:22.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.038+0000 7f3f8a918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.040 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.038+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d04305dd-0099-4412-a6cd-7aae325a600d", "id": 1} v 0) -- 0x7f3f64005370 con 0x7f3f840a3a90 2024-09-16T04:06:22.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.045+0000 7f3f80ff9640 1 -- 172.21.15.171:0/3925957424 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d04305dd-0099-4412-a6cd-7aae325a600d", "id": 1}]=0 v4) ==== 112+0+2 (unknown 2678623426 0 940483276) 0x7f3f740057c0 con 0x7f3f840a3a90 2024-09-16T04:06:22.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.045+0000 7f3f80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d04305dd-0099-4412-a6cd-7aae325a600d", "id": 1}] 2024-09-16T04:06:22.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.045+0000 7f3f80ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:22.048 INFO:teuthology.orchestra.run.smithi171.stdout:1 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 10 monclient: shutdown 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 >> v1:172.21.15.171:6789/0 conn(0x7f3f840a3a90 legacy=0x7f3f841cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 shutdown_connections 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 >> 172.21.15.171:0/3925957424 conn(0x7f3f840a9a40 msgr2=0x7f3f8413fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 shutdown_connections 2024-09-16T04:06:22.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.046+0000 7f3f8a918640 1 -- 172.21.15.171:0/3925957424 wait complete. 2024-09-16T04:06:22.059 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new c64f2f7b-204c-4e36-8960-822df4fbc0c3 2 2024-09-16T04:06:22.140 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:22.138+0000 7fd78a236040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.156+0000 7f353d19f640 10 monclient: get_monmap_and_config 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.156+0000 7f353d19f640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.156+0000 7f353d19f640 10 monclient: monmap: 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:06:22.157114+0000 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:06:22.157114+0000 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.159 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.159 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:22.159 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:22.159 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.156+0000 7f353d19f640 1 Processor -- start 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.156+0000 7f353d19f640 1 -- start start 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.156+0000 7f353d19f640 10 monclient: init 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient(hunting): picked mon.noname-b con 0x7f35380a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient(hunting): picked mon.noname-c con 0x7f35380a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient(hunting): picked mon.noname-a con 0x7f353813d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35380a0590 con 0x7f353813d3c0 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35380a1270 con 0x7f35380a4580 2024-09-16T04:06:22.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35380a4fd0 con 0x7f35380a3a90 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353d19f640 10 monclient(hunting): authenticate will time out at 757.662526s 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f35380a4580 0x7f353813bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45602/0 (socket says 172.21.15.171:45602) 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f353659c640 1 -- 172.21.15.171:0/3762811151 learned_addr learned my addr 172.21.15.171:0/3762811151 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236970345 0 0) 0x7f35380a4fd0 con 0x7f35380a3a90 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): my global_id is 4121 2024-09-16T04:06:22.167 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3520003520 con 0x7f35380a3a90 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2094777356 0 0) 0x7f3530001490 con 0x7f353813d3c0 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 10 monclient(hunting): my global_id is 4122 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.157+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35380a4fd0 con 0x7f353813d3c0 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 615473806 0 0) 0x7f3520003520 con 0x7f35380a3a90 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 >> v1:172.21.15.171:6789/0 conn(0x7f35380a4580 legacy=0x7f353813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 >> v1:172.21.15.148:6789/0 conn(0x7f353813d3c0 legacy=0x7f353813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 monclient: found mon.noname-b 2024-09-16T04:06:22.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353806e170 con 0x7f35380a3a90 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3528001610 con 0x7f35380a3a90 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 5 monclient: authenticate success, global_id 4121 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: dump: 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.169 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.170 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3528001930 con 0x7f35380a3a90 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 1 -- 172.21.15.171:0/3762811151 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3528001d00 con 0x7f35380a3a90 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 4 set_mon_vals no callback set 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f3535d9b640 10 monclient: dump: 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.171 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 10 monclient: shutdown 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 1 -- 172.21.15.171:0/3762811151 >> v1:172.21.15.148:6790/0 conn(0x7f35380a3a90 legacy=0x7f35380a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.172 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 1 -- 172.21.15.171:0/3762811151 shutdown_connections 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 1 -- 172.21.15.171:0/3762811151 >> 172.21.15.171:0/3762811151 conn(0x7f35380a9a40 msgr2=0x7f35380a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 1 -- 172.21.15.171:0/3762811151 shutdown_connections 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.158+0000 7f353d19f640 1 -- 172.21.15.171:0/3762811151 wait complete. 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.159+0000 7f353d19f640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.159+0000 7f353d19f640 10 monclient: monmap: 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:22.173 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 1 Processor -- start 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 1 -- start start 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient: init 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient(hunting): picked mon.noname-b con 0x7f35380a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient(hunting): picked mon.noname-a con 0x7f35380a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient(hunting): picked mon.noname-c con 0x7f353813d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353809ee90 con 0x7f35380a4580 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35381ccbe0 con 0x7f35380a3a90 2024-09-16T04:06:22.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3538140940 con 0x7f353813d3c0 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f353d19f640 10 monclient(hunting): authenticate will time out at 757.665668s 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f3536d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f35380a3a90 0x7f35381cbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45606/0 (socket says 172.21.15.171:45606) 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.160+0000 7f3536d9d640 1 -- 172.21.15.171:0/1667175290 learned_addr learned my addr 172.21.15.171:0/1667175290 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212481526 0 0) 0x7f3538140940 con 0x7f353813d3c0 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): my global_id is 4124 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3524003520 con 0x7f353813d3c0 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264116622 0 0) 0x7f353809ee90 con 0x7f35380a4580 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.175 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): my global_id is 4125 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3538140940 con 0x7f35380a4580 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3959131995 0 0) 0x7f3524003520 con 0x7f353813d3c0 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f353809ee90 con 0x7f353813d3c0 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3530001e00 con 0x7f353813d3c0 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.176 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 439548096 0 0) 0x7f3538140940 con 0x7f35380a4580 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3524003520 con 0x7f35380a4580 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f352c001760 con 0x7f35380a4580 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.177 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 881506041 0 0) 0x7f353809ee90 con 0x7f353813d3c0 2024-09-16T04:06:22.178 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 >> v1:172.21.15.171:6789/0 conn(0x7f35380a3a90 legacy=0x7f35381cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 >> v1:172.21.15.148:6789/0 conn(0x7f35380a4580 legacy=0x7f35381cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 monclient: found mon.c 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35380a5510 con 0x7f353813d3c0 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 5 monclient: authenticate success, global_id 4124 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 10 monclient: _renew_subs 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:22.179 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f353809ebb0 con 0x7f353813d3c0 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 10 monclient: _renew_subs 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f353809c550 con 0x7f353813d3c0 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3530003970 con 0x7f353813d3c0 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3530003cc0 con 0x7f353813d3c0 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f353559a640 4 set_mon_vals no callback set 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 10 monclient: dump: 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.180 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f35300041f0 con 0x7f353813d3c0 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.161+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 3702167423 0 0) 0x7f3530005940 con 0x7f353813d3c0 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.162+0000 7f353d19f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.162+0000 7f353d19f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.162+0000 7f353d19f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:22.181 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.162+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3518005080 con 0x7f353813d3c0 2024-09-16T04:06:22.182 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.164+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f35380a0590 con 0x7f353813d3c0 2024-09-16T04:06:22.182 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.164+0000 7f351f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.182 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.164+0000 7f351f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:22.195 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:22.193+0000 7fd78a236040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T04:06:22.236 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.234+0000 7f353d19f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c64f2f7b-204c-4e36-8960-822df4fbc0c3", "id": 2}] 2024-09-16T04:06:22.236 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.234+0000 7f353d19f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c64f2f7b-204c-4e36-8960-822df4fbc0c3", "id": 2}] 2024-09-16T04:06:22.236 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.234+0000 7f353d19f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:22.236 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.234+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c64f2f7b-204c-4e36-8960-822df4fbc0c3", "id": 2} v 0) -- 0x7f3518005370 con 0x7f353813d3c0 2024-09-16T04:06:22.239 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.237+0000 7f351f7fe640 1 -- 172.21.15.171:0/1667175290 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c64f2f7b-204c-4e36-8960-822df4fbc0c3", "id": 2}]=0 v5) ==== 112+0+2 (unknown 3236881683 0 216599125) 0x7f35300056d0 con 0x7f353813d3c0 2024-09-16T04:06:22.239 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.237+0000 7f351f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c64f2f7b-204c-4e36-8960-822df4fbc0c3", "id": 2}] 2024-09-16T04:06:22.239 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.237+0000 7f351f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:22.239 INFO:teuthology.orchestra.run.smithi171.stdout:2 2024-09-16T04:06:22.240 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 10 monclient: shutdown 2024-09-16T04:06:22.240 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.240 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 >> v1:172.21.15.148:6790/0 conn(0x7f353813d3c0 legacy=0x7f35381d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.240 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 shutdown_connections 2024-09-16T04:06:22.240 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 >> 172.21.15.171:0/1667175290 conn(0x7f35380a9a40 msgr2=0x7f35380a6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.241 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 shutdown_connections 2024-09-16T04:06:22.241 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.238+0000 7f353d19f640 1 -- 172.21.15.171:0/1667175290 wait complete. 2024-09-16T04:06:22.251 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 030ce3ae-cb8f-410f-8718-c6b1d9f10134 3 2024-09-16T04:06:22.349 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.347+0000 7f314d653640 10 monclient: get_monmap_and_config 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.347+0000 7f314d653640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.347+0000 7f314d653640 10 monclient: monmap: 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:06:22.348666+0000 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:06:22.348666+0000 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.350 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.351 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:22.351 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:22.351 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 1 Processor -- start 2024-09-16T04:06:22.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 1 -- start start 2024-09-16T04:06:22.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 10 monclient: init 2024-09-16T04:06:22.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:22.357 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 10 monclient(hunting): picked mon.noname-b con 0x7f31480a28c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 10 monclient(hunting): picked mon.noname-c con 0x7f31480a2ca0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.348+0000 7f314d653640 10 monclient(hunting): picked mon.noname-a con 0x7f314813aad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314809eb60 con 0x7f314813aad0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31480a0250 con 0x7f31480a2ca0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314809efa0 con 0x7f31480a28c0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 10 monclient(hunting): authenticate will time out at 757.854065s 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f31467fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f31480a2ca0 0x7f31480a30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45618/0 (socket says 172.21.15.171:45618) 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f31467fc640 1 -- 172.21.15.171:0/2799144318 learned_addr learned my addr 172.21.15.171:0/2799144318 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3852595927 0 0) 0x7f31480a0250 con 0x7f31480a2ca0 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.358 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): my global_id is 4111 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3134003520 con 0x7f31480a2ca0 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198945599 0 0) 0x7f314809efa0 con 0x7f31480a28c0 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): my global_id is 4127 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31480a0250 con 0x7f31480a28c0 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751776032 0 0) 0x7f314809eb60 con 0x7f314813aad0 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.359 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient(hunting): my global_id is 4128 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f314809efa0 con 0x7f314813aad0 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3577401327 0 0) 0x7f3134003520 con 0x7f31480a2ca0 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 >> v1:172.21.15.148:6790/0 conn(0x7f31480a28c0 legacy=0x7f314809f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 >> v1:172.21.15.148:6789/0 conn(0x7f314813aad0 legacy=0x7f314813cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 monclient: found mon.noname-c 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.360 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31480a4740 con 0x7f31480a2ca0 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3138001610 con 0x7f31480a2ca0 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: dump: 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.361 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 5 monclient: authenticate success, global_id 4111 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3138001930 con 0x7f31480a2ca0 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 1 -- 172.21.15.171:0/2799144318 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3138001d00 con 0x7f31480a2ca0 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f314d653640 4 set_mon_vals no callback set 2024-09-16T04:06:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.349+0000 7f3145ffb640 10 monclient: dump: 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.363 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 10 monclient: shutdown 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 1 -- 172.21.15.171:0/2799144318 >> v1:172.21.15.171:6789/0 conn(0x7f31480a2ca0 legacy=0x7f31480a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 1 -- 172.21.15.171:0/2799144318 shutdown_connections 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 1 -- 172.21.15.171:0/2799144318 >> 172.21.15.171:0/2799144318 conn(0x7f31480a39b0 msgr2=0x7f31480a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 1 -- 172.21.15.171:0/2799144318 shutdown_connections 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.350+0000 7f314d653640 1 -- 172.21.15.171:0/2799144318 wait complete. 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.351+0000 7f314d653640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.351+0000 7f314d653640 10 monclient: monmap: 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-09-16T04:06:22.364 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.351+0000 7f314d653640 1 Processor -- start 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.351+0000 7f314d653640 1 -- start start 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.351+0000 7f314d653640 10 monclient: init 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.365 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient(hunting): picked mon.noname-b con 0x7f31480a28c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient(hunting): picked mon.noname-a con 0x7f31480a2ca0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient(hunting): picked mon.noname-c con 0x7f314813aad0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3148138870 con 0x7f31480a2ca0 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314813a280 con 0x7f31480a28c0 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314809d550 con 0x7f314813aad0 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.366 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient(hunting): authenticate will time out at 757.857174s 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f3146ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f31480a28c0 0x7f31481c9ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45634/0 (socket says 172.21.15.171:45634) 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f3146ffd640 1 -- 172.21.15.171:0/1539173106 learned_addr learned my addr 172.21.15.171:0/1539173106 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 759143354 0 0) 0x7f314813a280 con 0x7f31480a28c0 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): my global_id is 4114 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3128003520 con 0x7f31480a28c0 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491977019 0 0) 0x7f314809d550 con 0x7f314813aad0 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.367 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): my global_id is 4130 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f314813a280 con 0x7f314813aad0 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104219123 0 0) 0x7f3148138870 con 0x7f31480a2ca0 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): my global_id is 4131 2024-09-16T04:06:22.368 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f314809d550 con 0x7f31480a2ca0 2024-09-16T04:06:22.368 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:22.352+0000 7fd78a236040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T04:06:22.371 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 771038921 0 0) 0x7f3128003520 con 0x7f31480a28c0 2024-09-16T04:06:22.371 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3148138870 con 0x7f31480a28c0 2024-09-16T04:06:22.371 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3130001830 con 0x7f31480a28c0 2024-09-16T04:06:22.371 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.371 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.372 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4293319495 0 0) 0x7f314813a280 con 0x7f314813aad0 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3128003520 con 0x7f314813aad0 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3140001bb0 con 0x7f314813aad0 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.373 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3429293302 0 0) 0x7f314809d550 con 0x7f31480a2ca0 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f314813a280 con 0x7f31480a2ca0 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3138001430 con 0x7f31480a2ca0 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.374 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2551349552 0 0) 0x7f3148138870 con 0x7f31480a28c0 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 >> v1:172.21.15.148:6790/0 conn(0x7f314813aad0 legacy=0x7f31481d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 >> v1:172.21.15.148:6789/0 conn(0x7f31480a2ca0 legacy=0x7f31481ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.375 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 monclient: found mon.b 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31480a6920 con 0x7f31480a28c0 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f313f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 5 monclient: authenticate success, global_id 4114 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient: _renew_subs 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.352+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3148139cc0 con 0x7f31480a28c0 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 10 monclient: _renew_subs 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.376 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3148137300 con 0x7f31480a28c0 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3130003610 con 0x7f31480a28c0 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3130003960 con 0x7f31480a28c0 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 10 monclient: dump: 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.377 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f31457fa640 4 set_mon_vals no callback set 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3130003e90 con 0x7f31480a28c0 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 3254177092 0 0) 0x7f31300067c0 con 0x7f31480a28c0 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.353+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f312c005080 con 0x7f31480a28c0 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.355+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f31300043b0 con 0x7f31480a28c0 2024-09-16T04:06:22.378 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.355+0000 7f313f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.379 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.355+0000 7f313f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:22.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.425+0000 7f314d653640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "030ce3ae-cb8f-410f-8718-c6b1d9f10134", "id": 3}] 2024-09-16T04:06:22.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.425+0000 7f314d653640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "030ce3ae-cb8f-410f-8718-c6b1d9f10134", "id": 3}] 2024-09-16T04:06:22.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.425+0000 7f314d653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.427 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.425+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "030ce3ae-cb8f-410f-8718-c6b1d9f10134", "id": 3} v 0) -- 0x7f312c005370 con 0x7f31480a28c0 2024-09-16T04:06:22.431 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.430+0000 7f313f7fe640 1 -- 172.21.15.171:0/1539173106 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "030ce3ae-cb8f-410f-8718-c6b1d9f10134", "id": 3}]=0 v6) ==== 112+0+2 (unknown 1227725601 0 525046306) 0x7f3130005be0 con 0x7f31480a28c0 2024-09-16T04:06:22.432 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.430+0000 7f313f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "030ce3ae-cb8f-410f-8718-c6b1d9f10134", "id": 3}] 2024-09-16T04:06:22.432 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.430+0000 7f313f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:22.432 INFO:teuthology.orchestra.run.smithi171.stdout:3 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 10 monclient: shutdown 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 >> v1:172.21.15.171:6789/0 conn(0x7f31480a28c0 legacy=0x7f31481c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 shutdown_connections 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 >> 172.21.15.171:0/1539173106 conn(0x7f31480a39b0 msgr2=0x7f31480a5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 shutdown_connections 2024-09-16T04:06:22.433 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.431+0000 7f314d653640 1 -- 172.21.15.171:0/1539173106 wait complete. 2024-09-16T04:06:22.443 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new f82ddc46-d03e-415c-8b1d-0f3c98df29c6 4 2024-09-16T04:06:22.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.537+0000 7fe68bded640 10 monclient: get_monmap_and_config 2024-09-16T04:06:22.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.537+0000 7fe68bded640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.537+0000 7fe68bded640 10 monclient: monmap: 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:06:22.538479+0000 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:06:22.538479+0000 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:22.540 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 1 Processor -- start 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 1 -- start start 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 10 monclient: init 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 10 monclient(hunting): picked mon.noname-b con 0x7fe6840a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.538+0000 7fe68bded640 10 monclient(hunting): picked mon.noname-a con 0x7fe6840a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 10 monclient(hunting): picked mon.noname-c con 0x7fe68413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840a0590 con 0x7fe6840a4580 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840a1270 con 0x7fe68413d3c0 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840a4fd0 con 0x7fe6840a3a90 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 10 monclient(hunting): authenticate will time out at 758.044067s 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68a38a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe68413d3c0 0x7fe68413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45644/0 (socket says 172.21.15.171:45644) 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68a38a640 1 -- 172.21.15.171:0/26222896 learned_addr learned my addr 172.21.15.171:0/26222896 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507793233 0 0) 0x7fe6840a1270 con 0x7fe68413d3c0 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): my global_id is 4117 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe670003520 con 0x7fe68413d3c0 2024-09-16T04:06:22.550 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581854046 0 0) 0x7fe6840a4fd0 con 0x7fe6840a3a90 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): my global_id is 4133 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6840a1270 con 0x7fe6840a3a90 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152523412 0 0) 0x7fe674001110 con 0x7fe6840a4580 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient(hunting): my global_id is 4134 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6840a4fd0 con 0x7fe6840a4580 2024-09-16T04:06:22.551 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2811428173 0 0) 0x7fe670003520 con 0x7fe68413d3c0 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 >> v1:172.21.15.148:6790/0 conn(0x7fe6840a3a90 legacy=0x7fe6840a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 >> v1:172.21.15.148:6789/0 conn(0x7fe6840a4580 legacy=0x7fe68413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 monclient: found mon.noname-c 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68406e170 con 0x7fe68413d3c0 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe67c001990 con 0x7fe68413d3c0 2024-09-16T04:06:22.552 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 5 monclient: authenticate success, global_id 4117 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: dump: 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.553 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe67c001cb0 con 0x7fe68413d3c0 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 1 -- 172.21.15.171:0/26222896 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe67c002080 con 0x7fe68413d3c0 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe68bded640 4 set_mon_vals no callback set 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.539+0000 7fe688b87640 10 monclient: dump: 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.554 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 10 monclient: shutdown 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 1 -- 172.21.15.171:0/26222896 >> v1:172.21.15.171:6789/0 conn(0x7fe68413d3c0 legacy=0x7fe68413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 1 -- 172.21.15.171:0/26222896 shutdown_connections 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 1 -- 172.21.15.171:0/26222896 >> 172.21.15.171:0/26222896 conn(0x7fe6840a9a40 msgr2=0x7fe6840a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.555 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 1 -- 172.21.15.171:0/26222896 shutdown_connections 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.540+0000 7fe68bded640 1 -- 172.21.15.171:0/26222896 wait complete. 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient: monmap: 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:22.556 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 1 Processor -- start 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 1 -- start start 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient: init 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient(hunting): picked mon.noname-a con 0x7fe6840a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient(hunting): picked mon.noname-c con 0x7fe6840a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient(hunting): picked mon.noname-b con 0x7fe68413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68409eec0 con 0x7fe6840a3a90 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68409da70 con 0x7fe68413d3c0 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68409dcf0 con 0x7fe6840a4580 2024-09-16T04:06:22.557 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68bded640 10 monclient(hunting): authenticate will time out at 758.046884s 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68a38a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe68413d3c0 0x7fe6841d4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45652/0 (socket says 172.21.15.171:45652) 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.541+0000 7fe68a38a640 1 -- 172.21.15.171:0/4204982713 learned_addr learned my addr 172.21.15.171:0/4204982713 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567763442 0 0) 0x7fe68409da70 con 0x7fe68413d3c0 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): my global_id is 4120 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe668003520 con 0x7fe68413d3c0 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2299285485 0 0) 0x7fe68409dcf0 con 0x7fe6840a4580 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.558 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): my global_id is 4136 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe68409da70 con 0x7fe6840a4580 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374988697 0 0) 0x7fe68409eec0 con 0x7fe6840a3a90 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): my global_id is 4137 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe68409dcf0 con 0x7fe6840a3a90 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1834555673 0 0) 0x7fe668003520 con 0x7fe68413d3c0 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe68409eec0 con 0x7fe68413d3c0 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe67c001950 con 0x7fe68413d3c0 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.559 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2681386712 0 0) 0x7fe68409da70 con 0x7fe6840a4580 2024-09-16T04:06:22.560 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe668003520 con 0x7fe6840a4580 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe674001a80 con 0x7fe6840a4580 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.561 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1746216271 0 0) 0x7fe68409dcf0 con 0x7fe6840a3a90 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe68409da70 con 0x7fe6840a3a90 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe680001830 con 0x7fe6840a3a90 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.562 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 307897762 0 0) 0x7fe68409eec0 con 0x7fe68413d3c0 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 >> v1:172.21.15.148:6790/0 conn(0x7fe6840a4580 legacy=0x7fe6841d1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 >> v1:172.21.15.148:6789/0 conn(0x7fe6840a3a90 legacy=0x7fe6841cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 monclient: found mon.b 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.563 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68409f1a0 con 0x7fe68413d3c0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 5 monclient: authenticate success, global_id 4120 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 10 monclient: _renew_subs 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6840a82a0 con 0x7fe68413d3c0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 10 monclient: _renew_subs 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe68409c500 con 0x7fe68413d3c0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe67c001cb0 con 0x7fe68413d3c0 2024-09-16T04:06:22.564 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67bfff640 4 set_mon_vals no callback set 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe67c005040 con 0x7fe68413d3c0 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 10 monclient: dump: 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.565 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe67c0053d0 con 0x7fe68413d3c0 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.542+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 2504994704 0 0) 0x7fe67c006c40 con 0x7fe68413d3c0 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.543+0000 7fe68bded640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.543+0000 7fe68bded640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.543+0000 7fe68bded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.543+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe66c005080 con 0x7fe68413d3c0 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.545+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7fe67c0067e0 con 0x7fe68413d3c0 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.545+0000 7fe67a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.566 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.545+0000 7fe67a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:22.632 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.630+0000 7fe68bded640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f82ddc46-d03e-415c-8b1d-0f3c98df29c6", "id": 4}] 2024-09-16T04:06:22.633 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.630+0000 7fe68bded640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f82ddc46-d03e-415c-8b1d-0f3c98df29c6", "id": 4}] 2024-09-16T04:06:22.633 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.630+0000 7fe68bded640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.633 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.630+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f82ddc46-d03e-415c-8b1d-0f3c98df29c6", "id": 4} v 0) -- 0x7fe66c005370 con 0x7fe68413d3c0 2024-09-16T04:06:22.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.638+0000 7fe67a7fc640 1 -- 172.21.15.171:0/4204982713 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f82ddc46-d03e-415c-8b1d-0f3c98df29c6", "id": 4}]=0 v7) ==== 112+0+2 (unknown 2864901467 0 1697012583) 0x7fe67c0058a0 con 0x7fe68413d3c0 2024-09-16T04:06:22.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.638+0000 7fe67a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f82ddc46-d03e-415c-8b1d-0f3c98df29c6", "id": 4}] 2024-09-16T04:06:22.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.638+0000 7fe67a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:22.640 INFO:teuthology.orchestra.run.smithi171.stdout:4 2024-09-16T04:06:22.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 10 monclient: shutdown 2024-09-16T04:06:22.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 >> v1:172.21.15.171:6789/0 conn(0x7fe68413d3c0 legacy=0x7fe6841d4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 shutdown_connections 2024-09-16T04:06:22.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 >> 172.21.15.171:0/4204982713 conn(0x7fe6840a9a40 msgr2=0x7fe68413ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 shutdown_connections 2024-09-16T04:06:22.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.639+0000 7fe68bded640 1 -- 172.21.15.171:0/4204982713 wait complete. 2024-09-16T04:06:22.651 DEBUG:teuthology.orchestra.run.smithi171:> sudo ceph --cluster ceph osd new 18ff6471-ce6d-4c23-ac7c-415157ed598e 5 2024-09-16T04:06:22.749 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.747+0000 7f96c3991640 10 monclient: get_monmap_and_config 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.747+0000 7f96c3991640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.747+0000 7f96c3991640 10 monclient: monmap: 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:06:22.748508+0000 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:06:22.748508+0000 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.750 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.751 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:22.751 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:22.751 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.752 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 1 Processor -- start 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 1 -- start start 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient: init 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient(hunting): picked mon.noname-b con 0x7f96bc0a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient(hunting): picked mon.noname-a con 0x7f96bc0a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient(hunting): picked mon.noname-c con 0x7f96bc13d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.759 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc0a0590 con 0x7f96bc0a4580 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc0a1270 con 0x7f96bc13d3c0 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc0a4fd0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.748+0000 7f96c3991640 10 monclient(hunting): authenticate will time out at 758.253993s 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c1f2e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f96bc13d3c0 0x7f96bc13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45668/0 (socket says 172.21.15.171:45668) 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c1f2e640 1 -- 172.21.15.171:0/201318948 learned_addr learned my addr 172.21.15.171:0/201318948 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435728528 0 0) 0x7f96bc0a1270 con 0x7f96bc13d3c0 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): my global_id is 4123 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96a8003520 con 0x7f96bc13d3c0 2024-09-16T04:06:22.760 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745461404 0 0) 0x7f96ac001110 con 0x7f96bc0a4580 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): my global_id is 4140 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96bc0a1270 con 0x7f96bc0a4580 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236356944 0 0) 0x7f96bc0a4fd0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient(hunting): my global_id is 4139 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96ac001110 con 0x7f96bc0a3a90 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2512761452 0 0) 0x7f96a8003520 con 0x7f96bc13d3c0 2024-09-16T04:06:22.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 >> v1:172.21.15.148:6790/0 conn(0x7f96bc0a3a90 legacy=0x7f96bc0a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 >> v1:172.21.15.148:6789/0 conn(0x7f96bc0a4580 legacy=0x7f96bc13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 monclient: found mon.noname-c 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc06e170 con 0x7f96bc13d3c0 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c3991640 5 monclient: authenticate success, global_id 4123 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96b4001990 con 0x7f96bc13d3c0 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c3991640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: dump: 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c3991640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f96b4001cb0 con 0x7f96bc13d3c0 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 1 -- 172.21.15.171:0/201318948 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96b4002080 con 0x7f96bc13d3c0 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c3991640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96c3991640 4 set_mon_vals no callback set 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.749+0000 7f96b3fff640 10 monclient: dump: 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.764 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 10 monclient: shutdown 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 1 -- 172.21.15.171:0/201318948 >> v1:172.21.15.171:6789/0 conn(0x7f96bc13d3c0 legacy=0x7f96bc13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 1 -- 172.21.15.171:0/201318948 shutdown_connections 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 1 -- 172.21.15.171:0/201318948 >> 172.21.15.171:0/201318948 conn(0x7f96bc0a9a40 msgr2=0x7f96bc0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 1 -- 172.21.15.171:0/201318948 shutdown_connections 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 1 -- 172.21.15.171:0/201318948 wait complete. 2024-09-16T04:06:22.765 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.750+0000 7f96c3991640 10 monclient: monmap: 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 0 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 0.000000 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:created 0.000000 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 1 Processor -- start 2024-09-16T04:06:22.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 1 -- start start 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient: init 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient(hunting): picked mon.noname-b con 0x7f96bc0a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient(hunting): picked mon.noname-c con 0x7f96bc0a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient(hunting): picked mon.noname-a con 0x7f96bc13d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc0a0280 con 0x7f96bc13d3c0 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc138510 con 0x7f96bc0a3a90 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96bc0a82a0 con 0x7f96bc0a4580 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c3991640 10 monclient(hunting): authenticate will time out at 758.256781s 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c172d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f96bc0a3a90 0x7f96bc09ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45676/0 (socket says 172.21.15.171:45676) 2024-09-16T04:06:22.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.751+0000 7f96c172d640 1 -- 172.21.15.171:0/24592820 learned_addr learned my addr 172.21.15.171:0/24592820 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152695707 0 0) 0x7f96bc138510 con 0x7f96bc0a3a90 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): my global_id is 4126 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96a0003520 con 0x7f96bc0a3a90 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 708371846 0 0) 0x7f96bc0a82a0 con 0x7f96bc0a4580 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): my global_id is 4142 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96bc138510 con 0x7f96bc0a4580 2024-09-16T04:06:22.768 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740885995 0 0) 0x7f96bc06e170 con 0x7f96bc13d3c0 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): my global_id is 4143 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96bc0a82a0 con 0x7f96bc13d3c0 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 436206859 0 0) 0x7f96a0003520 con 0x7f96bc0a3a90 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96bc06e170 con 0x7f96bc0a3a90 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96b8001930 con 0x7f96bc0a3a90 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.769 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2975655513 0 0) 0x7f96bc138510 con 0x7f96bc0a4580 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96a0003520 con 0x7f96bc0a4580 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96ac001a40 con 0x7f96bc0a4580 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.770 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2696399933 0 0) 0x7f96bc0a82a0 con 0x7f96bc13d3c0 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96bc138510 con 0x7f96bc13d3c0 2024-09-16T04:06:22.771 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96b40017b0 con 0x7f96bc13d3c0 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.772 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4220910100 0 0) 0x7f96bc06e170 con 0x7f96bc0a3a90 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 >> v1:172.21.15.148:6790/0 conn(0x7f96bc0a4580 legacy=0x7f96bc09d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 >> v1:172.21.15.148:6789/0 conn(0x7f96bc13d3c0 legacy=0x7f96bc09d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 monclient: found mon.b 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc13ce60 con 0x7f96bc0a3a90 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 5 monclient: authenticate success, global_id 4126 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 10 monclient: _renew_subs 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f96bc138ad0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f96b8001d00 con 0x7f96bc0a3a90 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96b8003a60 con 0x7f96bc0a3a90 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 10 monclient: dump: 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:epoch 1 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.774 INFO:teuthology.orchestra.run.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:election_strategy: 1 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96b8004010 con 0x7f96bc0a3a90 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b37fe640 4 set_mon_vals no callback set 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 10 monclient: _renew_subs 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f96bc1d4e20 con 0x7f96bc0a3a90 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.752+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 922276729 0 0) 0x7f96b8005d40 con 0x7f96bc0a3a90 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.753+0000 7f96c3991640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.753+0000 7f96c3991640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.753+0000 7f96c3991640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.753+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96bc0008d0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.755+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f96b80057c0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.776 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.755+0000 7f96b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:22.776 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.755+0000 7f96b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:22.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.824+0000 7f96c3991640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "18ff6471-ce6d-4c23-ac7c-415157ed598e", "id": 5}] 2024-09-16T04:06:22.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.824+0000 7f96c3991640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "18ff6471-ce6d-4c23-ac7c-415157ed598e", "id": 5}] 2024-09-16T04:06:22.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.824+0000 7f96c3991640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:22.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.824+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd new", "uuid": "18ff6471-ce6d-4c23-ac7c-415157ed598e", "id": 5} v 0) -- 0x7f96bc0a4bc0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.834 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.832+0000 7f96b1ffb640 1 -- 172.21.15.171:0/24592820 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18ff6471-ce6d-4c23-ac7c-415157ed598e", "id": 5}]=0 v8) ==== 112+0+2 (unknown 2110640305 0 1988412176) 0x7f96b80059a0 con 0x7f96bc0a3a90 2024-09-16T04:06:22.834 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.832+0000 7f96b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "18ff6471-ce6d-4c23-ac7c-415157ed598e", "id": 5}] 2024-09-16T04:06:22.834 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.832+0000 7f96b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:22.835 INFO:teuthology.orchestra.run.smithi171.stdout:5 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 10 monclient: shutdown 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 >> v1:172.21.15.171:6789/0 conn(0x7f96bc0a3a90 legacy=0x7f96bc09ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 shutdown_connections 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 >> 172.21.15.171:0/24592820 conn(0x7f96bc0a9a40 msgr2=0x7f96bc0a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 shutdown_connections 2024-09-16T04:06:22.836 INFO:teuthology.orchestra.run.smithi171.stderr:2024-09-16T04:06:22.833+0000 7f96c3991640 1 -- 172.21.15.171:0/24592820 wait complete. 2024-09-16T04:06:22.845 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T04:06:22.846 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T04:06:22.848 INFO:tasks.ceph.osd.0:Started 2024-09-16T04:06:22.849 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T04:06:22.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T04:06:22.851 INFO:tasks.ceph.osd.1:Started 2024-09-16T04:06:22.851 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T04:06:22.851 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T04:06:22.853 INFO:tasks.ceph.osd.2:Started 2024-09-16T04:06:22.854 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T04:06:22.854 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T04:06:22.856 INFO:tasks.ceph.osd.3:Started 2024-09-16T04:06:22.856 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T04:06:22.856 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T04:06:22.858 INFO:tasks.ceph.osd.4:Started 2024-09-16T04:06:22.859 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T04:06:22.859 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T04:06:22.861 INFO:tasks.ceph.osd.5:Started 2024-09-16T04:06:22.862 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:22.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.969+0000 7f3199b4b640 10 monclient: get_monmap_and_config 2024-09-16T04:06:22.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.969+0000 7f3199b4b640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.969+0000 7f3199b4b640 10 monclient: monmap: 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:22.970468+0000 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:22.970468+0000 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:22.972 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 1 Processor -- start 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 1 -- start start 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient: init 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f31940a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f31940a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f319413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a0e00 con 0x7f319413d060 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a10e0 con 0x7f31940a2410 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a4970 con 0x7f31940a1b10 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.970+0000 7f3199b4b640 10 monclient(hunting): authenticate will time out at 803.595886s 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f3192ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f31940a1b10 0x7f319413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33474/0 (socket says 172.21.15.148:33474) 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f3192ffd640 1 -- 172.21.15.148:0/4246689425 learned_addr learned my addr 172.21.15.148:0/4246689425 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091900801 0 0) 0x7f31940a0e00 con 0x7f319413d060 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): my global_id is 4164 2024-09-16T04:06:22.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f317c003520 con 0x7f319413d060 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69883487 0 0) 0x7f31940a4970 con 0x7f31940a1b10 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): my global_id is 4163 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31940a0e00 con 0x7f31940a1b10 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503341439 0 0) 0x7f31940a10e0 con 0x7f31940a2410 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient(hunting): my global_id is 4147 2024-09-16T04:06:22.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31940a4970 con 0x7f31940a2410 2024-09-16T04:06:22.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3276117764 0 0) 0x7f317c003520 con 0x7f319413d060 2024-09-16T04:06:22.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 >> v1:172.21.15.148:6790/0 conn(0x7f31940a1b10 legacy=0x7f319413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 >> v1:172.21.15.171:6789/0 conn(0x7f31940a2410 legacy=0x7f31940a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 monclient: found mon.noname-a 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319406dfd0 con 0x7f319413d060 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f318c001990 con 0x7f319413d060 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: dump: 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:22.987 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f3199b4b640 5 monclient: authenticate success, global_id 4164 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f3199b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f318c001cb0 con 0x7f319413d060 2024-09-16T04:06:22.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f3199b4b640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 1 -- 172.21.15.148:0/4246689425 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f318c0020c0 con 0x7f319413d060 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f3199b4b640 4 set_mon_vals no callback set 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.971+0000 7f31927fc640 10 monclient: dump: 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.989 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 10 monclient: shutdown 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 1 -- 172.21.15.148:0/4246689425 >> v1:172.21.15.148:6789/0 conn(0x7f319413d060 legacy=0x7f319413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 1 -- 172.21.15.148:0/4246689425 shutdown_connections 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 1 -- 172.21.15.148:0/4246689425 >> 172.21.15.148:0/4246689425 conn(0x7f31940a97b0 msgr2=0x7f31940a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 1 -- 172.21.15.148:0/4246689425 shutdown_connections 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 1 -- 172.21.15.148:0/4246689425 wait complete. 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 10 monclient: build_initial_monmap 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.972+0000 7f3199b4b640 10 monclient: monmap: 2024-09-16T04:06:22.990 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.973+0000 7f3199b4b640 1 Processor -- start 2024-09-16T04:06:22.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.973+0000 7f3199b4b640 1 -- start start 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.973+0000 7f3199b4b640 10 monclient: init 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.973+0000 7f3199b4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f31940a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f31940a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f319413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f319409e3b0 con 0x7f31940a1b10 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f31937fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f31940a1b10 0x7f319409b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43430/0 (socket says 172.21.15.148:43430) 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f31937fe640 1 -- 172.21.15.148:0/2517068868 learned_addr learned my addr 172.21.15.148:0/2517068868 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f319409c6e0 con 0x7f319413d060 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940a0130 con 0x7f31940a2410 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient(hunting): authenticate will time out at 803.599187s 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2390484051 0 0) 0x7f31840011d0 con 0x7f31940a1b10 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): my global_id is 4167 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3180003520 con 0x7f31940a1b10 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459253985 0 0) 0x7f31940a0130 con 0x7f31940a2410 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): my global_id is 4166 2024-09-16T04:06:22.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31840011d0 con 0x7f31940a2410 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 171244064 0 0) 0x7f3180003520 con 0x7f31940a1b10 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31940a0130 con 0x7f31940a1b10 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31840014c0 con 0x7f31940a1b10 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): dump: 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.994 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.995 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839275274 0 0) 0x7f319409c6e0 con 0x7f319413d060 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): my global_id is 4150 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3180003520 con 0x7f319413d060 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1706682656 0 0) 0x7f31840011d0 con 0x7f31940a2410 2024-09-16T04:06:22.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f319409c6e0 con 0x7f31940a2410 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3188001a70 con 0x7f31940a2410 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient(hunting): dump: 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:22.997 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1864681175 0 0) 0x7f31940a0130 con 0x7f31940a1b10 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 >> v1:172.21.15.148:6790/0 conn(0x7f31940a2410 legacy=0x7f319409bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 >> v1:172.21.15.171:6789/0 conn(0x7f319413d060 legacy=0x7f31941d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 monclient: found mon.a 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31940a54c0 con 0x7f31940a1b10 2024-09-16T04:06:22.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: _finish_auth 0 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 5 monclient: authenticate success, global_id 4167 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient: _renew_subs 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f319409d520 con 0x7f31940a1b10 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient: _renew_subs 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f319409f810 con 0x7f31940a1b10 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31840038a0 con 0x7f31940a1b10 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3184003c70 con 0x7f31940a1b10 2024-09-16T04:06:22.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f3191ffb640 4 set_mon_vals no callback set 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 10 monclient: dump: 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.000 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.974+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f31840042a0 con 0x7f31940a1b10 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.975+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 3167040044 0 0) 0x7f3184005d70 con 0x7f31940a1b10 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.975+0000 7f3199b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.975+0000 7f3199b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.975+0000 7f3199b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.975+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3174005080 con 0x7f31940a1b10 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.977+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f319409e3b0 con 0x7f31940a1b10 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.977+0000 7f317bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:23.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:22.977+0000 7f317bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f3199b4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f3199b4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f3199b4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3174005370 con 0x7f31940a1b10 2024-09-16T04:06:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f317bfff640 1 -- 172.21.15.148:0/2517068868 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 2506767339) 0x7f3184006080 con 0x7f31940a1b10 2024-09-16T04:06:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f317bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.033+0000 7f317bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:23.036 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:23.036 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":8,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:22.826228+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 10 monclient: shutdown 2024-09-16T04:06:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 >> v1:172.21.15.148:6789/0 conn(0x7f31940a1b10 legacy=0x7f319409b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 shutdown_connections 2024-09-16T04:06:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 >> 172.21.15.148:0/2517068868 conn(0x7f31940a97b0 msgr2=0x7f3194135cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 shutdown_connections 2024-09-16T04:06:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:23.034+0000 7f3199b4b640 1 -- 172.21.15.148:0/2517068868 wait complete. 2024-09-16T04:06:23.046 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T04:06:23.046 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T04:06:23.366 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:23.364+0000 7fd78a236040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T04:06:23.418 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:23.416+0000 7fd78a236040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T04:06:23.652 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:23.743 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.741+0000 7f66ad5d6640 10 monclient: get_monmap_and_config 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.741+0000 7f66ad5d6640 10 monclient: build_initial_monmap 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.741+0000 7f66ad5d6640 10 monclient: monmap: 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:06:23.742708+0000 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:06:23.742708+0000 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:23.744 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:23.745 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:23.745 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 1 Processor -- start 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 1 -- start start 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient: init 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f66a80a3a10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f66a80a1c20 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f66a80a4500 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a806e810 con 0x7f66a80a1c20 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a809fd60 con 0x7f66a80a4500 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a80a4ea0 con 0x7f66a80a3a10 2024-09-16T04:06:23.753 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.742+0000 7f66ad5d6640 10 monclient(hunting): authenticate will time out at 804.367923s 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f669e7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f66a80a1c20 0x7f66a813bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43436/0 (socket says 172.21.15.148:43436) 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f66a80a3a10 0x7f66a80a3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33496/0 (socket says 172.21.15.148:33496) 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f669e7fc640 1 -- 172.21.15.148:0/12351118 learned_addr learned my addr 172.21.15.148:0/12351118 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473931934 0 0) 0x7f66a80a4ea0 con 0x7f66a80a3a10 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): my global_id is 4169 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6694003520 con 0x7f66a80a3a10 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174850235 0 0) 0x7f66a806e810 con 0x7f66a80a1c20 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:23.754 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): my global_id is 4170 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a80a4ea0 con 0x7f66a80a1c20 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1390799696 0 0) 0x7f66a809fd60 con 0x7f66a80a4500 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient(hunting): my global_id is 4153 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a806e810 con 0x7f66a80a4500 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3261928135 0 0) 0x7f6694003520 con 0x7f66a80a3a10 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 >> v1:172.21.15.171:6789/0 conn(0x7f66a80a4500 legacy=0x7f66a813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 >> v1:172.21.15.148:6789/0 conn(0x7f66a80a1c20 legacy=0x7f66a813bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.755 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 monclient: found mon.noname-b 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a80a11f0 con 0x7f66a80a3a10 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66ad5d6640 5 monclient: authenticate success, global_id 4169 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6698001610 con 0x7f66a80a3a10 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66ad5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:23.756 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: dump: 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66ad5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6698001930 con 0x7f66a80a3a10 2024-09-16T04:06:23.757 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 1 -- 172.21.15.148:0/12351118 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6698001d40 con 0x7f66a80a3a10 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66ad5d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66ad5d6640 4 set_mon_vals no callback set 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.743+0000 7f66a67fc640 10 monclient: dump: 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.758 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 10 monclient: shutdown 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 1 -- 172.21.15.148:0/12351118 >> v1:172.21.15.148:6790/0 conn(0x7f66a80a3a10 legacy=0x7f66a80a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 1 -- 172.21.15.148:0/12351118 shutdown_connections 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 1 -- 172.21.15.148:0/12351118 >> 172.21.15.148:0/12351118 conn(0x7f66a80a9970 msgr2=0x7f66a80a5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 1 -- 172.21.15.148:0/12351118 shutdown_connections 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 1 -- 172.21.15.148:0/12351118 wait complete. 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 10 monclient: build_initial_monmap 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.744+0000 7f66ad5d6640 10 monclient: monmap: 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2024-09-16T04:06:23.759 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 1 Processor -- start 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 1 -- start start 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient: init 2024-09-16T04:06:23.760 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f66a80a1c20 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f66a80a3a10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f66a80a4500 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a813fcc0 con 0x7f66a80a3a10 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a813ea90 con 0x7f66a80a4500 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a813ed70 con 0x7f66a80a1c20 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66ad5d6640 10 monclient(hunting): authenticate will time out at 804.370703s 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66a6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f66a80a1c20 0x7f66a813b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33510/0 (socket says 172.21.15.148:33510) 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f66a6ffd640 1 -- 172.21.15.148:0/1970067930 learned_addr learned my addr 172.21.15.148:0/1970067930 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:23.761 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279113061 0 0) 0x7f66a813ed70 con 0x7f66a80a1c20 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): my global_id is 4172 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6688003520 con 0x7f66a80a1c20 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419092571 0 0) 0x7f66a813fcc0 con 0x7f66a80a3a10 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 10 monclient(hunting): my global_id is 4173 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.745+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a813ed70 con 0x7f66a80a3a10 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022796118 0 0) 0x7f66a813ea90 con 0x7f66a80a4500 2024-09-16T04:06:23.762 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): my global_id is 4156 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a813fcc0 con 0x7f66a80a4500 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4210862932 0 0) 0x7f6688003520 con 0x7f66a80a1c20 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66a813ea90 con 0x7f66a80a1c20 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f66980015d0 con 0x7f66a80a1c20 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): dump: 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.763 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3468314551 0 0) 0x7f66a813ed70 con 0x7f66a80a3a10 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6688003520 con 0x7f66a80a3a10 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f66900010b0 con 0x7f66a80a3a10 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:23.764 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): dump: 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3483594743 0 0) 0x7f66a813fcc0 con 0x7f66a80a4500 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66a813ed70 con 0x7f66a80a4500 2024-09-16T04:06:23.765 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f66a0001c30 con 0x7f66a80a4500 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient(hunting): dump: 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.766 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 70309886 0 0) 0x7f66a813ea90 con 0x7f66a80a1c20 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 >> v1:172.21.15.171:6789/0 conn(0x7f66a80a4500 legacy=0x7f66a81d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 >> v1:172.21.15.148:6789/0 conn(0x7f66a80a3a10 legacy=0x7f66a813b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 monclient: found mon.c 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a813bf30 con 0x7f66a80a1c20 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: _finish_auth 0 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 5 monclient: authenticate success, global_id 4172 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 10 monclient: _renew_subs 2024-09-16T04:06:23.767 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f66a813dae0 con 0x7f66a80a1c20 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 10 monclient: _renew_subs 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f66a813d640 con 0x7f66a80a1c20 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6698004c00 con 0x7f66a80a1c20 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6698004f50 con 0x7f66a80a1c20 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 10 monclient: dump: 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:23.768 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f66a5ffb640 4 set_mon_vals no callback set 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6698005480 con 0x7f66a80a1c20 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.746+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 3167040044 0 0) 0x7f6698005d60 con 0x7f66a80a1c20 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.747+0000 7f66ad5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.747+0000 7f66ad5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.747+0000 7f66ad5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:23.769 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.747+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a80a0200 con 0x7f66a80a1c20 2024-09-16T04:06:23.770 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.749+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (unknown 1092875540 0 80963817) 0x7f6698005770 con 0x7f66a80a1c20 2024-09-16T04:06:23.770 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.749+0000 7f669ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:23.770 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.749+0000 7f669ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:23.781 INFO:tasks.ceph.osd.1.smithi148.stderr:2024-09-16T04:06:23.779+0000 7fa9db345640 -1 Falling back to public interface 2024-09-16T04:06:23.783 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:23.782+0000 7fd78a236040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.807+0000 7f66ad5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.807+0000 7f66ad5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.807+0000 7f66ad5d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.807+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66a8075560 con 0x7f66a80a1c20 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.808+0000 7f669ffff640 1 -- 172.21.15.148:0/1970067930 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 2506767339) 0x7f6698005950 con 0x7f66a80a1c20 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.808+0000 7f669ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:23.810 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.808+0000 7f669ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:23.811 INFO:teuthology.misc.health.smithi148.stdout: 2024-09-16T04:06:23.811 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":8,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:22.826228+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:23.811 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 10 monclient: shutdown 2024-09-16T04:06:23.811 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:23.812 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 >> v1:172.21.15.148:6790/0 conn(0x7f66a80a1c20 legacy=0x7f66a813b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:23.812 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 shutdown_connections 2024-09-16T04:06:23.812 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 >> 172.21.15.148:0/1970067930 conn(0x7f66a80a9970 msgr2=0x7f66a809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:23.812 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 shutdown_connections 2024-09-16T04:06:23.812 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:23.809+0000 7f66ad5d6640 1 -- 172.21.15.148:0/1970067930 wait complete. 2024-09-16T04:06:23.820 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T04:06:23.870 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:23.868+0000 7fd78a236040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T04:06:23.977 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:23.975+0000 7fd78a236040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T04:06:24.023 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.021+0000 7fd78a236040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T04:06:24.054 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-09-16T04:06:24.052+0000 7f34a0a31640 -1 Falling back to public interface 2024-09-16T04:06:24.060 INFO:tasks.ceph.osd.2.smithi148.stderr:2024-09-16T04:06:24.058+0000 7f0c08830640 -1 Falling back to public interface 2024-09-16T04:06:24.138 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.137+0000 7fd78a236040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T04:06:24.192 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.190+0000 7fd78a236040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T04:06:24.236 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.234+0000 7fd78a236040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T04:06:24.308 INFO:tasks.ceph.osd.0.smithi148.stderr:2024-09-16T04:06:24.306+0000 7f3a4a41c640 -1 Falling back to public interface 2024-09-16T04:06:24.324 INFO:tasks.ceph.osd.3.smithi171.stderr:2024-09-16T04:06:24.322+0000 7f4443b4d640 -1 Falling back to public interface 2024-09-16T04:06:24.325 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-09-16T04:06:24.323+0000 7feeb9517640 -1 Falling back to public interface 2024-09-16T04:06:24.349 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.347+0000 7fd78a236040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T04:06:24.400 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.398+0000 7fd78a236040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T04:06:24.451 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.449+0000 7fd78a236040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T04:06:24.519 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.516+0000 7fd78a236040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T04:06:24.563 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.561+0000 7fd78a236040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T04:06:24.607 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:24.605+0000 7fd78a236040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T04:06:25.261 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:25.259+0000 7fd78a236040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T04:06:26.273 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:26.271+0000 7fd7299d6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T04:06:26.273 INFO:tasks.ceph.mgr.x.smithi171.stderr:2024-09-16T04:06:26.271+0000 7fd7299d6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T04:06:27.869 INFO:tasks.ceph.osd.1.smithi148.stderr:2024-09-16T04:06:27.867+0000 7fa9db345640 -1 osd.1 0 log_to_monitors true 2024-09-16T04:06:28.086 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-09-16T04:06:28.084+0000 7f34a0a31640 -1 osd.5 0 log_to_monitors true 2024-09-16T04:06:28.099 INFO:tasks.ceph.osd.0.smithi148.stderr:2024-09-16T04:06:28.097+0000 7f3a4a41c640 -1 osd.0 0 log_to_monitors true 2024-09-16T04:06:28.150 INFO:tasks.ceph.osd.2.smithi148.stderr:2024-09-16T04:06:28.149+0000 7f0c08830640 -1 osd.2 0 log_to_monitors true 2024-09-16T04:06:28.166 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-09-16T04:06:28.165+0000 7feeb9517640 -1 osd.4 0 log_to_monitors true 2024-09-16T04:06:28.181 INFO:tasks.ceph.osd.3.smithi171.stderr:2024-09-16T04:06:28.180+0000 7f4443b4d640 -1 osd.3 0 log_to_monitors true 2024-09-16T04:06:29.282 INFO:tasks.ceph.osd.3.smithi171.stderr:2024-09-16T04:06:29.280+0000 7f443fadc640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T04:06:29.284 INFO:tasks.ceph.osd.2.smithi148.stderr:2024-09-16T04:06:29.282+0000 7f0c03d72640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:06:29.284 INFO:tasks.ceph.osd.2.smithi148.stderr:2024-09-16T04:06:29.282+0000 7f0c03d72640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:06:29.285 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-09-16T04:06:29.283+0000 7f349c9c0640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:06:29.285 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-09-16T04:06:29.283+0000 7f349c9c0640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:06:29.285 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-09-16T04:06:29.283+0000 7f349d1d4640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T04:06:29.286 INFO:tasks.ceph.osd.0.smithi148.stderr:2024-09-16T04:06:29.283+0000 7f3a45da1640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:06:29.286 INFO:tasks.ceph.osd.0.smithi148.stderr:2024-09-16T04:06:29.283+0000 7f3a45da1640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:06:29.286 INFO:tasks.ceph.osd.0.smithi148.stderr:2024-09-16T04:06:29.284+0000 7f3a465b5640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T04:06:29.286 INFO:tasks.ceph.osd.3.smithi171.stderr:2024-09-16T04:06:29.284+0000 7f443b904640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:06:29.287 INFO:tasks.ceph.osd.2.smithi148.stderr:2024-09-16T04:06:29.285+0000 7f0c04586640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T04:06:29.289 INFO:tasks.ceph.osd.5.smithi171.stderr:2024-09-16T04:06:29.287+0000 7f34987e8640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:06:29.289 INFO:tasks.ceph.osd.0.smithi148.stderr:2024-09-16T04:06:29.287+0000 7f3a41bc9640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:06:29.296 INFO:tasks.ceph.osd.2.smithi148.stderr:2024-09-16T04:06:29.295+0000 7f0bffb9a640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:06:30.425 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:30.517 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.515+0000 7fa423d2c640 10 monclient: get_monmap_and_config 2024-09-16T04:06:30.517 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.515+0000 7fa423d2c640 10 monclient: build_initial_monmap 2024-09-16T04:06:30.517 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.515+0000 7fa423d2c640 10 monclient: monmap: 2024-09-16T04:06:30.517 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2024-09-16T04:06:30.517 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:06:30.516435+0000 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:06:30.516435+0000 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:30.518 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 1 Processor -- start 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 1 -- start start 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient: init 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient(hunting): picked mon.noname-a con 0x7fa41c0a3a10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fa41c0a1c20 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fa41c0a4500 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c06e810 con 0x7fa41c0a3a10 2024-09-16T04:06:30.529 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c09fd60 con 0x7fa41c0a1c20 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c0a4ea0 con 0x7fa41c0a4500 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa423d2c640 10 monclient(hunting): authenticate will time out at 811.141908s 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa421ac8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa41c0a3a10 0x7fa41c0a3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43490/0 (socket says 172.21.15.148:43490) 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.516+0000 7fa421ac8640 1 -- 172.21.15.148:0/3349611389 learned_addr learned my addr 172.21.15.148:0/3349611389 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494899338 0 0) 0x7fa41c0a4ea0 con 0x7fa41c0a4500 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): my global_id is 4208 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa408003520 con 0x7fa41c0a4500 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421338119 0 0) 0x7fa41c06e810 con 0x7fa41c0a3a10 2024-09-16T04:06:30.530 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): my global_id is 4209 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa41c0a4ea0 con 0x7fa41c0a3a10 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742530292 0 0) 0x7fa41c09fd60 con 0x7fa41c0a1c20 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient(hunting): my global_id is 4192 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa41c06e810 con 0x7fa41c0a1c20 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1541880709 0 0) 0x7fa408003520 con 0x7fa41c0a4500 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 >> v1:172.21.15.171:6789/0 conn(0x7fa41c0a1c20 legacy=0x7fa41c13bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.531 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 >> v1:172.21.15.148:6789/0 conn(0x7fa41c0a3a10 legacy=0x7fa41c0a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: _finish_hunting 0 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 monclient: found mon.noname-b 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c0a11f0 con 0x7fa41c0a4500 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: _finish_auth 0 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: _check_auth_tickets 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4140018c0 con 0x7fa41c0a4500 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa423d2c640 5 monclient: authenticate success, global_id 4208 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:30.532 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: dump: 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa423d2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa414001b20 con 0x7fa41c0a4500 2024-09-16T04:06:30.533 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa423d2c640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 1 -- 172.21.15.148:0/3349611389 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa414001f30 con 0x7fa41c0a4500 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa423d2c640 4 set_mon_vals no callback set 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.517+0000 7fa420ac6640 10 monclient: dump: 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.534 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 10 monclient: shutdown 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 1 -- 172.21.15.148:0/3349611389 >> v1:172.21.15.148:6790/0 conn(0x7fa41c0a4500 legacy=0x7fa41c13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 1 -- 172.21.15.148:0/3349611389 shutdown_connections 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 1 -- 172.21.15.148:0/3349611389 >> 172.21.15.148:0/3349611389 conn(0x7fa41c0a9970 msgr2=0x7fa41c0a5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 1 -- 172.21.15.148:0/3349611389 shutdown_connections 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 1 -- 172.21.15.148:0/3349611389 wait complete. 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 10 monclient: build_initial_monmap 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.518+0000 7fa423d2c640 10 monclient: monmap: 2024-09-16T04:06:30.535 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 1 Processor -- start 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 1 -- start start 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient: init 2024-09-16T04:06:30.536 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient(hunting): picked mon.noname-a con 0x7fa41c0a1c20 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient(hunting): picked mon.noname-b con 0x7fa41c0a3a10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient(hunting): picked mon.noname-c con 0x7fa41c0a4500 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c13efd0 con 0x7fa41c0a1c20 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c13c140 con 0x7fa41c0a3a10 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c13db20 con 0x7fa41c0a4500 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa423d2c640 10 monclient(hunting): authenticate will time out at 811.144709s 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa421ac8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa41c0a1c20 0x7fa41c13b320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:43498/0 (socket says 172.21.15.148:43498) 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa421ac8640 1 -- 172.21.15.148:0/1698823828 learned_addr learned my addr 172.21.15.148:0/1698823828 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:30.537 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939256268 0 0) 0x7fa41c13db20 con 0x7fa41c0a4500 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa4127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa4127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa4127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.519+0000 7fa4127fc640 10 monclient(hunting): my global_id is 4211 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa400003520 con 0x7fa41c0a4500 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663724932 0 0) 0x7fa41c13efd0 con 0x7fa41c0a1c20 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): my global_id is 4212 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa41c13db20 con 0x7fa41c0a1c20 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755357757 0 0) 0x7fa41c13c140 con 0x7fa41c0a3a10 2024-09-16T04:06:30.538 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): my global_id is 4195 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa41c13efd0 con 0x7fa41c0a3a10 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3063778987 0 0) 0x7fa400003520 con 0x7fa41c0a4500 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa41c13c140 con 0x7fa41c0a4500 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa414001eb0 con 0x7fa41c0a4500 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): dump: 2024-09-16T04:06:30.539 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4031939848 0 0) 0x7fa41c13db20 con 0x7fa41c0a1c20 2024-09-16T04:06:30.540 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa400003520 con 0x7fa41c0a1c20 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1060270364 0 0) 0x7fa41c13efd0 con 0x7fa41c0a3a10 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa41c13db20 con 0x7fa41c0a3a10 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa418001990 con 0x7fa41c0a1c20 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): dump: 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.541 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa40c001830 con 0x7fa41c0a3a10 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient(hunting): dump: 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.542 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2189659746 0 0) 0x7fa41c13c140 con 0x7fa41c0a4500 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 >> v1:172.21.15.171:6789/0 conn(0x7fa41c0a3a10 legacy=0x7fa41c13ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 >> v1:172.21.15.148:6789/0 conn(0x7fa41c0a1c20 legacy=0x7fa41c13b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 monclient: found mon.c 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:30.543 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c13f2b0 con 0x7fa41c0a4500 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 5 monclient: authenticate success, global_id 4211 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 10 monclient: _renew_subs 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c0a8160 con 0x7fa41c0a4500 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 10 monclient: _renew_subs 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa41c13ecf0 con 0x7fa41c0a4500 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa414003c70 con 0x7fa41c0a4500 2024-09-16T04:06:30.544 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa414003e20 con 0x7fa41c0a4500 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa413fff640 4 set_mon_vals no callback set 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.520+0000 7fa4127fc640 10 monclient: dump: 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:30.545 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 4) ==== 45646+0+0 (unknown 2050619950 0 0) 0x7fa414003e20 con 0x7fa41c0a4500 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa423d2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa423d2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa423d2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa404005080 con 0x7fa41c0a4500 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(11..11 src has 1..11) ==== 2487+0+0 (unknown 2314064180 0 0) 0x7fa4140456b0 con 0x7fa41c0a4500 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.521+0000 7fa421ac8640 10 monclient: get_auth_request con 0x7fa400039100 auth_method 0 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.526+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa414014c00 con 0x7fa41c0a4500 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.526+0000 7fa4127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:30.546 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.526+0000 7fa4127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:30.704 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa423d2c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:30.704 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa423d2c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:30.704 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa423d2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:30.704 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa404005370 con 0x7fa41c0a4500 2024-09-16T04:06:30.705 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa4127fc640 1 -- 172.21.15.148:0/1698823828 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6565 (unknown 238102980 0 3226949915) 0x7fa414014500 con 0x7fa41c0a4500 2024-09-16T04:06:30.705 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa4127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:30.705 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.702+0000 7fa4127fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:30.705 INFO:teuthology.misc.health.smithi148.stdout: 2024-09-16T04:06:30.705 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":11,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:30.274812+0000","last_up_change":"2024-09-16T04:06:30.274812+0000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":985869260}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":985869260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":985869260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":985869260}]},"public_addr":"172.21.15.148:6808/985869260","cluster_addr":"172.21.15.148:6809/985869260","heartbeat_back_addr":"172.21.15.148:6811/985869260","heartbeat_front_addr":"172.21.15.148:6810/985869260","state":["exists","up"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":262588452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":262588452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":262588452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":262588452}]},"public_addr":"172.21.15.148:6804/262588452","cluster_addr":"172.21.15.148:6805/262588452","heartbeat_back_addr":"172.21.15.148:6807/262588452","heartbeat_front_addr":"172.21.15.148:6806/262588452","state":["exists","up"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":772155526}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":772155526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":772155526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":772155526}]},"public_addr":"172.21.15.171:6804/772155526","cluster_addr":"172.21.15.171:6805/772155526","heartbeat_back_addr":"172.21.15.171:6807/772155526","heartbeat_front_addr":"172.21.15.171:6806/772155526","state":["exists","up"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":466524840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":466524840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":466524840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":466524840}]},"public_addr":"172.21.15.171:6800/466524840","cluster_addr":"172.21.15.171:6801/466524840","heartbeat_back_addr":"172.21.15.171:6803/466524840","heartbeat_front_addr":"172.21.15.171:6802/466524840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.704+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 >> v1:172.21.15.171:6812/7327273 conn(0x7fa400039100 legacy=0x7fa40003b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.704+0000 7fa423d2c640 10 monclient: shutdown 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.704+0000 7fa423d2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.704+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 >> v1:172.21.15.148:6790/0 conn(0x7fa41c0a4500 legacy=0x7fa41c1d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.704+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 shutdown_connections 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.705+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 >> 172.21.15.148:0/1698823828 conn(0x7fa41c0a9970 msgr2=0x7fa41c09f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.705+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 shutdown_connections 2024-09-16T04:06:30.707 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:30.705+0000 7fa423d2c640 1 -- 172.21.15.148:0/1698823828 wait complete. 2024-09-16T04:06:30.716 DEBUG:teuthology.misc:4 of 6 OSDs are up 2024-09-16T04:06:31.922 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-09-16T04:06:31.920+0000 7feeb54a6640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T04:06:31.924 INFO:tasks.ceph.osd.4.smithi171.stderr:2024-09-16T04:06:31.923+0000 7feead2c6640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:06:31.947 INFO:tasks.ceph.osd.1.smithi148.stderr:2024-09-16T04:06:31.945+0000 7fa9d7ae8640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T04:06:31.951 INFO:tasks.ceph.osd.1.smithi148.stderr:2024-09-16T04:06:31.949+0000 7fa9cf0f4640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T04:06:37.322 DEBUG:teuthology.orchestra.run.smithi148:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:37.422 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.420+0000 7ff0b19e4640 10 monclient: get_monmap_and_config 2024-09-16T04:06:37.422 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.420+0000 7ff0b19e4640 10 monclient: build_initial_monmap 2024-09-16T04:06:37.422 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.420+0000 7ff0b19e4640 10 monclient: monmap: 2024-09-16T04:06:37.422 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2024-09-16T04:06:37.422 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:06:37.421106+0000 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:06:37.421106+0000 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:37.423 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.431 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.420+0000 7ff0b19e4640 1 Processor -- start 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.420+0000 7ff0b19e4640 1 -- start start 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.420+0000 7ff0b19e4640 10 monclient: init 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0ac0a3a10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient(hunting): picked mon.noname-a con 0x7ff0ac0a1c20 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac0a4500 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac06e810 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac09fd60 con 0x7ff0ac0a4500 2024-09-16T04:06:37.432 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac0a4ea0 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 10 monclient(hunting): authenticate will time out at 818.046322s 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0aaffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff0ac0a3a10 0x7ff0ac0a3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55698/0 (socket says 172.21.15.148:55698) 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0aaffd640 1 -- 172.21.15.148:0/3755268617 learned_addr learned my addr 172.21.15.148:0/3755268617 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0aa7fc640 1 --1- 172.21.15.148:0/3755268617 >> v1:172.21.15.148:6789/0 conn(0x7ff0ac0a1c20 0x7ff0ac13bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54666/0 (socket says 172.21.15.148:54666) 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964938949 0 0) 0x7ff0ac06e810 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): my global_id is 4224 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff098003520 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.433 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2381908270 0 0) 0x7ff0ac0a4ea0 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): my global_id is 4223 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac06e810 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665821804 0 0) 0x7ff0ac09fd60 con 0x7ff0ac0a4500 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient(hunting): my global_id is 4207 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac0a4ea0 con 0x7ff0ac0a4500 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3674702775 0 0) 0x7ff098003520 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.434 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 >> v1:172.21.15.148:6790/0 conn(0x7ff0ac0a3a10 legacy=0x7ff0ac0a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 >> v1:172.21.15.171:6789/0 conn(0x7ff0ac0a4500 legacy=0x7ff0ac13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 monclient: found mon.noname-a 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac0a11f0 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 5 monclient: authenticate success, global_id 4224 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff094001540 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0b19e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:37.435 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.421+0000 7ff0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 10 monclient: dump: 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.436 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff094001860 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 1 -- 172.21.15.148:0/3755268617 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff094001c30 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 4 set_mon_vals no callback set 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0a9ffb640 10 monclient: dump: 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:37.437 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 10 monclient: shutdown 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 1 -- 172.21.15.148:0/3755268617 >> v1:172.21.15.148:6789/0 conn(0x7ff0ac0a1c20 legacy=0x7ff0ac13bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 1 -- 172.21.15.148:0/3755268617 shutdown_connections 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 1 -- 172.21.15.148:0/3755268617 >> 172.21.15.148:0/3755268617 conn(0x7ff0ac0a9970 msgr2=0x7ff0ac0a5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:37.438 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 1 -- 172.21.15.148:0/3755268617 shutdown_connections 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.422+0000 7ff0b19e4640 1 -- 172.21.15.148:0/3755268617 wait complete. 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.423+0000 7ff0b19e4640 10 monclient: build_initial_monmap 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.423+0000 7ff0b19e4640 10 monclient: monmap: 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:epoch 0 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:created 0.000000 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:37.439 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.423+0000 7ff0b19e4640 1 Processor -- start 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.423+0000 7ff0b19e4640 1 -- start start 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.423+0000 7ff0b19e4640 10 monclient: init 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0ac0a1c20 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient(hunting): picked mon.noname-a con 0x7ff0ac0a3a10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0ac0a4500 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac13f430 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac0a0200 con 0x7ff0ac0a4500 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac13f150 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0b19e4640 10 monclient(hunting): authenticate will time out at 818.049395s 2024-09-16T04:06:37.440 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0aa7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff0ac0a3a10 0x7ff0ac1d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54668/0 (socket says 172.21.15.148:54668) 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0aa7fc640 1 -- 172.21.15.148:0/1778543400 learned_addr learned my addr 172.21.15.148:0/1778543400 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008391894 0 0) 0x7ff0ac13f430 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): my global_id is 4227 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff08c003520 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886859553 0 0) 0x7ff0ac13f150 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): my global_id is 4226 2024-09-16T04:06:37.441 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac13f430 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2439129086 0 0) 0x7ff0ac0a0200 con 0x7ff0ac0a4500 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): my global_id is 4210 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0ac13f150 con 0x7ff0ac0a4500 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1450527721 0 0) 0x7ff08c003520 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0ac0a0200 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff094001500 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): dump: 2024-09-16T04:06:37.442 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3018321025 0 0) 0x7ff0ac13f430 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.443 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff08c003520 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0a4001b10 con 0x7ff0ac0a1c20 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.424+0000 7ff0a3fff640 10 monclient(hunting): dump: 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.444 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4111018282 0 0) 0x7ff0ac13f150 con 0x7ff0ac0a4500 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0ac13f430 con 0x7ff0ac0a4500 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff09c001870 con 0x7ff0ac0a4500 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient(hunting): dump: 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.445 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.446 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.446 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4033735359 0 0) 0x7ff0ac0a0200 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 >> v1:172.21.15.148:6790/0 conn(0x7ff0ac0a1c20 legacy=0x7ff0ac1cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 >> v1:172.21.15.171:6789/0 conn(0x7ff0ac0a4500 legacy=0x7ff0ac1d4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 monclient: found mon.a 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:37.448 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac13f710 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 5 monclient: authenticate success, global_id 4227 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 10 monclient: _renew_subs 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac13dd00 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 10 monclient: _renew_subs 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0ac13c790 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.449 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff094001860 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff094004b80 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a97fa640 4 set_mon_vals no callback set 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.425+0000 7ff0a3fff640 10 monclient: dump: 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:epoch 1 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.450 INFO:teuthology.misc.health.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr: 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff094004b80 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0b19e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0b19e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0b19e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff090005080 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 3153+0+0 (unknown 1072785214 0 0) 0x7ff094046a10 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.451 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.426+0000 7ff0aaffd640 10 monclient: get_auth_request con 0x7ff08c039100 auth_method 0 2024-09-16T04:06:37.452 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.431+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff094015b20 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.452 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.431+0000 7ff0a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:37.452 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.431+0000 7ff0a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:37.598 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.596+0000 7ff0b19e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:37.598 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.596+0000 7ff0b19e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:37.599 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.596+0000 7ff0b19e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.599 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.596+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff090005370 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.599 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.597+0000 7ff0a3fff640 1 -- 172.21.15.148:0/1778543400 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9049 (unknown 92231148 0 1536579737) 0x7ff094015420 con 0x7ff0ac0a3a10 2024-09-16T04:06:37.599 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.597+0000 7ff0a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:37.599 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.597+0000 7ff0a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:37.600 INFO:teuthology.misc.health.smithi148.stdout: 2024-09-16T04:06:37.600 INFO:teuthology.misc.health.smithi148.stdout:{"epoch":14,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:33.307994+0000","last_up_change":"2024-09-16T04:06:32.301387+0000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:06:31.297815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":985869260}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":985869260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":985869260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":985869260}]},"public_addr":"172.21.15.148:6808/985869260","cluster_addr":"172.21.15.148:6809/985869260","heartbeat_back_addr":"172.21.15.148:6811/985869260","heartbeat_front_addr":"172.21.15.148:6810/985869260","state":["exists","up"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":3044422561}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":3044422561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":3044422561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":3044422561}]},"public_addr":"172.21.15.148:6800/3044422561","cluster_addr":"172.21.15.148:6801/3044422561","heartbeat_back_addr":"172.21.15.148:6803/3044422561","heartbeat_front_addr":"172.21.15.148:6802/3044422561","state":["exists","up"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":262588452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":262588452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":262588452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":262588452}]},"public_addr":"172.21.15.148:6804/262588452","cluster_addr":"172.21.15.148:6805/262588452","heartbeat_back_addr":"172.21.15.148:6807/262588452","heartbeat_front_addr":"172.21.15.148:6806/262588452","state":["exists","up"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":772155526}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":772155526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":772155526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":772155526}]},"public_addr":"172.21.15.171:6804/772155526","cluster_addr":"172.21.15.171:6805/772155526","heartbeat_back_addr":"172.21.15.171:6807/772155526","heartbeat_front_addr":"172.21.15.171:6806/772155526","state":["exists","up"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":4274747803}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":4274747803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":4274747803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":4274747803}]},"public_addr":"172.21.15.171:6808/4274747803","cluster_addr":"172.21.15.171:6809/4274747803","heartbeat_back_addr":"172.21.15.171:6811/4274747803","heartbeat_front_addr":"172.21.15.171:6810/4274747803","state":["exists","up"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":466524840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":466524840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":466524840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":466524840}]},"public_addr":"172.21.15.171:6800/466524840","cluster_addr":"172.21.15.171:6801/466524840","heartbeat_back_addr":"172.21.15.171:6803/466524840","heartbeat_front_addr":"172.21.15.171:6802/466524840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.060468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:28.867753+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140301+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.136223+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.080093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:37.601 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 >> v1:172.21.15.171:6812/7327273 conn(0x7ff08c039100 legacy=0x7ff08c03b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.601 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 10 monclient: shutdown 2024-09-16T04:06:37.602 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:37.602 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 >> v1:172.21.15.148:6789/0 conn(0x7ff0ac0a3a10 legacy=0x7ff0ac1d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.602 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 shutdown_connections 2024-09-16T04:06:37.602 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 >> 172.21.15.148:0/1778543400 conn(0x7ff0ac0a9970 msgr2=0x7ff0ac0a28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:37.602 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 shutdown_connections 2024-09-16T04:06:37.602 INFO:teuthology.misc.health.smithi148.stderr:2024-09-16T04:06:37.599+0000 7ff0b19e4640 1 -- 172.21.15.148:0/1778543400 wait complete. 2024-09-16T04:06:37.610 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T04:06:37.610 INFO:tasks.ceph:Creating RBD pool 2024-09-16T04:06:37.611 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T04:06:37.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.706+0000 7f72a185e640 10 monclient: get_monmap_and_config 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.706+0000 7f72a185e640 10 monclient: build_initial_monmap 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.706+0000 7f72a185e640 10 monclient: monmap: 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:37.707072+0000 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:37.707072+0000 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:37.709 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:37.710 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.706+0000 7f72a185e640 1 Processor -- start 2024-09-16T04:06:37.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.706+0000 7f72a185e640 1 -- start start 2024-09-16T04:06:37.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.706+0000 7f72a185e640 10 monclient: init 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient(hunting): picked mon.noname-a con 0x7f729c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient(hunting): picked mon.noname-c con 0x7f729c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient(hunting): picked mon.noname-b con 0x7f729c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c0a0e00 con 0x7f729c0a2410 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c0a10e0 con 0x7f729c0a1b10 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c0a4970 con 0x7f729c13d060 2024-09-16T04:06:37.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 10 monclient(hunting): authenticate will time out at 818.332309s 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729affd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f729c0a2410 0x7f729c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54676/0 (socket says 172.21.15.148:54676) 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729affd640 1 -- 172.21.15.148:0/142979054 learned_addr learned my addr 172.21.15.148:0/142979054 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729b7fe640 1 --1- 172.21.15.148:0/142979054 >> v1:172.21.15.148:6790/0 conn(0x7f729c13d060 0x7f729c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55718/0 (socket says 172.21.15.148:55718) 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793074471 0 0) 0x7f729c0a0e00 con 0x7f729c0a2410 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): my global_id is 4230 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7288003520 con 0x7f729c0a2410 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972703296 0 0) 0x7f729c0a4970 con 0x7f729c13d060 2024-09-16T04:06:37.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient(hunting): my global_id is 4229 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f729c0a0e00 con 0x7f729c13d060 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2216967759 0 0) 0x7f7288003520 con 0x7f729c0a2410 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 >> v1:172.21.15.148:6790/0 conn(0x7f729c13d060 legacy=0x7f729c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 >> v1:172.21.15.171:6789/0 conn(0x7f729c0a1b10 legacy=0x7f729c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 monclient: found mon.noname-a 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c06dfd0 con 0x7f729c0a2410 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:37.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f728c001610 con 0x7f729c0a2410 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 5 monclient: authenticate success, global_id 4230 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 10 monclient: dump: 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.721 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f72a185e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.707+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f728c0018f0 con 0x7f729c0a2410 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f729a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f729a7fc640 1 -- 172.21.15.148:0/142979054 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f728c001d00 con 0x7f729c0a2410 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 4 set_mon_vals no callback set 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f729a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f729a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f729a7fc640 10 monclient: dump: 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.722 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 10 monclient: shutdown 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 1 -- 172.21.15.148:0/142979054 >> v1:172.21.15.148:6789/0 conn(0x7f729c0a2410 legacy=0x7f729c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 1 -- 172.21.15.148:0/142979054 shutdown_connections 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 1 -- 172.21.15.148:0/142979054 >> 172.21.15.148:0/142979054 conn(0x7f729c0a97b0 msgr2=0x7f729c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 1 -- 172.21.15.148:0/142979054 shutdown_connections 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.708+0000 7f72a185e640 1 -- 172.21.15.148:0/142979054 wait complete. 2024-09-16T04:06:37.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 10 monclient: build_initial_monmap 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 10 monclient: monmap: 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 1 Processor -- start 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 1 -- start start 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 10 monclient: init 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.709+0000 7f72a185e640 10 monclient(hunting): picked mon.noname-c con 0x7f729c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient(hunting): picked mon.noname-a con 0x7f729c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient(hunting): picked mon.noname-b con 0x7f729c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c139ae0 con 0x7f729c0a2410 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c1378a0 con 0x7f729c13d060 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729c138350 con 0x7f729c0a1b10 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient(hunting): authenticate will time out at 818.335077s 2024-09-16T04:06:37.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f729affd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f729c0a1b10 0x7f729c136a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55730/0 (socket says 172.21.15.148:55730) 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72927fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f729c0a2410 0x7f729c137190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54682/0 (socket says 172.21.15.148:54682) 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f729affd640 1 -- 172.21.15.148:0/3026246275 learned_addr learned my addr 172.21.15.148:0/3026246275 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1218009337 0 0) 0x7f729c138350 con 0x7f729c0a1b10 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): my global_id is 4232 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f727c003520 con 0x7f729c0a1b10 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969649435 0 0) 0x7f729c139ae0 con 0x7f729c0a2410 2024-09-16T04:06:37.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): my global_id is 4233 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f729c138350 con 0x7f729c0a2410 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262043742 0 0) 0x7f729c1378a0 con 0x7f729c13d060 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): my global_id is 4216 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f729c139ae0 con 0x7f729c13d060 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4190696808 0 0) 0x7f727c003520 con 0x7f729c0a1b10 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f729c1378a0 con 0x7f729c0a1b10 2024-09-16T04:06:37.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f728c001c40 con 0x7f729c0a1b10 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3411073005 0 0) 0x7f729c138350 con 0x7f729c0a2410 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f727c003520 con 0x7f729c0a2410 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72840018b0 con 0x7f729c0a2410 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.729 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2856510855 0 0) 0x7f729c139ae0 con 0x7f729c13d060 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f729c138350 con 0x7f729c13d060 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7294001470 con 0x7f729c13d060 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.730 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1991914831 0 0) 0x7f729c1378a0 con 0x7f729c0a1b10 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 >> v1:172.21.15.171:6789/0 conn(0x7f729c13d060 legacy=0x7f729c1d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 >> v1:172.21.15.148:6789/0 conn(0x7f729c0a2410 legacy=0x7f729c137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 monclient: found mon.c 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:37.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c0a0130 con 0x7f729c0a1b10 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: _finish_auth 0 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 5 monclient: authenticate success, global_id 4232 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient: _renew_subs 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f729c09f810 con 0x7f729c0a1b10 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient: _renew_subs 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f729c138690 con 0x7f729c0a1b10 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f728c004cc0 con 0x7f729c0a1b10 2024-09-16T04:06:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f728c004e70 con 0x7f729c0a1b10 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7299ffb640 4 set_mon_vals no callback set 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.710+0000 7f7298ff9640 10 monclient: dump: 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:37.733 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.711+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f728c0102a0 con 0x7f729c0a1b10 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.711+0000 7f72a185e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.711+0000 7f72a185e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.711+0000 7f72a185e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.711+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7280005080 con 0x7f729c0a1b10 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.711+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(14..14 src has 1..14) ==== 3153+0+0 (unknown 1072785214 0 0) 0x7f728c046c00 con 0x7f729c0a1b10 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.712+0000 7f72927fc640 10 monclient: get_auth_request con 0x7f727c0391c0 auth_method 0 2024-09-16T04:06:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.716+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f728c015f10 con 0x7f729c0a1b10 2024-09-16T04:06:37.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.716+0000 7f7298ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:37.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.716+0000 7f7298ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:37.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.881+0000 7f72a185e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T04:06:37.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.881+0000 7f72a185e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T04:06:37.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.881+0000 7f72a185e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:37.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:37.881+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f7280005b80 con 0x7f729c0a1b10 2024-09-16T04:06:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.343+0000 7f7298ff9640 1 -- 172.21.15.148:0/3026246275 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) ==== 109+0+0 (unknown 1237045831 0 0) 0x7f728c015810 con 0x7f729c0a1b10 2024-09-16T04:06:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.343+0000 7f7298ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T04:06:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.343+0000 7f7298ff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T04:06:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:pool 'rbd' created 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.345+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 >> v1:172.21.15.171:6812/7327273 conn(0x7f727c0391c0 legacy=0x7f727c03b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.345+0000 7f72a185e640 10 monclient: shutdown 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.345+0000 7f72a185e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.345+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 >> v1:172.21.15.148:6790/0 conn(0x7f729c0a1b10 legacy=0x7f729c136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.346+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 shutdown_connections 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.346+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 >> 172.21.15.148:0/3026246275 conn(0x7f729c0a97b0 msgr2=0x7f729c13faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.346+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 shutdown_connections 2024-09-16T04:06:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.346+0000 7f72a185e640 1 -- 172.21.15.148:0/3026246275 wait complete. 2024-09-16T04:06:38.359 DEBUG:teuthology.orchestra.run.smithi148:> rbd --cluster ceph pool init rbd 2024-09-16T04:06:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.480+0000 7f3512ea9d80 10 monclient: get_monmap_and_config 2024-09-16T04:06:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.480+0000 7f3512ea9d80 10 monclient: build_initial_monmap 2024-09-16T04:06:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.480+0000 7f3512ea9d80 10 monclient: monmap: 2024-09-16T04:06:38.482 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:38.481390+0000 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:38.481390+0000 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:38.483 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 1 Processor -- start 2024-09-16T04:06:38.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 1 -- start start 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient: init 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient: _reopen_session rank -1 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient(hunting): picked mon.noname-c con 0x558163cbfcc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient(hunting): picked mon.noname-b con 0x558163c2f750 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient(hunting): picked mon.noname-a con 0x558163c29c90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x558163a90050 con 0x558163c29c90 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x558163a79ce0 con 0x558163cbfcc0 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x558163c0d950 con 0x558163c2f750 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient(hunting): _renew_subs 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512ea9d80 10 monclient(hunting): authenticate will time out at 819.106666s 2024-09-16T04:06:38.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f35111a0640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x558163c2f750 0x558163c0d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55734/0 (socket says 172.21.15.148:55734) 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f3512c03640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x558163c29c90 0x558163cc8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54690/0 (socket says 172.21.15.148:54690) 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.481+0000 7f35111a0640 1 -- 172.21.15.148:0/2404899741 learned_addr learned my addr 172.21.15.148:0/2404899741 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2708023181 0 0) 0x558163c0d950 con 0x558163c2f750 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _negotiate 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): my global_id is 4235 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34f4001e10 con 0x558163c2f750 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217330076 0 0) 0x558163a90050 con 0x558163c29c90 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _negotiate 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): my global_id is 4236 2024-09-16T04:06:38.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x558163c0d950 con 0x558163c29c90 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375692203 0 0) 0x558163a79ce0 con 0x558163cbfcc0 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _negotiate 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient(hunting): my global_id is 4219 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x558163a90050 con 0x558163cbfcc0 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2472242210 0 0) 0x7f34f4001e10 con 0x558163c2f750 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 >> v1:172.21.15.171:6789/0 conn(0x558163cbfcc0 legacy=0x558163c0cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 >> v1:172.21.15.148:6789/0 conn(0x558163c29c90 legacy=0x558163cc8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: _finish_hunting 0 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 monclient: found mon.noname-b 2024-09-16T04:06:38.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x558163abc790 con 0x558163c2f750 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: _finish_auth 0 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: _check_auth_tickets 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3500001590 con 0x558163c2f750 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 5 monclient: authenticate success, global_id 4235 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: dump: 2024-09-16T04:06:38.495 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35000017f0 con 0x558163c2f750 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:38.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 1 -- 172.21.15.148:0/2404899741 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3500001c00 con 0x558163c2f750 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 10 monclient: get_monmap_and_config success 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 4 set_mon_vals no callback set 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f351099f640 10 monclient: dump: 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.497 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 10 monclient: shutdown 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 1 -- 172.21.15.148:0/2404899741 >> v1:172.21.15.148:6790/0 conn(0x558163c2f750 legacy=0x558163c0d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 1 -- 172.21.15.148:0/2404899741 shutdown_connections 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 1 -- 172.21.15.148:0/2404899741 >> 172.21.15.148:0/2404899741 conn(0x558163c2da30 msgr2=0x558163c2de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.482+0000 7f3512ea9d80 1 -- 172.21.15.148:0/2404899741 shutdown_connections 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.483+0000 7f3512ea9d80 1 -- 172.21.15.148:0/2404899741 wait complete. 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.483+0000 7f3512ea9d80 10 monclient: build_initial_monmap 2024-09-16T04:06:38.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.483+0000 7f3512ea9d80 10 monclient: monmap: 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.483+0000 7f3512ea9d80 1 Processor -- start 2024-09-16T04:06:38.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.483+0000 7f3512ea9d80 1 -- start start 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.483+0000 7f3512ea9d80 10 monclient: init 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient: _reopen_session rank -1 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient(hunting): picked mon.noname-a con 0x558163c29c90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient(hunting): picked mon.noname-c con 0x558163c2f750 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient(hunting): picked mon.noname-b con 0x558163cbfcc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x558163cc0560 con 0x558163c29c90 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x558163cbcef0 con 0x558163cbfcc0 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x558163cacf50 con 0x558163c2f750 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient(hunting): _renew_subs 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3512ea9d80 10 monclient(hunting): authenticate will time out at 819.109373s 2024-09-16T04:06:38.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f35111a0640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x558163c2f750 0x558163cd3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55736/0 (socket says 172.21.15.148:55736) 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f35111a0640 1 -- 172.21.15.148:0/3436434231 learned_addr learned my addr 172.21.15.148:0/3436434231 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593733284 0 0) 0x558163a90050 con 0x558163c29c90 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): my global_id is 4239 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34f8001e10 con 0x558163c29c90 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472094645 0 0) 0x558163cacf50 con 0x558163c2f750 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): my global_id is 4238 2024-09-16T04:06:38.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x558163a90050 con 0x558163c2f750 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1862157128 0 0) 0x7f34f8001e10 con 0x558163c29c90 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x558163cacf50 con 0x558163c29c90 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f34fc001870 con 0x558163c29c90 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:38.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4001823523 0 0) 0x558163a90050 con 0x558163c2f750 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34f8001e10 con 0x558163c2f750 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3500001b40 con 0x558163c2f750 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:38.503 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.484+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059366906 0 0) 0x558163cbcef0 con 0x558163cbfcc0 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:38.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient(hunting): my global_id is 4222 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x558163a90050 con 0x558163cbfcc0 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3963312082 0 0) 0x558163cacf50 con 0x558163c29c90 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 >> v1:172.21.15.148:6790/0 conn(0x558163c2f750 legacy=0x558163cd3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 >> v1:172.21.15.171:6789/0 conn(0x558163cbfcc0 legacy=0x558163cd6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 monclient: found mon.a 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x558163c2bee0 con 0x558163c29c90 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:38.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 5 monclient: authenticate success, global_id 4239 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 10 monclient: _renew_subs 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x558163c105b0 con 0x558163c29c90 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 10 monclient: _renew_subs 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x558163cacc70 con 0x558163c29c90 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f34fc001f80 con 0x558163c29c90 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f34fc002350 con 0x558163c29c90 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:38.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 10 monclient: dump: 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f350bfff640 4 set_mon_vals no callback set 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.485+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f34fc00d800 con 0x558163c29c90 2024-09-16T04:06:38.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.486+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3495+0+0 (unknown 3365391291 0 0) 0x7f34fc044420 con 0x558163c29c90 2024-09-16T04:06:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.486+0000 7f3512ea9d80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T04:06:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.486+0000 7f3512ea9d80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T04:06:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.486+0000 7f3512ea9d80 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.486+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x558163caa290 con 0x558163c29c90 2024-09-16T04:06:38.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:38.486+0000 7f35111a0640 10 monclient: get_auth_request con 0x7f34f8037960 auth_method 0 2024-09-16T04:06:39.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.343+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v16) ==== 141+0+0 (unknown 2855932958 0 0) 0x558163cc0560 con 0x558163c29c90 2024-09-16T04:06:39.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.343+0000 7f3509ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T04:06:39.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.343+0000 7f3509ffb640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T04:06:39.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.343+0000 7f3512ea9d80 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:39.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.343+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f35000017f0 con 0x558163c29c90 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (unknown 3847591702 0 0) 0x7f34fc012b90 con 0x558163c29c90 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f3509ffb640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f350bfff640 10 monclient: _renew_subs 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f350bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f350bfff640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=16}) -- 0x7f34ec002ae0 con 0x558163c29c90 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 11 ==== osd_map(16..16 src has 1..16) ==== 658+0+0 (unknown 265119331 0 0) 0x7f34fc013020 con 0x558163c29c90 2024-09-16T04:06:39.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.171:6800/466524840 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x558163cdcce0 con 0x558163cd91d0 2024-09-16T04:06:39.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.350+0000 7f3512c03640 10 monclient: get_auth_request con 0x558163cd91d0 auth_method 0 2024-09-16T04:06:39.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.360+0000 7f3512c03640 1 -- 172.21.15.148:0/3436434231 <== osd.5 v1:172.21.15.171:6800/466524840 1 ==== osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 2685446591 0 0) 0x7f3504001c10 con 0x558163cd91d0 2024-09-16T04:06:39.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.363+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.171:6800/466524840 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x558163d2e690 con 0x558163cd91d0 2024-09-16T04:06:39.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.363+0000 7f3512c03640 1 -- 172.21.15.148:0/3436434231 <== osd.5 v1:172.21.15.171:6800/466524840 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 2026326479 0 0) 0x7f3504001c10 con 0x558163cd91d0 2024-09-16T04:06:39.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.363+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.171:6800/466524840 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x558163d2d7a0 con 0x558163cd91d0 2024-09-16T04:06:39.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.363+0000 7f3512c03640 1 -- 172.21.15.148:0/3436434231 <== osd.5 v1:172.21.15.171:6800/466524840 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 2039985091 0 0) 0x7f3504001c10 con 0x558163cd91d0 2024-09-16T04:06:39.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.363+0000 7f350b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:39.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:39.363+0000 7f350b7fe640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f34e4001f00 con 0x558163c29c90 2024-09-16T04:06:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.352+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 12 ==== pool_op_reply(tid 4 (0) Success v17) ==== 55+0+0 (unknown 93546325 0 0) 0x7f34fc0134c0 con 0x558163c29c90 2024-09-16T04:06:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.352+0000 7f3509ffb640 10 monclient: _renew_subs 2024-09-16T04:06:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.352+0000 7f3509ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.352+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=17}) -- 0x558163cacf50 con 0x558163c29c90 2024-09-16T04:06:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.352+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 13 ==== osd_map(17..17 src has 1..17) ==== 626+0+0 (unknown 3976597585 0 0) 0x7f34fc013220 con 0x558163c29c90 2024-09-16T04:06:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.352+0000 7f350bfff640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.171:6800/466524840 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x7f34ec0056c0 con 0x558163cd91d0 2024-09-16T04:06:40.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.358+0000 7f3512c03640 1 -- 172.21.15.148:0/3436434231 <== osd.5 v1:172.21.15.171:6800/466524840 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 3115756469 0 0) 0x7f3504001c10 con 0x558163cd91d0 2024-09-16T04:06:40.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.358+0000 7f350b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:40.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:40.358+0000 7f350b7fe640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v17) -- 0x7f34e4003320 con 0x558163c29c90 2024-09-16T04:06:41.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.367+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 14 ==== pool_op_reply(tid 6 (0) Success v18) ==== 43+0+0 (unknown 381912340 0 0) 0x7f34fc012b50 con 0x558163c29c90 2024-09-16T04:06:41.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.367+0000 7f3509ffb640 10 monclient: _renew_subs 2024-09-16T04:06:41.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.367+0000 7f3509ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.367+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=18}) -- 0x7f34f8001e10 con 0x558163c29c90 2024-09-16T04:06:41.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.367+0000 7f3509ffb640 1 -- 172.21.15.148:0/3436434231 <== mon.0 v1:172.21.15.148:6789/0 15 ==== osd_map(18..18 src has 1..18) ==== 654+0+0 (unknown 3461669983 0 0) 0x7f34fc011ce0 con 0x558163c29c90 2024-09-16T04:06:41.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.367+0000 7f350bfff640 1 -- 172.21.15.148:0/3436434231 --> v1:172.21.15.171:6800/466524840 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f34ec005aa0 con 0x558163cd91d0 2024-09-16T04:06:41.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.369+0000 7f3512c03640 1 -- 172.21.15.148:0/3436434231 <== osd.5 v1:172.21.15.171:6800/466524840 5 ==== osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 4030416863 0 0) 0x7f3504001c10 con 0x558163cd91d0 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.369+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 >> v1:172.21.15.171:6800/466524840 conn(0x558163cd91d0 legacy=0x558163cdb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.369+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 >> v1:172.21.15.171:6812/7327273 conn(0x7f34f8037960 legacy=0x7f34f8039e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.369+0000 7f3512ea9d80 10 monclient: shutdown 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.369+0000 7f3512ea9d80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.369+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 >> v1:172.21.15.148:6789/0 conn(0x558163c29c90 legacy=0x558163cbc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.370+0000 7f3512c03640 1 -- 172.21.15.148:0/3436434231 reap_dead start 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.370+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 shutdown_connections 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.370+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 >> 172.21.15.148:0/3436434231 conn(0x558163c2da30 msgr2=0x558163c28bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:41.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.370+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 shutdown_connections 2024-09-16T04:06:41.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.370+0000 7f3512ea9d80 1 -- 172.21.15.148:0/3436434231 wait complete. 2024-09-16T04:06:41.376 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T04:06:41.376 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T04:06:41.377 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T04:06:41.377 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T04:06:41.377 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T04:06:41.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.481+0000 7f7e9ff4b640 10 monclient: get_monmap_and_config 2024-09-16T04:06:41.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.481+0000 7f7e9ff4b640 10 monclient: build_initial_monmap 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.481+0000 7f7e9ff4b640 10 monclient: monmap: 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:41.482599+0000 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:41.482599+0000 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:41.484 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:41.485 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 1 Processor -- start 2024-09-16T04:06:41.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 1 -- start start 2024-09-16T04:06:41.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient: init 2024-09-16T04:06:41.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:41.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e980733a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7e98073780 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7e9813d010 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e980a3410 con 0x7f7e98073780 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e980a3190 con 0x7f7e980733a0 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e9809e4b0 con 0x7f7e9813d010 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.482+0000 7f7e9ff4b640 10 monclient(hunting): authenticate will time out at 822.107999s 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9d4e6640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7e98073780 0x7f7e9813b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54718/0 (socket says 172.21.15.148:54718) 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9e4e8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7e9813d010 0x7f7e9813f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55750/0 (socket says 172.21.15.148:55750) 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9d4e6640 1 -- 172.21.15.148:0/1580397801 learned_addr learned my addr 172.21.15.148:0/1580397801 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:41.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3324961319 0 0) 0x7f7e9809e4b0 con 0x7f7e9813d010 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): my global_id is 4241 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e84003520 con 0x7f7e9813d010 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147152961 0 0) 0x7f7e980a3410 con 0x7f7e98073780 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): my global_id is 4242 2024-09-16T04:06:41.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e9809e4b0 con 0x7f7e98073780 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1425814544 0 0) 0x7f7e980a3190 con 0x7f7e980733a0 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient(hunting): my global_id is 4225 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e980a3410 con 0x7f7e980733a0 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1258063399 0 0) 0x7f7e84003520 con 0x7f7e9813d010 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 >> v1:172.21.15.171:6789/0 conn(0x7f7e980733a0 legacy=0x7f7e9809ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 >> v1:172.21.15.148:6789/0 conn(0x7f7e98073780 legacy=0x7f7e9813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: _finish_hunting 0 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 monclient: found mon.noname-b 2024-09-16T04:06:41.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9806e700 con 0x7f7e9813d010 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: _finish_auth 0 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: _check_auth_tickets 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e90001610 con 0x7f7e9813d010 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9ff4b640 5 monclient: authenticate success, global_id 4241 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9ff4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: dump: 2024-09-16T04:06:41.498 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9ff4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e900018c0 con 0x7f7e9813d010 2024-09-16T04:06:41.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 1 -- 172.21.15.148:0/1580397801 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e90001cd0 con 0x7f7e9813d010 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9ff4b640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9ff4b640 4 set_mon_vals no callback set 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.483+0000 7f7e9cce5640 10 monclient: dump: 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.500 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 10 monclient: shutdown 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/1580397801 >> v1:172.21.15.148:6790/0 conn(0x7f7e9813d010 legacy=0x7f7e9813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/1580397801 shutdown_connections 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/1580397801 >> 172.21.15.148:0/1580397801 conn(0x7f7e980a78b0 msgr2=0x7f7e980a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/1580397801 shutdown_connections 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/1580397801 wait complete. 2024-09-16T04:06:41.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 10 monclient: build_initial_monmap 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.484+0000 7f7e9ff4b640 10 monclient: monmap: 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 1 Processor -- start 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 1 -- start start 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient: init 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7e980733a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7e98073780 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e9813d010 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e98124090 con 0x7f7e980733a0 2024-09-16T04:06:41.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e98122590 con 0x7f7e98073780 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e98124370 con 0x7f7e9813d010 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9ff4b640 10 monclient(hunting): authenticate will time out at 822.110811s 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9dce7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7e980733a0 0x7f7e98121770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54722/0 (socket says 172.21.15.148:54722) 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.485+0000 7f7e9dce7640 1 -- 172.21.15.148:0/91166047 learned_addr learned my addr 172.21.15.148:0/91166047 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974186925 0 0) 0x7f7e98124370 con 0x7f7e9813d010 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): my global_id is 4244 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e7c003520 con 0x7f7e9813d010 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220369328 0 0) 0x7f7e98124090 con 0x7f7e980733a0 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): my global_id is 4245 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e98124370 con 0x7f7e980733a0 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579996378 0 0) 0x7f7e98122590 con 0x7f7e98073780 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): my global_id is 4228 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e98124090 con 0x7f7e98073780 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3613973518 0 0) 0x7f7e98124370 con 0x7f7e980733a0 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e98122590 con 0x7f7e980733a0 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e94001ad0 con 0x7f7e980733a0 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.506 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 694380839 0 0) 0x7f7e7c003520 con 0x7f7e9813d010 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e98124370 con 0x7f7e9813d010 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e90001a70 con 0x7f7e9813d010 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.507 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2741775881 0 0) 0x7f7e98124090 con 0x7f7e98073780 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e7c003520 con 0x7f7e98073780 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e88001250 con 0x7f7e98073780 2024-09-16T04:06:41.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4176966930 0 0) 0x7f7e98122590 con 0x7f7e980733a0 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 >> v1:172.21.15.148:6790/0 conn(0x7f7e9813d010 legacy=0x7f7e981d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 >> v1:172.21.15.171:6789/0 conn(0x7f7e98073780 legacy=0x7f7e98121e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 monclient: found mon.a 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9806d4d0 con 0x7f7e980733a0 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:41.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 5 monclient: authenticate success, global_id 4245 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 10 monclient: _renew_subs 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e980741d0 con 0x7f7e980733a0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 10 monclient: _renew_subs 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e98123db0 con 0x7f7e980733a0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e94003990 con 0x7f7e980733a0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e94003ce0 con 0x7f7e980733a0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:41.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8ffff640 4 set_mon_vals no callback set 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.486+0000 7f7e8e7fc640 10 monclient: dump: 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.487+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7e94003ce0 con 0x7f7e980733a0 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.487+0000 7f7e9ff4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.487+0000 7f7e9ff4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.487+0000 7f7e9ff4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.487+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e80005080 con 0x7f7e980733a0 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.487+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f7e94046cf0 con 0x7f7e980733a0 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.488+0000 7f7e9d4e6640 10 monclient: get_auth_request con 0x7f7e7c039250 auth_method 0 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.492+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7e94015c10 con 0x7f7e980733a0 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.492+0000 7f7e8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:41.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.492+0000 7f7e8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.704+0000 7f7e9ff4b640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T04:06:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.704+0000 7f7e9ff4b640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T04:06:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.704+0000 7f7e9ff4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.704+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f7e80005370 con 0x7f7e980733a0 2024-09-16T04:06:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.706+0000 7f7e8e7fc640 1 -- 172.21.15.148:0/91166047 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+87422 (unknown 481731318 0 3760130862) 0x7f7e94015510 con 0x7f7e980733a0 2024-09-16T04:06:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.706+0000 7f7e8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T04:06:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.706+0000 7f7e8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:41.709 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 >> v1:172.21.15.171:6812/7327273 conn(0x7f7e7c039250 legacy=0x7f7e7c03b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 10 monclient: shutdown 2024-09-16T04:06:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 >> v1:172.21.15.148:6789/0 conn(0x7f7e980733a0 legacy=0x7f7e98121770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 shutdown_connections 2024-09-16T04:06:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 >> 172.21.15.148:0/91166047 conn(0x7f7e980a78b0 msgr2=0x7f7e9809d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 shutdown_connections 2024-09-16T04:06:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.708+0000 7f7e9ff4b640 1 -- 172.21.15.148:0/91166047 wait complete. 2024-09-16T04:06:41.720 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6812","nonce":7327273}]},"active_addr":"172.21.15.171:6812/7327273","active_change":"2024-09-16T04:06:25.261879+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":565145707}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":3449837665}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":1092890529}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.171:0","nonce":3301966265}]}]} 2024-09-16T04:06:41.723 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T04:06:41.723 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T04:06:41.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:41.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.818+0000 7fc6d8c80640 10 monclient: get_monmap_and_config 2024-09-16T04:06:41.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.818+0000 7fc6d8c80640 10 monclient: build_initial_monmap 2024-09-16T04:06:41.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.818+0000 7fc6d8c80640 10 monclient: monmap: 2024-09-16T04:06:41.820 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:41.819412+0000 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:41.819412+0000 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:41.821 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 1 Processor -- start 2024-09-16T04:06:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 1 -- start start 2024-09-16T04:06:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient: init 2024-09-16T04:06:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:41.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d4133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient(hunting): picked mon.noname-a con 0x7fc6d406c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d413d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d409b0d0 con 0x7fc6d406c430 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d409b340 con 0x7fc6d413d0e0 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d409b620 con 0x7fc6d4133910 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d8c80640 10 monclient(hunting): authenticate will time out at 822.444815s 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d259c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc6d4133910 0x7fc6d4133cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55768/0 (socket says 172.21.15.148:55768) 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d259c640 1 -- 172.21.15.148:0/1709170418 learned_addr learned my addr 172.21.15.148:0/1709170418 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.819+0000 7fc6d1d9b640 1 --1- 172.21.15.148:0/1709170418 >> v1:172.21.15.148:6789/0 conn(0x7fc6d406c430 0x7fc6d413b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54728/0 (socket says 172.21.15.148:54728) 2024-09-16T04:06:41.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1690207256 0 0) 0x7fc6d409b620 con 0x7fc6d4133910 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): my global_id is 4247 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6c0003520 con 0x7fc6d4133910 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2386472788 0 0) 0x7fc6d409b0d0 con 0x7fc6d406c430 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): my global_id is 4248 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d409b620 con 0x7fc6d406c430 2024-09-16T04:06:41.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2975404559 0 0) 0x7fc6d409b340 con 0x7fc6d413d0e0 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient(hunting): my global_id is 4231 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d409b0d0 con 0x7fc6d413d0e0 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3908335543 0 0) 0x7fc6d409b620 con 0x7fc6d406c430 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 >> v1:172.21.15.148:6790/0 conn(0x7fc6d4133910 legacy=0x7fc6d4133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 >> v1:172.21.15.171:6789/0 conn(0x7fc6d413d0e0 legacy=0x7fc6d413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: _finish_hunting 0 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 monclient: found mon.noname-a 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d40633c0 con 0x7fc6d406c430 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: _finish_auth 0 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: _check_auth_tickets 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d8c80640 5 monclient: authenticate success, global_id 4248 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6c4001610 con 0x7fc6d406c430 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d8c80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: dump: 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d8c80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6c4001870 con 0x7fc6d406c430 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d8c80640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 1 -- 172.21.15.148:0/1709170418 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6c4001c80 con 0x7fc6d406c430 2024-09-16T04:06:41.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d8c80640 4 set_mon_vals no callback set 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d159a640 10 monclient: dump: 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.820+0000 7fc6d8c80640 10 monclient: shutdown 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 1 -- 172.21.15.148:0/1709170418 >> v1:172.21.15.148:6789/0 conn(0x7fc6d406c430 legacy=0x7fc6d413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 1 -- 172.21.15.148:0/1709170418 shutdown_connections 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 1 -- 172.21.15.148:0/1709170418 >> 172.21.15.148:0/1709170418 conn(0x7fc6d4062630 msgr2=0x7fc6d4062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 1 -- 172.21.15.148:0/1709170418 shutdown_connections 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 1 -- 172.21.15.148:0/1709170418 wait complete. 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 10 monclient: build_initial_monmap 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.821+0000 7fc6d8c80640 10 monclient: monmap: 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:41.836 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 1 Processor -- start 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 1 -- start start 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient: init 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient(hunting): picked mon.noname-a con 0x7fc6d406c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:41.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d4133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d413d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d406e000 con 0x7fc6d406c430 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d4130730 con 0x7fc6d413d0e0 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d4132a90 con 0x7fc6d4133910 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d8c80640 10 monclient(hunting): authenticate will time out at 822.447698s 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc6d406c430 0x7fc6d412ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54736/0 (socket says 172.21.15.148:54736) 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6d259c640 1 -- 172.21.15.148:0/4022137969 learned_addr learned my addr 172.21.15.148:0/4022137969 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640029643 0 0) 0x7fc6d406e000 con 0x7fc6d406c430 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): my global_id is 4251 2024-09-16T04:06:41.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b4003520 con 0x7fc6d406c430 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2013533827 0 0) 0x7fc6d4132a90 con 0x7fc6d4133910 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 10 monclient(hunting): my global_id is 4250 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.822+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d406e000 con 0x7fc6d4133910 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543304089 0 0) 0x7fc6d4130730 con 0x7fc6d413d0e0 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): my global_id is 4234 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d4132a90 con 0x7fc6d413d0e0 2024-09-16T04:06:41.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2320676487 0 0) 0x7fc6b4003520 con 0x7fc6d406c430 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6d4130730 con 0x7fc6d406c430 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6bc001960 con 0x7fc6d406c430 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): dump: 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.840 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 814848279 0 0) 0x7fc6d406e000 con 0x7fc6d4133910 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6b4003520 con 0x7fc6d4133910 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6c4001bc0 con 0x7fc6d4133910 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient(hunting): dump: 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.841 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1371956028 0 0) 0x7fc6d4130730 con 0x7fc6d406c430 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 >> v1:172.21.15.148:6790/0 conn(0x7fc6d4133910 legacy=0x7fc6d412f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 >> v1:172.21.15.171:6789/0 conn(0x7fc6d413d0e0 legacy=0x7fc6d412faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 monclient: found mon.a 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:41.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d41321f0 con 0x7fc6d406c430 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 5 monclient: authenticate success, global_id 4251 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 10 monclient: _renew_subs 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d4132eb0 con 0x7fc6d406c430 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 10 monclient: _renew_subs 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6d4133380 con 0x7fc6d406c430 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6bc001cf0 con 0x7fc6d406c430 2024-09-16T04:06:41.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6bc004c30 con 0x7fc6d406c430 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6d0d99640 4 set_mon_vals no callback set 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.823+0000 7fc6caffd640 10 monclient: dump: 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:41.844 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6d8c80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc6bc010060 con 0x7fc6d406c430 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6d8c80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6d8c80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b8005080 con 0x7fc6d406c430 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fc6bc046cc0 con 0x7fc6d406c430 2024-09-16T04:06:41.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.824+0000 7fc6d1d9b640 10 monclient: get_auth_request con 0x7fc6b4039070 auth_method 0 2024-09-16T04:06:41.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.829+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc6bc077d40 con 0x7fc6d406c430 2024-09-16T04:06:41.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.829+0000 7fc6caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:41.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.829+0000 7fc6caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:41.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6d8c80640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:42.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6d8c80640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:42.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6d8c80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6b8005370 con 0x7fc6d406c430 2024-09-16T04:06:42.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6caffd640 1 -- 172.21.15.148:0/4022137969 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10953 (unknown 3338201251 0 1957694494) 0x7fc6bc015c60 con 0x7fc6d406c430 2024-09-16T04:06:42.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:42.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:41.998+0000 7fc6caffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:42.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:42.001 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":18,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:41.354592+0000","last_up_change":"2024-09-16T04:06:32.301387+0000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:06:31.297815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:06:37.900873+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":985869260}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":985869260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":985869260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":985869260}]},"public_addr":"172.21.15.148:6808/985869260","cluster_addr":"172.21.15.148:6809/985869260","heartbeat_back_addr":"172.21.15.148:6811/985869260","heartbeat_front_addr":"172.21.15.148:6810/985869260","state":["exists","up"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":3044422561}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":3044422561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":3044422561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":3044422561}]},"public_addr":"172.21.15.148:6800/3044422561","cluster_addr":"172.21.15.148:6801/3044422561","heartbeat_back_addr":"172.21.15.148:6803/3044422561","heartbeat_front_addr":"172.21.15.148:6802/3044422561","state":["exists","up"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":262588452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":262588452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":262588452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":262588452}]},"public_addr":"172.21.15.148:6804/262588452","cluster_addr":"172.21.15.148:6805/262588452","heartbeat_back_addr":"172.21.15.148:6807/262588452","heartbeat_front_addr":"172.21.15.148:6806/262588452","state":["exists","up"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":772155526}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":772155526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":772155526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":772155526}]},"public_addr":"172.21.15.171:6804/772155526","cluster_addr":"172.21.15.171:6805/772155526","heartbeat_back_addr":"172.21.15.171:6807/772155526","heartbeat_front_addr":"172.21.15.171:6806/772155526","state":["exists","up"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":4274747803}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":4274747803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":4274747803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":4274747803}]},"public_addr":"172.21.15.171:6808/4274747803","cluster_addr":"172.21.15.171:6809/4274747803","heartbeat_back_addr":"172.21.15.171:6811/4274747803","heartbeat_front_addr":"172.21.15.171:6810/4274747803","state":["exists","up"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":466524840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":466524840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":466524840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":466524840}]},"public_addr":"172.21.15.171:6800/466524840","cluster_addr":"172.21.15.171:6801/466524840","heartbeat_back_addr":"172.21.15.171:6803/466524840","heartbeat_front_addr":"172.21.15.171:6802/466524840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.060468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:28.867753+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140301+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.136223+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.080093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:42.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 >> v1:172.21.15.171:6812/7327273 conn(0x7fc6b4039070 legacy=0x7fc6b403b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 10 monclient: shutdown 2024-09-16T04:06:42.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 >> v1:172.21.15.148:6789/0 conn(0x7fc6d406c430 legacy=0x7fc6d412ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 shutdown_connections 2024-09-16T04:06:42.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 >> 172.21.15.148:0/4022137969 conn(0x7fc6d4062630 msgr2=0x7fc6d412c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 shutdown_connections 2024-09-16T04:06:42.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.000+0000 7fc6d8c80640 1 -- 172.21.15.148:0/4022137969 wait complete. 2024-09-16T04:06:42.013 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T04:06:42.013 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.109+0000 7fd89bba6640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.109+0000 7fd89bba6640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.109+0000 7fd89bba6640 10 monclient: monmap: 2024-09-16T04:06:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.111 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.110353+0000 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.110353+0000 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.112 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.109+0000 7fd89bba6640 1 Processor -- start 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 1 -- start start 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient: init 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient(hunting): picked mon.noname-a con 0x7fd894133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient(hunting): picked mon.noname-b con 0x7fd89406c430 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient(hunting): picked mon.noname-c con 0x7fd89413d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89409b0d0 con 0x7fd894133910 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89409b340 con 0x7fd89413d0e0 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89409b620 con 0x7fd89406c430 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd89bba6640 10 monclient(hunting): authenticate will time out at 822.735707s 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd899942640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd894133910 0x7fd894133cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54740/0 (socket says 172.21.15.148:54740) 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd899141640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd89406c430 0x7fd89413b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55780/0 (socket says 172.21.15.148:55780) 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.110+0000 7fd899942640 1 -- 172.21.15.148:0/2759559856 learned_addr learned my addr 172.21.15.148:0/2759559856 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915433271 0 0) 0x7fd89409b620 con 0x7fd89406c430 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): my global_id is 4253 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd880003520 con 0x7fd89406c430 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2811636398 0 0) 0x7fd89409b0d0 con 0x7fd894133910 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient(hunting): my global_id is 4254 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd89409b620 con 0x7fd894133910 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3467380979 0 0) 0x7fd880003520 con 0x7fd89406c430 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 >> v1:172.21.15.171:6789/0 conn(0x7fd89413d0e0 legacy=0x7fd89413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 >> v1:172.21.15.148:6789/0 conn(0x7fd894133910 legacy=0x7fd894133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 monclient: found mon.noname-b 2024-09-16T04:06:42.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8940633c0 con 0x7fd89406c430 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 5 monclient: authenticate success, global_id 4253 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd888001610 con 0x7fd89406c430 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: dump: 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.124 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8880018f0 con 0x7fd89406c430 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 1 -- 172.21.15.148:0/2759559856 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd888001d00 con 0x7fd89406c430 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 4 set_mon_vals no callback set 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd898940640 10 monclient: dump: 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.125 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 10 monclient: shutdown 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 1 -- 172.21.15.148:0/2759559856 >> v1:172.21.15.148:6790/0 conn(0x7fd89406c430 legacy=0x7fd89413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 1 -- 172.21.15.148:0/2759559856 shutdown_connections 2024-09-16T04:06:42.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.111+0000 7fd89bba6640 1 -- 172.21.15.148:0/2759559856 >> 172.21.15.148:0/2759559856 conn(0x7fd894062630 msgr2=0x7fd894062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 1 -- 172.21.15.148:0/2759559856 shutdown_connections 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 1 -- 172.21.15.148:0/2759559856 wait complete. 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 10 monclient: monmap: 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.127 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 1 Processor -- start 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 1 -- start start 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.112+0000 7fd89bba6640 10 monclient: init 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:42.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient(hunting): picked mon.noname-c con 0x7fd89406c430 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient(hunting): picked mon.noname-a con 0x7fd894133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient(hunting): picked mon.noname-b con 0x7fd89413d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd894138c90 con 0x7fd894133910 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8940a4d70 con 0x7fd89413d0e0 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd894140810 con 0x7fd89406c430 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd89bba6640 10 monclient(hunting): authenticate will time out at 822.738331s 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd899942640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd89406c430 0x7fd8941cb0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:55786/0 (socket says 172.21.15.148:55786) 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd899942640 1 -- 172.21.15.148:0/1761199107 learned_addr learned my addr 172.21.15.148:0/1761199107 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314676927 0 0) 0x7fd894140810 con 0x7fd89406c430 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): my global_id is 4256 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd878003520 con 0x7fd89406c430 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 32681361 0 0) 0x7fd894138c90 con 0x7fd894133910 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): my global_id is 4257 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd894140810 con 0x7fd894133910 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660268276 0 0) 0x7fd8940a4d70 con 0x7fd89413d0e0 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): my global_id is 4240 2024-09-16T04:06:42.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd894138c90 con 0x7fd89413d0e0 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3213404955 0 0) 0x7fd878003520 con 0x7fd89406c430 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8940a4d70 con 0x7fd89406c430 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4202258500 0 0) 0x7fd894140810 con 0x7fd894133910 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd878003520 con 0x7fd894133910 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd8900018e0 con 0x7fd89406c430 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): dump: 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.131 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd888001c40 con 0x7fd894133910 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.113+0000 7fd8867fc640 10 monclient(hunting): dump: 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.132 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1216505916 0 0) 0x7fd894138c90 con 0x7fd89413d0e0 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd894140810 con 0x7fd89413d0e0 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd88c001b30 con 0x7fd89413d0e0 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient(hunting): dump: 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.133 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2388568406 0 0) 0x7fd878003520 con 0x7fd894133910 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 >> v1:172.21.15.148:6790/0 conn(0x7fd89406c430 legacy=0x7fd8941cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 >> v1:172.21.15.171:6789/0 conn(0x7fd89413d0e0 legacy=0x7fd8941d3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 monclient: found mon.a 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd894139e60 con 0x7fd894133910 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 5 monclient: authenticate success, global_id 4257 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 10 monclient: _renew_subs 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd894138a10 con 0x7fd894133910 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 10 monclient: _renew_subs 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd89413c060 con 0x7fd894133910 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8880018f0 con 0x7fd894133910 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd888004da0 con 0x7fd894133910 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd887fff640 4 set_mon_vals no callback set 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.114+0000 7fd8867fc640 10 monclient: dump: 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd888010030 con 0x7fd894133910 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd89bba6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd89bba6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd89bba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd87c005080 con 0x7fd894133910 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fd888046aa0 con 0x7fd894133910 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.115+0000 7fd899942640 10 monclient: get_auth_request con 0x7fd878039120 auth_method 0 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.119+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd888015b00 con 0x7fd894133910 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.119+0000 7fd8867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.119+0000 7fd8867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.285+0000 7fd89bba6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.285+0000 7fd89bba6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.285+0000 7fd89bba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.285+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd87c005370 con 0x7fd894133910 2024-09-16T04:06:42.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.286+0000 7fd8867fc640 1 -- 172.21.15.148:0/1761199107 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10953 (unknown 3338201251 0 1957694494) 0x7fd88801dfb0 con 0x7fd894133910 2024-09-16T04:06:42.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.286+0000 7fd8867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:42.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.286+0000 7fd8867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:42.288 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:42.288 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":18,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:41.354592+0000","last_up_change":"2024-09-16T04:06:32.301387+0000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:06:31.297815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:06:37.900873+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":985869260}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":985869260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":985869260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":985869260}]},"public_addr":"172.21.15.148:6808/985869260","cluster_addr":"172.21.15.148:6809/985869260","heartbeat_back_addr":"172.21.15.148:6811/985869260","heartbeat_front_addr":"172.21.15.148:6810/985869260","state":["exists","up"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":3044422561}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":3044422561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":3044422561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":3044422561}]},"public_addr":"172.21.15.148:6800/3044422561","cluster_addr":"172.21.15.148:6801/3044422561","heartbeat_back_addr":"172.21.15.148:6803/3044422561","heartbeat_front_addr":"172.21.15.148:6802/3044422561","state":["exists","up"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":262588452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":262588452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":262588452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":262588452}]},"public_addr":"172.21.15.148:6804/262588452","cluster_addr":"172.21.15.148:6805/262588452","heartbeat_back_addr":"172.21.15.148:6807/262588452","heartbeat_front_addr":"172.21.15.148:6806/262588452","state":["exists","up"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":772155526}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":772155526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":772155526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":772155526}]},"public_addr":"172.21.15.171:6804/772155526","cluster_addr":"172.21.15.171:6805/772155526","heartbeat_back_addr":"172.21.15.171:6807/772155526","heartbeat_front_addr":"172.21.15.171:6806/772155526","state":["exists","up"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":4274747803}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":4274747803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":4274747803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":4274747803}]},"public_addr":"172.21.15.171:6808/4274747803","cluster_addr":"172.21.15.171:6809/4274747803","heartbeat_back_addr":"172.21.15.171:6811/4274747803","heartbeat_front_addr":"172.21.15.171:6810/4274747803","state":["exists","up"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":466524840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":466524840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":466524840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":466524840}]},"public_addr":"172.21.15.171:6800/466524840","cluster_addr":"172.21.15.171:6801/466524840","heartbeat_back_addr":"172.21.15.171:6803/466524840","heartbeat_front_addr":"172.21.15.171:6802/466524840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.060468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:28.867753+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140301+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.136223+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.080093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:42.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 >> v1:172.21.15.171:6812/7327273 conn(0x7fd878039120 legacy=0x7fd87803b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 10 monclient: shutdown 2024-09-16T04:06:42.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 >> v1:172.21.15.148:6789/0 conn(0x7fd894133910 legacy=0x7fd89413b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 shutdown_connections 2024-09-16T04:06:42.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 >> 172.21.15.148:0/1761199107 conn(0x7fd894062630 msgr2=0x7fd8940a43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 shutdown_connections 2024-09-16T04:06:42.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.288+0000 7fd89bba6640 1 -- 172.21.15.148:0/1761199107 wait complete. 2024-09-16T04:06:42.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T04:06:42.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T04:06:42.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T04:06:42.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T04:06:42.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T04:06:42.303 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T04:06:42.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.417+0000 7f6bdd92c640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.417+0000 7f6bdd92c640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.417+0000 7f6bdd92c640 10 monclient: monmap: 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.418315+0000 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.418315+0000 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.420 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 1 Processor -- start 2024-09-16T04:06:42.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 1 -- start start 2024-09-16T04:06:42.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 10 monclient: init 2024-09-16T04:06:42.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:42.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd80a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd80a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bdd92c640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd813d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bd77fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6bd813d3c0 0x7f6bd813f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35420/0 (socket says 172.21.15.148:35420) 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.418+0000 7f6bd77fe640 1 -- 172.21.15.148:0/1249488929 learned_addr learned my addr 172.21.15.148:0/1249488929 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd80a0590 con 0x7f6bd80a4580 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd80a1270 con 0x7f6bd80a3a90 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd80a4fd0 con 0x7f6bd813d3c0 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bdd92c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bdd92c640 10 monclient(hunting): authenticate will time out at 823.044515s 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317886011 0 0) 0x7f6bd80a1270 con 0x7f6bd80a3a90 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): my global_id is 4243 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc0003520 con 0x7f6bd80a3a90 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2898973104 0 0) 0x7f6bcc001110 con 0x7f6bd80a4580 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): my global_id is 4260 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd80a1270 con 0x7f6bd80a4580 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506540141 0 0) 0x7f6bd80a4fd0 con 0x7f6bd813d3c0 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 10 monclient(hunting): my global_id is 4259 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.419+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bcc001110 con 0x7f6bd813d3c0 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1889961699 0 0) 0x7f6bc0003520 con 0x7f6bd80a3a90 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 >> v1:172.21.15.148:6790/0 conn(0x7f6bd813d3c0 legacy=0x7f6bd813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 >> v1:172.21.15.148:6789/0 conn(0x7f6bd80a4580 legacy=0x7f6bd813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 monclient: found mon.noname-c 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd806e170 con 0x7f6bd80a3a90 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6bc8001610 con 0x7f6bd80a3a90 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: dump: 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.435 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bdd92c640 5 monclient: authenticate success, global_id 4243 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bdd92c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bc80018e0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bdd92c640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bdd92c640 4 set_mon_vals no callback set 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 1 -- 172.21.15.148:0/1249488929 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6bc8001cb0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.420+0000 7f6bd5ffb640 10 monclient: dump: 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.436 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 10 monclient: shutdown 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 >> v1:172.21.15.171:6789/0 conn(0x7f6bd80a3a90 legacy=0x7f6bd80a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 shutdown_connections 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 >> 172.21.15.148:0/1249488929 conn(0x7f6bd80a9a40 msgr2=0x7f6bd80a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 shutdown_connections 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 1 -- 172.21.15.148:0/1249488929 wait complete. 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.421+0000 7f6bdd92c640 10 monclient: monmap: 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.438 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.422+0000 7f6bdd92c640 1 Processor -- start 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.422+0000 7f6bdd92c640 1 -- start start 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.422+0000 7f6bdd92c640 10 monclient: init 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 10 monclient(hunting): picked mon.noname-c con 0x7f6bd80a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bd6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6bd80a3a90 0x7f6bd8136af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35430/0 (socket says 172.21.15.148:35430) 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bd6ffd640 1 -- 172.21.15.148:0/2620185118 learned_addr learned my addr 172.21.15.148:0/2620185118 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 10 monclient(hunting): picked mon.noname-b con 0x7f6bd80a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 10 monclient(hunting): picked mon.noname-a con 0x7f6bd813d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd8072aa0 con 0x7f6bd813d3c0 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.423+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd8072d80 con 0x7f6bd80a4580 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd809aef0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476287312 0 0) 0x7f6bd8072aa0 con 0x7f6bd813d3c0 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bdd92c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bdd92c640 10 monclient(hunting): authenticate will time out at 823.049370s 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): my global_id is 4263 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bc4003520 con 0x7f6bd813d3c0 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1446907743 0 0) 0x7f6bd8072d80 con 0x7f6bd80a4580 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): my global_id is 4246 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd8072aa0 con 0x7f6bd80a4580 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873232926 0 0) 0x7f6bd809aef0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 10 monclient(hunting): my global_id is 4262 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd8072d80 con 0x7f6bd80a3a90 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4266966534 0 0) 0x7f6bd8072aa0 con 0x7f6bd80a4580 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.424+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bd809aef0 con 0x7f6bd80a4580 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1656036341 0 0) 0x7f6bd8072d80 con 0x7f6bd80a3a90 2024-09-16T04:06:42.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bd8072aa0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6bcc001580 con 0x7f6bd80a4580 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): dump: 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.442 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6bc8001550 con 0x7f6bd80a3a90 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient(hunting): dump: 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.443 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2801148303 0 0) 0x7f6bc4003520 con 0x7f6bd813d3c0 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6bd8072d80 con 0x7f6bd813d3c0 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 650290465 0 0) 0x7f6bd8072aa0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 >> v1:172.21.15.171:6789/0 conn(0x7f6bd80a4580 legacy=0x7f6bd8137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 >> v1:172.21.15.148:6789/0 conn(0x7f6bd813d3c0 legacy=0x7f6bd81d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 monclient: found mon.c 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd806da10 con 0x7f6bd80a3a90 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6bc8003960 con 0x7f6bd80a3a90 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6bc8003cb0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bd57fa640 4 set_mon_vals no callback set 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.425+0000 7f6bbffff640 10 monclient: dump: 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.445 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.426+0000 7f6bdd92c640 5 monclient: authenticate success, global_id 4262 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.426+0000 7f6bdd92c640 10 monclient: _renew_subs 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.426+0000 7f6bdd92c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.426+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd80a0280 con 0x7f6bd80a3a90 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.427+0000 7f6bdd92c640 10 monclient: _renew_subs 2024-09-16T04:06:42.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.427+0000 7f6bdd92c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.427+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6bc8003ed0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.427+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6bd8071e90 con 0x7f6bd80a3a90 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.427+0000 7f6bd67fc640 10 monclient: get_auth_request con 0x7f6bc40391f0 auth_method 0 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.427+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f6bc8045bf0 con 0x7f6bd80a3a90 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.428+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.171:6808/4274747803 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bd80a7bb0 con 0x7f6bd80679c0 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.428+0000 7f6bd77fe640 10 monclient: get_auth_request con 0x7f6bd80679c0 auth_method 0 2024-09-16T04:06:42.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.429+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== osd.4 v1:172.21.15.171:6808/4274747803 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6bd80a7bb0 con 0x7f6bd80679c0 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.429+0000 7f3160680640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.429+0000 7f3160680640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.429+0000 7f3160680640 10 monclient: monmap: 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.430582+0000 2024-09-16T04:06:42.456 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.430582+0000 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.430+0000 7f3160680640 1 Processor -- start 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.430+0000 7f3160680640 1 -- start start 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.430+0000 7f3160680640 10 monclient: init 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.431+0000 7f3160680640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.431+0000 7f3160680640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.431+0000 7f3160680640 10 monclient(hunting): picked mon.noname-c con 0x7f31580a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.431+0000 7f3160680640 10 monclient(hunting): picked mon.noname-a con 0x7f31580a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315e41c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f31580a3a90 0x7f31580a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49390/0 (socket says 172.21.15.148:49390) 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315dc1b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f31580a4580 0x7f315813bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45366/0 (socket says 172.21.15.148:45366) 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315ec1d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f315813d3c0 0x7f315813f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35434/0 (socket says 172.21.15.148:35434) 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f3160680640 10 monclient(hunting): picked mon.noname-b con 0x7f315813d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315e41c640 1 -- 172.21.15.148:0/3758823176 learned_addr learned my addr 172.21.15.148:0/3758823176 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31580a0590 con 0x7f31580a4580 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31580a1270 con 0x7f31580a3a90 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31580a4fd0 con 0x7f315813d3c0 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f3160680640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f3160680640 10 monclient(hunting): authenticate will time out at 823.057275s 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041176617 0 0) 0x7f31580a4fd0 con 0x7f315813d3c0 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): my global_id is 4265 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3144003520 con 0x7f315813d3c0 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903561254 0 0) 0x7f3148001110 con 0x7f31580a4580 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 10 monclient(hunting): my global_id is 4266 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.432+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31580a4fd0 con 0x7f31580a4580 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3860846516 0 0) 0x7f3144003520 con 0x7f315813d3c0 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 >> v1:172.21.15.171:6789/0 conn(0x7f31580a3a90 legacy=0x7f31580a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 >> v1:172.21.15.148:6789/0 conn(0x7f31580a4580 legacy=0x7f315813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 monclient: found mon.noname-b 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315806e170 con 0x7f315813d3c0 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31500018c0 con 0x7f315813d3c0 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: dump: 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.460 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f3160680640 5 monclient: authenticate success, global_id 4265 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f3160680640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3150001b20 con 0x7f315813d3c0 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 1 -- 172.21.15.148:0/3758823176 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3150001f30 con 0x7f315813d3c0 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f3160680640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f3160680640 4 set_mon_vals no callback set 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.433+0000 7f315d41a640 10 monclient: dump: 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 10 monclient: shutdown 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 >> v1:172.21.15.148:6790/0 conn(0x7f315813d3c0 legacy=0x7f315813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 shutdown_connections 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 >> 172.21.15.148:0/3758823176 conn(0x7f31580a9a40 msgr2=0x7f31580a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 shutdown_connections 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.434+0000 7f3160680640 1 -- 172.21.15.148:0/3758823176 wait complete. 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.435+0000 7f3160680640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.435+0000 7f3160680640 10 monclient: monmap: 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.463 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.436+0000 7f3160680640 1 Processor -- start 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.436+0000 7f3160680640 1 -- start start 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.436+0000 7f3160680640 10 monclient: init 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.436+0000 7f3160680640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.436+0000 7f3160680640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.436+0000 7f3160680640 10 monclient(hunting): picked mon.noname-c con 0x7f31580a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f315e41c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f31580a3a90 0x7f31581c7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35442/0 (socket says 172.21.15.148:35442) 2024-09-16T04:06:42.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f315dc1b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f31580a4580 0x7f31581c8570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45368/0 (socket says 172.21.15.148:45368) 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 10 monclient(hunting): picked mon.noname-a con 0x7f31580a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f315e41c640 1 -- 172.21.15.148:0/4050053083 learned_addr learned my addr 172.21.15.148:0/4050053083 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 10 monclient(hunting): picked mon.noname-b con 0x7f315813d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f315813bbc0 con 0x7f31580a4580 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562444082 0 0) 0x7f315813bbc0 con 0x7f31580a4580 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f315813bea0 con 0x7f315813d3c0 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31581c8c80 con 0x7f31580a3a90 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f3160680640 10 monclient(hunting): authenticate will time out at 823.062724s 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f314effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f314effd640 10 monclient(hunting): my global_id is 4269 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.437+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f313c003520 con 0x7f31580a4580 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2192130985 0 0) 0x7f31581c8c80 con 0x7f31580a3a90 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): my global_id is 4268 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f315813bbc0 con 0x7f31580a3a90 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2704100295 0 0) 0x7f315813bea0 con 0x7f315813d3c0 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): my global_id is 4252 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31581c8c80 con 0x7f315813d3c0 2024-09-16T04:06:42.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2417051940 0 0) 0x7f313c003520 con 0x7f31580a4580 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f315813bea0 con 0x7f31580a4580 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3148001a00 con 0x7f31580a4580 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): dump: 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.467 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3775358630 0 0) 0x7f315813bbc0 con 0x7f31580a3a90 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f313c003520 con 0x7f31580a3a90 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1241659021 0 0) 0x7f315813bea0 con 0x7f31580a4580 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 >> v1:172.21.15.148:6790/0 conn(0x7f31580a3a90 legacy=0x7f31581c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 >> v1:172.21.15.171:6789/0 conn(0x7f315813d3c0 legacy=0x7f31581d3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 monclient: found mon.a 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315813ac10 con 0x7f31580a4580 2024-09-16T04:06:42.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f314effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 5 monclient: authenticate success, global_id 4269 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 10 monclient: _renew_subs 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31580a81c0 con 0x7f31580a4580 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 10 monclient: _renew_subs 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.438+0000 7f3160680640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f315813c8f0 con 0x7f31580a4580 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3148001140 con 0x7f31580a4580 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f314effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3148004a90 con 0x7f31580a4580 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f314effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f314effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f314effd640 10 monclient: dump: 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.439+0000 7f315cc19640 4 set_mon_vals no callback set 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.440+0000 7f314cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.440+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.148:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f3134000f80 con 0x7f31580a4580 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.441+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f314800fec0 con 0x7f31580a4580 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.441+0000 7f4ec5ef8640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.441+0000 7f4ec5ef8640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.441+0000 7f4ec5ef8640 10 monclient: monmap: 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.442635+0000 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.442635+0000 2024-09-16T04:06:42.471 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.472 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.472 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.472 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.472 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.472 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.441+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f3148046a30 con 0x7f31580a4580 2024-09-16T04:06:42.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.171:6800/466524840 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f313c043880 con 0x7f313c03fbd0 2024-09-16T04:06:42.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f31480115b0 con 0x7f31580a4580 2024-09-16T04:06:42.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f314effd640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T04:06:42.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f315e41c640 10 monclient: get_auth_request con 0x7f313c038f80 auth_method 0 2024-09-16T04:06:42.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f315ec1d640 10 monclient: get_auth_request con 0x7f313c03fbd0 auth_method 0 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f4ec5ef8640 1 Processor -- start 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f4ec5ef8640 1 -- start start 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.442+0000 7f4ec5ef8640 10 monclient: init 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec00a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec00a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec013d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec00a0590 con 0x7f4ec00a3a90 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec00a1270 con 0x7f4ec00a4580 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec00a4fd0 con 0x7f4ec013d3c0 2024-09-16T04:06:42.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ec5ef8640 10 monclient(hunting): authenticate will time out at 823.068728s 2024-09-16T04:06:42.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== osd.5 v1:172.21.15.171:6800/466524840 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f313c043880 con 0x7f313c03fbd0 2024-09-16T04:06:42.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.443+0000 7f4ebeffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4ec00a4580 0x7f4ec013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49398/0 (socket says 172.21.15.148:49398) 2024-09-16T04:06:42.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebeffd640 1 -- 172.21.15.148:0/879793846 learned_addr learned my addr 172.21.15.148:0/879793846 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3478081688 0 0) 0x7f4ec00a1270 con 0x7f4ec00a4580 2024-09-16T04:06:42.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): my global_id is 4255 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ea8003520 con 0x7f4ec00a4580 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311217408 0 0) 0x7f4ec00a4fd0 con 0x7f4ec013d3c0 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient(hunting): my global_id is 4271 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec00a1270 con 0x7f4ec013d3c0 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364627155 0 0) 0x7f4ea8003520 con 0x7f4ec00a4580 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 >> v1:172.21.15.148:6790/0 conn(0x7f4ec013d3c0 legacy=0x7f4ec013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 >> v1:172.21.15.148:6789/0 conn(0x7f4ec00a3a90 legacy=0x7f4ec00a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 monclient: found mon.noname-c 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec006e170 con 0x7f4ec00a4580 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4eb4001610 con 0x7f4ec00a4580 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ebe7fc640 10 monclient: dump: 2024-09-16T04:06:42.478 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ec5ef8640 5 monclient: authenticate success, global_id 4255 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.444+0000 7f4ec5ef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4eb4001870 con 0x7f4ec00a4580 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ebe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ebe7fc640 1 -- 172.21.15.148:0/879793846 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4eb4001c40 con 0x7f4ec00a4580 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ec5ef8640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ec5ef8640 4 set_mon_vals no callback set 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ebe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ebe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.445+0000 7f4ebe7fc640 10 monclient: dump: 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.480 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.446+0000 7f4ec5ef8640 10 monclient: shutdown 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.446+0000 7f4ec5ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.446+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/879793846 >> v1:172.21.15.171:6789/0 conn(0x7f4ec00a4580 legacy=0x7f4ec013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.447+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/879793846 shutdown_connections 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.447+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/879793846 >> 172.21.15.148:0/879793846 conn(0x7f4ec00a9a40 msgr2=0x7f4ec00a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.447+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/879793846 shutdown_connections 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.447+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/879793846 wait complete. 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.448+0000 7f4ec5ef8640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.448+0000 7f4ec5ef8640 10 monclient: monmap: 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.481 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.448+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 --> v1:172.21.15.171:6808/4274747803 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6bd80a4bc0 con 0x7f6bd80679c0 2024-09-16T04:06:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.448+0000 7f4ec5ef8640 1 Processor -- start 2024-09-16T04:06:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 1 -- start start 2024-09-16T04:06:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bbffff640 1 -- 172.21.15.148:0/2620185118 <== osd.4 v1:172.21.15.171:6808/4274747803 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 258058390) 0x7f6bd80a4bc0 con 0x7f6bd80679c0 2024-09-16T04:06:42.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient: init 2024-09-16T04:06:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 >> v1:172.21.15.171:6808/4274747803 conn(0x7f6bd80679c0 legacy=0x7f6bd8067da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:42.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec00a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec00a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec013d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec006f960 con 0x7f4ec00a4580 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec013ce60 con 0x7f4ec013d3c0 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec013b7e0 con 0x7f4ec00a3a90 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f4ec5ef8640 10 monclient(hunting): authenticate will time out at 823.074984s 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 >> v1:172.21.15.171:6812/7327273 conn(0x7f6bc40391f0 legacy=0x7f6bc403b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bdd92c640 10 monclient: shutdown 2024-09-16T04:06:42.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bdd92c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 >> v1:172.21.15.148:6790/0 conn(0x7f6bd80a3a90 legacy=0x7f6bd8136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.449+0000 7f6bd77fe640 1 -- 172.21.15.148:0/2620185118 reap_dead start 2024-09-16T04:06:42.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ebf7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f4ec00a3a90 0x7f4ec0136eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35456/0 (socket says 172.21.15.148:35456) 2024-09-16T04:06:42.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ebf7fe640 1 -- 172.21.15.148:0/3330036362 learned_addr learned my addr 172.21.15.148:0/3330036362 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 shutdown_connections 2024-09-16T04:06:42.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 >> 172.21.15.148:0/2620185118 conn(0x7f6bd80a9a40 msgr2=0x7f6bd80a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845367963 0 0) 0x7f4ec013ce60 con 0x7f4ec013d3c0 2024-09-16T04:06:42.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 shutdown_connections 2024-09-16T04:06:42.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f6bdd92c640 1 -- 172.21.15.148:0/2620185118 wait complete. 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): my global_id is 4258 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4eac003520 con 0x7f4ec013d3c0 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3417561546 0 0) 0x7f4ec013b7e0 con 0x7f4ec00a3a90 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): my global_id is 4274 2024-09-16T04:06:42.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec013ce60 con 0x7f4ec00a3a90 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4137844998 0 0) 0x7f4ec006f960 con 0x7f4ec00a4580 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): my global_id is 4272 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec013b7e0 con 0x7f4ec00a4580 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4197454514 0 0) 0x7f4ec013ce60 con 0x7f4ec00a3a90 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ec006f960 con 0x7f4ec00a3a90 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4eb0001930 con 0x7f4ec00a3a90 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): dump: 2024-09-16T04:06:42.492 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.450+0000 7f4ea7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3056564860 0 0) 0x7f4ec013b7e0 con 0x7f4ec00a4580 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ec013ce60 con 0x7f4ec00a4580 2024-09-16T04:06:42.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4eb4001a20 con 0x7f4ec00a4580 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient(hunting): dump: 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2270780435 0 0) 0x7f4ec006f960 con 0x7f4ec00a3a90 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 >> v1:172.21.15.171:6789/0 conn(0x7f4ec013d3c0 legacy=0x7f4ec01d8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 >> v1:172.21.15.148:6789/0 conn(0x7f4ec00a4580 legacy=0x7f4ec01d4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 monclient: found mon.c 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec006eeb0 con 0x7f4ec00a3a90 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 5 monclient: authenticate success, global_id 4274 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 10 monclient: _renew_subs 2024-09-16T04:06:42.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec006f680 con 0x7f4ec00a3a90 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 10 monclient: _renew_subs 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7f4ec5ef8640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ec0072840 con 0x7f4ec00a3a90 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4eb0001c70 con 0x7f4ec00a3a90 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4eb0003a60 con 0x7f4ec00a3a90 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea7fff640 10 monclient: dump: 2024-09-16T04:06:42.496 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ebdffb640 4 set_mon_vals no callback set 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7fef6ffff640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7fef6ffff640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.451+0000 7fef6ffff640 10 monclient: monmap: 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.452759+0000 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.452759+0000 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.498 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7fef6ffff640 1 Processor -- start 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4ec006fba0 con 0x7f4ec00a3a90 2024-09-16T04:06:42.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4eb0003c00 con 0x7f4ec00a3a90 2024-09-16T04:06:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fe87fb2c640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fe87fb2c640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f4eb0046d70 con 0x7f4ec00a3a90 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fe87fb2c640 10 monclient: monmap: 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.454239+0000 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.454239+0000 2024-09-16T04:06:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7fef6ffff640 1 -- start start 2024-09-16T04:06:42.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.452+0000 7fef6ffff640 10 monclient: init 2024-09-16T04:06:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6800/3044422561 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4eb4001080 con 0x7f4eac03fee0 2024-09-16T04:06:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f4eb00150e0 con 0x7f4ec00a3a90 2024-09-16T04:06:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ea7fff640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T04:06:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:42.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fef7009eb30 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fef7009bdb0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fef70139240 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7009c190 con 0x7fef70139240 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7009b880 con 0x7fef7009bdb0 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7009ef10 con 0x7fef7009eb30 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6ffff640 10 monclient(hunting): authenticate will time out at 823.078753s 2024-09-16T04:06:42.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ebffff640 10 monclient: get_auth_request con 0x7f4eac03fee0 auth_method 0 2024-09-16T04:06:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7f4ebeffd640 10 monclient: get_auth_request con 0x7f4eac039300 auth_method 0 2024-09-16T04:06:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6effd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fef7009eb30 0x7fef7009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35470/0 (socket says 172.21.15.148:35470) 2024-09-16T04:06:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.453+0000 7fef6effd640 1 -- 172.21.15.148:0/3729304751 learned_addr learned my addr 172.21.15.148:0/3729304751 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fe87fb2c640 1 Processor -- start 2024-09-16T04:06:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1146610572 0 0) 0x7fef7009c190 con 0x7fef70139240 2024-09-16T04:06:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fe87fb2c640 1 -- start start 2024-09-16T04:06:42.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fe87fb2c640 10 monclient: init 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): my global_id is 4275 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef5c003520 con 0x7fef70139240 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3893864458 0 0) 0x7fef7009b880 con 0x7fef7009bdb0 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): my global_id is 4261 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7009c190 con 0x7fef7009bdb0 2024-09-16T04:06:42.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364572929 0 0) 0x7fef7009ef10 con 0x7fef7009eb30 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient(hunting): my global_id is 4277 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7009b880 con 0x7fef7009eb30 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2496082766 0 0) 0x7fef7009c190 con 0x7fef7009bdb0 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 >> v1:172.21.15.148:6790/0 conn(0x7fef7009eb30 legacy=0x7fef7009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 >> v1:172.21.15.148:6789/0 conn(0x7fef70139240 legacy=0x7fef7013b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 monclient: found mon.noname-c 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7009cba0 con 0x7fef7009bdb0 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef580015f0 con 0x7fef7009bdb0 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: dump: 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.509 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.510 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6ffff640 5 monclient: authenticate success, global_id 4261 2024-09-16T04:06:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== osd.1 v1:172.21.15.148:6800/3044422561 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4eb4001080 con 0x7f4eac03fee0 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient(hunting): picked mon.noname-b con 0x7fe8780a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient(hunting): picked mon.noname-a con 0x7fe8780a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient(hunting): picked mon.noname-c con 0x7fe87813d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8780a0590 con 0x7fe8780a4580 2024-09-16T04:06:42.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef58001850 con 0x7fef7009bdb0 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 1 -- 172.21.15.148:0/3729304751 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef58001be0 con 0x7fef7009bdb0 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6dffb640 10 monclient: dump: 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.512 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6ffff640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.454+0000 7fef6ffff640 4 set_mon_vals no callback set 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8780a1270 con 0x7fe87813d3c0 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8780a4fd0 con 0x7fe8780a3a90 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87fb2c640 10 monclient(hunting): authenticate will time out at 823.080238s 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87d8c8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe8780a3a90 0x7fe8780a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35472/0 (socket says 172.21.15.148:35472) 2024-09-16T04:06:42.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87d8c8640 1 -- 172.21.15.148:0/3570835432 learned_addr learned my addr 172.21.15.148:0/3570835432 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fef6ffff640 10 monclient: shutdown 2024-09-16T04:06:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fef6ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fef6ffff640 1 -- 172.21.15.148:0/3729304751 >> v1:172.21.15.171:6789/0 conn(0x7fef7009bdb0 legacy=0x7fef7009b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622182948 0 0) 0x7fe8780a4fd0 con 0x7fe8780a3a90 2024-09-16T04:06:42.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87c8c6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fef6ffff640 1 -- 172.21.15.148:0/3729304751 shutdown_connections 2024-09-16T04:06:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fef6ffff640 1 -- 172.21.15.148:0/3729304751 >> 172.21.15.148:0/3729304751 conn(0x7fef700a1530 msgr2=0x7fef700a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87c8c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87c8c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7f5c754dd640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7f5c754dd640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7f5c754dd640 10 monclient: monmap: 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.457106+0000 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.457106+0000 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.516 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87c8c6640 10 monclient(hunting): my global_id is 4280 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.455+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe864003520 con 0x7fe8780a3a90 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268231519 0 0) 0x7fe8780a1270 con 0x7fe87813d3c0 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): my global_id is 4264 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8780a4fd0 con 0x7fe87813d3c0 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3146421723 0 0) 0x7fe874001110 con 0x7fe8780a4580 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient(hunting): my global_id is 4278 2024-09-16T04:06:42.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8780a1270 con 0x7fe8780a4580 2024-09-16T04:06:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7f5c754dd640 1 Processor -- start 2024-09-16T04:06:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fef6ffff640 1 -- 172.21.15.148:0/3729304751 shutdown_connections 2024-09-16T04:06:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7f5c754dd640 1 -- start start 2024-09-16T04:06:42.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7f5c754dd640 10 monclient: init 2024-09-16T04:06:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 851436057 0 0) 0x7fe864003520 con 0x7fe8780a3a90 2024-09-16T04:06:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 >> v1:172.21.15.171:6789/0 conn(0x7fe87813d3c0 legacy=0x7fe87813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 >> v1:172.21.15.148:6789/0 conn(0x7fe8780a4580 legacy=0x7fe87813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 monclient: found mon.noname-b 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87806e170 con 0x7fe8780a3a90 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe86c001610 con 0x7fe8780a3a90 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.456+0000 7fe87c8c6640 10 monclient: dump: 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.521 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87fb2c640 5 monclient: authenticate success, global_id 4280 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87fb2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe86c001870 con 0x7fe8780a3a90 2024-09-16T04:06:42.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87c8c6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87fb2c640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87fb2c640 4 set_mon_vals no callback set 2024-09-16T04:06:42.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87c8c6640 1 -- 172.21.15.148:0/3570835432 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe86c001c40 con 0x7fe8780a3a90 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87c8c6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87c8c6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fe87c8c6640 10 monclient: dump: 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.524 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.525 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient(hunting): picked mon.noname-c con 0x7f5c700a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient(hunting): picked mon.noname-b con 0x7f5c700a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient(hunting): picked mon.noname-a con 0x7f5c7013d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c700a0590 con 0x7f5c7013d3c0 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c700a1270 con 0x7f5c700a3a90 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c700a4fd0 con 0x7f5c700a4580 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c754dd640 10 monclient(hunting): authenticate will time out at 823.082442s 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c6f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5c7013d3c0 0x7f5c7013f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45424/0 (socket says 172.21.15.148:45424) 2024-09-16T04:06:42.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7f5c6f7fe640 1 -- 172.21.15.148:0/907277948 learned_addr learned my addr 172.21.15.148:0/907277948 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.457+0000 7fef6ffff640 1 -- 172.21.15.148:0/3729304751 wait complete. 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782131408 0 0) 0x7f5c680013c0 con 0x7f5c7013d3c0 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): my global_id is 4281 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c58003520 con 0x7f5c7013d3c0 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55449213 0 0) 0x7f5c700a4fd0 con 0x7f5c700a4580 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fef6ffff640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fef6ffff640 10 monclient: monmap: 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 10 monclient: shutdown 2024-09-16T04:06:42.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient(hunting): my global_id is 4283 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c680013c0 con 0x7f5c700a4580 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1535274380 0 0) 0x7f5c58003520 con 0x7f5c7013d3c0 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 >> v1:172.21.15.148:6790/0 conn(0x7f5c700a4580 legacy=0x7f5c7013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 >> v1:172.21.15.171:6789/0 conn(0x7f5c700a3a90 legacy=0x7f5c700a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 monclient: found mon.noname-a 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c7006e170 con 0x7f5c7013d3c0 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c68001a90 con 0x7f5c7013d3c0 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: dump: 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.532 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.533 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c754dd640 5 monclient: authenticate success, global_id 4281 2024-09-16T04:06:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c754dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fef6ffff640 1 Processor -- start 2024-09-16T04:06:42.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 1 -- 172.21.15.148:0/3570835432 >> v1:172.21.15.148:6790/0 conn(0x7fe8780a3a90 legacy=0x7fe8780a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 1 -- 172.21.15.148:0/3570835432 shutdown_connections 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c68001cf0 con 0x7f5c7013d3c0 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c754dd640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c754dd640 4 set_mon_vals no callback set 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 1 -- 172.21.15.148:0/907277948 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c680020c0 con 0x7f5c7013d3c0 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7f5c6dffb640 10 monclient: dump: 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.534 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 1 -- 172.21.15.148:0/3570835432 >> 172.21.15.148:0/3570835432 conn(0x7fe8780a9a40 msgr2=0x7fe8780a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 1 -- 172.21.15.148:0/3570835432 shutdown_connections 2024-09-16T04:06:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fef6ffff640 1 -- start start 2024-09-16T04:06:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fef6ffff640 10 monclient: init 2024-09-16T04:06:42.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.458+0000 7fe87fb2c640 1 -- 172.21.15.148:0/3570835432 wait complete. 2024-09-16T04:06:42.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fef7009bdb0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fef7009eb30 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fef70139240 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef701203d0 con 0x7fef70139240 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef70121330 con 0x7fef7009bdb0 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef701218f0 con 0x7fef7009eb30 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6ffff640 10 monclient(hunting): authenticate will time out at 823.084506s 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6e7fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fef7009eb30 0x7fef701d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35498/0 (socket says 172.21.15.148:35498) 2024-09-16T04:06:42.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fef6e7fc640 1 -- 172.21.15.148:0/782432782 learned_addr learned my addr 172.21.15.148:0/782432782 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fe87fb2c640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.459+0000 7fe87fb2c640 10 monclient: monmap: 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.539 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3103992611 0 0) 0x7fef701218f0 con 0x7fef7009eb30 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 10 monclient: shutdown 2024-09-16T04:06:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 1 -- 172.21.15.148:0/907277948 >> v1:172.21.15.148:6789/0 conn(0x7f5c7013d3c0 legacy=0x7f5c7013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 1 -- 172.21.15.148:0/907277948 shutdown_connections 2024-09-16T04:06:42.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 1 -- 172.21.15.148:0/907277948 >> 172.21.15.148:0/907277948 conn(0x7f5c700a9a40 msgr2=0x7f5c700a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): my global_id is 4286 2024-09-16T04:06:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef50003550 con 0x7fef7009eb30 2024-09-16T04:06:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768329798 0 0) 0x7fef701203d0 con 0x7fef70139240 2024-09-16T04:06:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fe87fb2c640 1 Processor -- start 2024-09-16T04:06:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fe87fb2c640 1 -- start start 2024-09-16T04:06:42.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fe87fb2c640 10 monclient: init 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): my global_id is 4284 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef701218f0 con 0x7fef70139240 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2974830075 0 0) 0x7fef50003550 con 0x7fef7009eb30 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef701203d0 con 0x7fef7009eb30 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef580035f0 con 0x7fef7009eb30 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): dump: 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.544 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 1 -- 172.21.15.148:0/907277948 shutdown_connections 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7f5c754dd640 1 -- 172.21.15.148:0/907277948 wait complete. 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7f5c754dd640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7f5c754dd640 10 monclient: monmap: 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.546 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3495756243 0 0) 0x7fef701203d0 con 0x7fef7009eb30 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 >> v1:172.21.15.171:6789/0 conn(0x7fef7009bdb0 legacy=0x7fef70132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 >> v1:172.21.15.148:6789/0 conn(0x7fef70139240 legacy=0x7fef701d3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 monclient: found mon.c 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef70132b70 con 0x7fef7009eb30 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.460+0000 7fef67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient(hunting): picked mon.noname-a con 0x7fe8780a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient(hunting): picked mon.noname-c con 0x7fe8780a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient(hunting): picked mon.noname-b con 0x7fe87813d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87813cef0 con 0x7fe8780a3a90 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87809b270 con 0x7fe87813d3c0 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87813ad30 con 0x7fe8780a4580 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87fb2c640 10 monclient(hunting): authenticate will time out at 823.086319s 2024-09-16T04:06:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87d8c8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe8780a3a90 0x7fe878139f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45446/0 (socket says 172.21.15.148:45446) 2024-09-16T04:06:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe87d8c8640 1 -- 172.21.15.148:0/1793115986 learned_addr learned my addr 172.21.15.148:0/1793115986 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef58003b00 con 0x7fef7009eb30 2024-09-16T04:06:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef67fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef58003ed0 con 0x7fef7009eb30 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef67fff640 10 monclient: dump: 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.551 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7f5c754dd640 1 Processor -- start 2024-09-16T04:06:42.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6cff9640 4 set_mon_vals no callback set 2024-09-16T04:06:42.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 127970572 0 0) 0x7fe87806e170 con 0x7fe8780a3a90 2024-09-16T04:06:42.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe86a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe86a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe86a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe86a7fc640 10 monclient(hunting): my global_id is 4287 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe85c003520 con 0x7fe8780a3a90 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940672837 0 0) 0x7fe87813ad30 con 0x7fe8780a4580 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): my global_id is 4289 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87806e170 con 0x7fe8780a4580 2024-09-16T04:06:42.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 5 monclient: authenticate success, global_id 4286 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 10 monclient: _renew_subs 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef70071ff0 con 0x7fef7009eb30 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 10 monclient: _renew_subs 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fef70120150 con 0x7fef7009eb30 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7f5c754dd640 1 -- start start 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.461+0000 7f5c754dd640 10 monclient: init 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:42.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient(hunting): picked mon.noname-a con 0x7f5c700a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient(hunting): picked mon.noname-c con 0x7f5c700a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient(hunting): picked mon.noname-b con 0x7f5c7013d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c7013ccd0 con 0x7f5c700a3a90 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c7013c460 con 0x7f5c7013d3c0 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c7013ced0 con 0x7f5c700a4580 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c754dd640 10 monclient(hunting): authenticate will time out at 823.087569s 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c6effd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5c700a3a90 0x7f5c70138860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45458/0 (socket says 172.21.15.148:45458) 2024-09-16T04:06:42.556 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7f5c6effd640 1 -- 172.21.15.148:0/1996625827 learned_addr learned my addr 172.21.15.148:0/1996625827 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2624432137 0 0) 0x7fe87806e170 con 0x7fe8780a4580 2024-09-16T04:06:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe87813ad30 con 0x7fe8780a4580 2024-09-16T04:06:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe8740018e0 con 0x7fe8780a4580 2024-09-16T04:06:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fef6ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fef7009bb90 con 0x7fef7009eb30 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.558 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982582414 0 0) 0x7fe87809b270 con 0x7fe87813d3c0 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 10 monclient(hunting): my global_id is 4273 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.462+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87806e170 con 0x7fe87813d3c0 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2499827806 0 0) 0x7fe85c003520 con 0x7fe8780a3a90 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe87809b270 con 0x7fe8780a3a90 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 402581754 0 0) 0x7fe87813ad30 con 0x7fe8780a4580 2024-09-16T04:06:42.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 >> v1:172.21.15.171:6789/0 conn(0x7fe87813d3c0 legacy=0x7fe8781d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 >> v1:172.21.15.148:6789/0 conn(0x7fe8780a3a90 legacy=0x7fe878139f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 monclient: found mon.c 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 663327178 0 0) 0x7f5c7013ced0 con 0x7f5c700a4580 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): my global_id is 4292 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c5c003520 con 0x7f5c700a4580 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881260797 0 0) 0x7f5c7013ccd0 con 0x7f5c700a3a90 2024-09-16T04:06:42.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): my global_id is 4290 2024-09-16T04:06:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c7013ced0 con 0x7f5c700a3a90 2024-09-16T04:06:42.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488367075 0 0) 0x7f5c7013c460 con 0x7f5c7013d3c0 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): my global_id is 4276 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c7013ccd0 con 0x7f5c7013d3c0 2024-09-16T04:06:42.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3892330701 0 0) 0x7f5c5c003520 con 0x7f5c700a4580 2024-09-16T04:06:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c7013c460 con 0x7f5c700a4580 2024-09-16T04:06:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c640018d0 con 0x7f5c700a4580 2024-09-16T04:06:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fef58004070 con 0x7fef7009eb30 2024-09-16T04:06:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87809fac0 con 0x7fe8780a4580 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7fe86a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe87fb2c640 5 monclient: authenticate success, global_id 4289 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe87fb2c640 10 monclient: _renew_subs 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe87fb2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe87809b830 con 0x7fe8780a4580 2024-09-16T04:06:42.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): dump: 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2215881392 0 0) 0x7f5c7013ced0 con 0x7f5c700a3a90 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c5c003520 con 0x7f5c700a3a90 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c60001930 con 0x7f5c700a3a90 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): dump: 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.567 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 193113868 0 0) 0x7f5c7013ccd0 con 0x7f5c7013d3c0 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c7013ced0 con 0x7f5c7013d3c0 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe874003610 con 0x7fe8780a4580 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe874003960 con 0x7fe8780a4580 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86a7fc640 10 monclient: dump: 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fe86b7fe640 4 set_mon_vals no callback set 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c68001990 con 0x7f5c7013d3c0 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.463+0000 7f5c577fe640 10 monclient(hunting): dump: 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.571 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 126378862 0 0) 0x7f5c7013c460 con 0x7f5c700a4580 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 >> v1:172.21.15.171:6789/0 conn(0x7f5c7013d3c0 legacy=0x7f5c701d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 >> v1:172.21.15.148:6789/0 conn(0x7f5c700a3a90 legacy=0x7f5c70138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 monclient: found mon.c 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c700a8030 con 0x7f5c700a4580 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fef58046cc0 con 0x7fef7009eb30 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6808/985869260 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef50043af0 con 0x7fef5003fe40 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7fef580470c0 con 0x7fef7009eb30 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fef67fff640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fef6effd640 10 monclient: get_auth_request con 0x7fef500391f0 auth_method 0 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7fef6f7fe640 10 monclient: get_auth_request con 0x7fef5003fe40 auth_method 0 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.465+0000 7fe87fb2c640 10 monclient: _renew_subs 2024-09-16T04:06:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.465+0000 7fe87fb2c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.465+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe87813cba0 con 0x7fe8780a4580 2024-09-16T04:06:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 5 monclient: authenticate success, global_id 4292 2024-09-16T04:06:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 10 monclient: _renew_subs 2024-09-16T04:06:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c7013bea0 con 0x7f5c700a4580 2024-09-16T04:06:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 10 monclient: _renew_subs 2024-09-16T04:06:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c754dd640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c7013a9e0 con 0x7f5c700a4580 2024-09-16T04:06:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c64003610 con 0x7f5c700a4580 2024-09-16T04:06:42.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.465+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe87400ebf0 con 0x7fe8780a4580 2024-09-16T04:06:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c64003960 con 0x7f5c700a4580 2024-09-16T04:06:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c577fe640 10 monclient: dump: 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.580 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.464+0000 7f5c6cff9640 4 set_mon_vals no callback set 2024-09-16T04:06:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.465+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fe874045bd0 con 0x7fe8780a4580 2024-09-16T04:06:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5c50000f80 con 0x7f5c700a4580 2024-09-16T04:06:42.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7fe87d8c8640 10 monclient: get_auth_request con 0x7fe85c038e50 auth_method 0 2024-09-16T04:06:42.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.171:6804/772155526 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8780a4bc0 con 0x7fe8780a76a0 2024-09-16T04:06:42.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7fe87e0c9640 10 monclient: get_auth_request con 0x7fe8780a76a0 auth_method 0 2024-09-16T04:06:42.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5c64003960 con 0x7f5c700a4580 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f5c64045970 con 0x7f5c700a4580 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6804/262588452 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c5c04c3c0 con 0x7f5c5c048710 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f5c64045d80 con 0x7f5c700a4580 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.466+0000 7f5c577fe640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.467+0000 7f5c6f7fe640 10 monclient: get_auth_request con 0x7f5c5c048710 auth_method 0 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.467+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== osd.0 v1:172.21.15.148:6808/985869260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fef50043af0 con 0x7fef5003fe40 2024-09-16T04:06:42.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.467+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== osd.3 v1:172.21.15.171:6804/772155526 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe8780a4bc0 con 0x7fe8780a76a0 2024-09-16T04:06:42.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.468+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== osd.2 v1:172.21.15.148:6804/262588452 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5c5c04c3c0 con 0x7f5c5c048710 2024-09-16T04:06:42.584 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2024-09-16T04:06:42.585 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:06:42.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.471+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 --> v1:172.21.15.171:6800/466524840 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3134002c50 con 0x7f313c03fbd0 2024-09-16T04:06:42.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.471+0000 7f314effd640 1 -- 172.21.15.148:0/4050053083 <== osd.5 v1:172.21.15.171:6800/466524840 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7f3134002c50 con 0x7f313c03fbd0 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 >> v1:172.21.15.171:6800/466524840 conn(0x7f313c03fbd0 legacy=0x7f313c042010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 >> v1:172.21.15.171:6812/7327273 conn(0x7f313c038f80 legacy=0x7f313c03b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 10 monclient: shutdown 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 >> v1:172.21.15.148:6789/0 conn(0x7f31580a4580 legacy=0x7f31581c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f315ec1d640 1 -- 172.21.15.148:0/4050053083 reap_dead start 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 shutdown_connections 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 >> 172.21.15.148:0/4050053083 conn(0x7f31580a9a40 msgr2=0x7f31580a8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 shutdown_connections 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.472+0000 7f314cff9640 1 -- 172.21.15.148:0/4050053083 wait complete. 2024-09-16T04:06:42.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.475+0000 7f5c6effd640 10 monclient: get_auth_request con 0x7f5c5c041ac0 auth_method 0 2024-09-16T04:06:42.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.475+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 --> v1:172.21.15.148:6800/3044422561 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ec01375c0 con 0x7f4eac03fee0 2024-09-16T04:06:42.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.475+0000 7f4ea7fff640 1 -- 172.21.15.148:0/3330036362 <== osd.1 v1:172.21.15.148:6800/3044422561 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4245302165) 0x7f4ec01375c0 con 0x7f4eac03fee0 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.475+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 >> v1:172.21.15.148:6800/3044422561 conn(0x7f4eac03fee0 legacy=0x7f4eac042320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 >> v1:172.21.15.171:6812/7327273 conn(0x7f4eac039300 legacy=0x7f4eac03b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 10 monclient: shutdown 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 >> v1:172.21.15.148:6790/0 conn(0x7f4ec00a3a90 legacy=0x7f4ec0136eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ebffff640 1 -- 172.21.15.148:0/3330036362 reap_dead start 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 shutdown_connections 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 >> 172.21.15.148:0/3330036362 conn(0x7f4ec00a9a40 msgr2=0x7f4ec00a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 shutdown_connections 2024-09-16T04:06:42.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.476+0000 7f4ea5ffb640 1 -- 172.21.15.148:0/3330036362 wait complete. 2024-09-16T04:06:42.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.481+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 --> v1:172.21.15.148:6808/985869260 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fef700719f0 con 0x7fef5003fe40 2024-09-16T04:06:42.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.482+0000 7fef67fff640 1 -- 172.21.15.148:0/782432782 <== osd.0 v1:172.21.15.148:6808/985869260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7fef700719f0 con 0x7fef5003fe40 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.482+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 >> v1:172.21.15.148:6808/985869260 conn(0x7fef5003fe40 legacy=0x7fef50042280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.482+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 >> v1:172.21.15.171:6812/7327273 conn(0x7fef500391f0 legacy=0x7fef5003b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 10 monclient: shutdown 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 >> v1:172.21.15.148:6790/0 conn(0x7fef7009eb30 legacy=0x7fef701d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6f7fe640 1 -- 172.21.15.148:0/782432782 reap_dead start 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 shutdown_connections 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 >> 172.21.15.148:0/782432782 conn(0x7fef700a1530 msgr2=0x7fef700a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 shutdown_connections 2024-09-16T04:06:42.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.483+0000 7fef6ffff640 1 -- 172.21.15.148:0/782432782 wait complete. 2024-09-16T04:06:42.592 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:42.592 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:06:42.593 INFO:teuthology.orchestra.run.smithi148.stdout:55834574855 2024-09-16T04:06:42.593 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:06:42.594 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:42.594 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:06:42.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.498+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 --> v1:172.21.15.171:6804/772155526 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8780a7a80 con 0x7fe8780a76a0 2024-09-16T04:06:42.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.498+0000 7fe86a7fc640 1 -- 172.21.15.148:0/1793115986 <== osd.3 v1:172.21.15.171:6804/772155526 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7fe8780a7a80 con 0x7fe8780a76a0 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 >> v1:172.21.15.171:6804/772155526 conn(0x7fe8780a76a0 legacy=0x7fe8780679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 >> v1:172.21.15.171:6812/7327273 conn(0x7fe85c038e50 legacy=0x7fe85c03b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 10 monclient: shutdown 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 >> v1:172.21.15.148:6790/0 conn(0x7fe8780a4580 legacy=0x7fe87813a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87e0c9640 1 -- 172.21.15.148:0/1793115986 reap_dead start 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 shutdown_connections 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 >> 172.21.15.148:0/1793115986 conn(0x7fe8780a9a40 msgr2=0x7fe8780a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 shutdown_connections 2024-09-16T04:06:42.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7fe87fb2c640 1 -- 172.21.15.148:0/1793115986 wait complete. 2024-09-16T04:06:42.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 --> v1:172.21.15.148:6804/262588452 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c50002c50 con 0x7f5c5c048710 2024-09-16T04:06:42.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.499+0000 7f5c577fe640 1 -- 172.21.15.148:0/1996625827 <== osd.2 v1:172.21.15.148:6804/262588452 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7f5c50002c50 con 0x7f5c5c048710 2024-09-16T04:06:42.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 >> v1:172.21.15.148:6804/262588452 conn(0x7f5c5c048710 legacy=0x7f5c5c04ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 >> v1:172.21.15.171:6812/7327273 conn(0x7f5c5c041ac0 legacy=0x7f5c5c043f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 10 monclient: shutdown 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 >> v1:172.21.15.148:6790/0 conn(0x7f5c700a4580 legacy=0x7f5c70138f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c6f7fe640 1 -- 172.21.15.148:0/1996625827 reap_dead start 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 shutdown_connections 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 >> 172.21.15.148:0/1996625827 conn(0x7f5c700a9a40 msgr2=0x7f5c700a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 shutdown_connections 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.500+0000 7f5c557fa640 1 -- 172.21.15.148:0/1996625827 wait complete. 2024-09-16T04:06:42.599 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:42.600 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:42.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:06:42.601 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f8d24d42640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f8d24d42640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f8d24d42640 10 monclient: monmap: 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.744077+0000 2024-09-16T04:06:42.745 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.744077+0000 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f2527435640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f2527435640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f2527435640 10 monclient: monmap: 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.744635+0000 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.744635+0000 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.747 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.743+0000 7f8d24d42640 1 Processor -- start 2024-09-16T04:06:42.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d24d42640 1 -- start start 2024-09-16T04:06:42.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d24d42640 10 monclient: init 2024-09-16T04:06:42.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f2527435640 1 Processor -- start 2024-09-16T04:06:42.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f2527435640 1 -- start start 2024-09-16T04:06:42.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f2527435640 10 monclient: init 2024-09-16T04:06:42.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d24d42640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d24d42640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d24d42640 10 monclient(hunting): picked mon.noname-a con 0x7f8d200a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d24d42640 10 monclient(hunting): picked mon.noname-c con 0x7f8d200a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.744+0000 7f8d1e59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8d200a2410 0x7f8d200a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45470/0 (socket says 172.21.15.148:45470) 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1e59c640 1 -- 172.21.15.148:0/3547048352 learned_addr learned my addr 172.21.15.148:0/3547048352 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d24d42640 10 monclient(hunting): picked mon.noname-b con 0x7f8d2013d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d200a0e00 con 0x7f8d200a2410 2024-09-16T04:06:42.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:42.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient(hunting): picked mon.noname-a con 0x7f25200a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d200a10e0 con 0x7f8d200a1b10 2024-09-16T04:06:42.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d200a4970 con 0x7f8d2013d060 2024-09-16T04:06:42.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d24d42640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d24d42640 10 monclient(hunting): authenticate will time out at 823.370517s 2024-09-16T04:06:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f25249d0640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f25200a4580 0x7f252013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35544/0 (socket says 172.21.15.148:35544) 2024-09-16T04:06:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f25249d0640 1 -- 172.21.15.148:0/341538863 learned_addr learned my addr 172.21.15.148:0/341538863 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient(hunting): picked mon.noname-b con 0x7f25200a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3215923823 0 0) 0x7f8d200a10e0 con 0x7f8d200a1b10 2024-09-16T04:06:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): my global_id is 4279 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d08003520 con 0x7f8d200a1b10 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566340234 0 0) 0x7f8d200a0e00 con 0x7f8d200a2410 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 10 monclient(hunting): my global_id is 4293 2024-09-16T04:06:42.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d200a10e0 con 0x7f8d200a2410 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient(hunting): picked mon.noname-c con 0x7f252013d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25200a0590 con 0x7f25200a3a90 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25200a1270 con 0x7f252013d3c0 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25200a4fd0 con 0x7f25200a4580 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603365428 0 0) 0x7f8d200a4970 con 0x7f8d2013d060 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient(hunting): my global_id is 4295 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d200a0e00 con 0x7f8d2013d060 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2718191482 0 0) 0x7f8d08003520 con 0x7f8d200a1b10 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 >> v1:172.21.15.148:6790/0 conn(0x7f8d2013d060 legacy=0x7f8d2013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 >> v1:172.21.15.148:6789/0 conn(0x7f8d200a2410 legacy=0x7f8d200a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 monclient: found mon.noname-c 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2006dfd0 con 0x7f8d200a1b10 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d24d42640 5 monclient: authenticate success, global_id 4279 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d24d42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d14001610 con 0x7f8d200a1b10 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: dump: 2024-09-16T04:06:42.756 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d24d42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.745+0000 7f2527435640 10 monclient(hunting): authenticate will time out at 823.370850s 2024-09-16T04:06:42.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249806745 0 0) 0x7f251c001110 con 0x7f25200a3a90 2024-09-16T04:06:42.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d14001930 con 0x7f8d200a1b10 2024-09-16T04:06:42.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d1d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d24d42640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f8d24d42640 4 set_mon_vals no callback set 2024-09-16T04:06:42.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): my global_id is 4296 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f250c003520 con 0x7f25200a3a90 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709849903 0 0) 0x7f25200a4fd0 con 0x7f25200a4580 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient(hunting): my global_id is 4298 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f251c001110 con 0x7f25200a4580 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3728888736 0 0) 0x7f250c003520 con 0x7f25200a3a90 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 >> v1:172.21.15.148:6790/0 conn(0x7f25200a4580 legacy=0x7f252013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 >> v1:172.21.15.171:6789/0 conn(0x7f252013d3c0 legacy=0x7f252013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 monclient: found mon.noname-a 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252006e170 con 0x7f25200a3a90 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f251c0017e0 con 0x7f25200a3a90 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: dump: 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.761 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2527435640 5 monclient: authenticate success, global_id 4296 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2527435640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f251c001a40 con 0x7f25200a3a90 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2517fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2527435640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.746+0000 7f2527435640 4 set_mon_vals no callback set 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 10 monclient: shutdown 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 >> v1:172.21.15.171:6789/0 conn(0x7f8d200a1b10 legacy=0x7f8d2013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d1d59a640 1 -- 172.21.15.148:0/3547048352 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d14001470 con 0x7f8d200a1b10 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d1d59a640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2517fff640 1 -- 172.21.15.148:0/341538863 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f251c0035f0 con 0x7f25200a3a90 2024-09-16T04:06:42.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2517fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2517fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2517fff640 10 monclient: dump: 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.764 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 shutdown_connections 2024-09-16T04:06:42.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 >> 172.21.15.148:0/3547048352 conn(0x7f8d200a97b0 msgr2=0x7f8d200a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 shutdown_connections 2024-09-16T04:06:42.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2527435640 10 monclient: shutdown 2024-09-16T04:06:42.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f8d24d42640 1 -- 172.21.15.148:0/3547048352 wait complete. 2024-09-16T04:06:42.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2527435640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.747+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 >> v1:172.21.15.148:6789/0 conn(0x7f25200a3a90 legacy=0x7f25200a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.748+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 shutdown_connections 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.748+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 >> 172.21.15.148:0/341538863 conn(0x7f25200a9a40 msgr2=0x7f25200a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.748+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 shutdown_connections 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.748+0000 7f2527435640 1 -- 172.21.15.148:0/341538863 wait complete. 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.748+0000 7f2527435640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.748+0000 7f2527435640 10 monclient: monmap: 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.767 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f8d24d42640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f8d24d42640 10 monclient: monmap: 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.768 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f2527435640 1 Processor -- start 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f2527435640 1 -- start start 2024-09-16T04:06:42.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f2527435640 10 monclient: init 2024-09-16T04:06:42.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f8d24d42640 1 Processor -- start 2024-09-16T04:06:42.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f8d24d42640 1 -- start start 2024-09-16T04:06:42.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.749+0000 7f8d24d42640 10 monclient: init 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f2527435640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f2527435640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f2527435640 10 monclient(hunting): picked mon.noname-c con 0x7f25200a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f25251d1640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f25200a3a90 0x7f252009f160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35546/0 (socket says 172.21.15.148:35546) 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f25249d0640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f25200a4580 0x7f25201d4430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45484/0 (socket says 172.21.15.148:45484) 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f2527435640 10 monclient(hunting): picked mon.noname-a con 0x7f25200a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f25251d1640 1 -- 172.21.15.148:0/135693250 learned_addr learned my addr 172.21.15.148:0/135693250 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f8d24d42640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f2527435640 10 monclient(hunting): picked mon.noname-b con 0x7f252013d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f8d24d42640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:42.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252006d860 con 0x7f25200a4580 2024-09-16T04:06:42.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f8d24d42640 10 monclient(hunting): picked mon.noname-c con 0x7f8d200a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f8d1e59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8d200a1b10 0x7f8d201cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35562/0 (socket says 172.21.15.148:35562) 2024-09-16T04:06:42.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.750+0000 7f8d1e59c640 1 -- 172.21.15.148:0/2296134046 learned_addr learned my addr 172.21.15.148:0/2296134046 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252013cff0 con 0x7f252013d3c0 2024-09-16T04:06:42.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 10 monclient(hunting): picked mon.noname-b con 0x7f8d200a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751623171 0 0) 0x7f252006d860 con 0x7f25200a4580 2024-09-16T04:06:42.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 10 monclient(hunting): picked mon.noname-a con 0x7f8d2013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252013c560 con 0x7f25200a3a90 2024-09-16T04:06:42.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2527435640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2527435640 10 monclient(hunting): authenticate will time out at 823.376468s 2024-09-16T04:06:42.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d20137d20 con 0x7f8d2013d060 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): my global_id is 4299 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2504003520 con 0x7f25200a4580 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450924857 0 0) 0x7f252013cff0 con 0x7f252013d3c0 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): my global_id is 4285 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252006d860 con 0x7f252013d3c0 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050305141 0 0) 0x7f252013c560 con 0x7f25200a3a90 2024-09-16T04:06:42.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 10 monclient(hunting): my global_id is 4301 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252013cff0 con 0x7f25200a3a90 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2013ca90 con 0x7f8d200a2410 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d200a0130 con 0x7f8d200a1b10 2024-09-16T04:06:42.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d24d42640 10 monclient(hunting): authenticate will time out at 823.376712s 2024-09-16T04:06:42.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1158612257 0 0) 0x7f2504003520 con 0x7f25200a4580 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2767297407 0 0) 0x7f8d20137d20 con 0x7f8d2013d060 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d0effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d0effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f8d0effd640 10 monclient(hunting): my global_id is 4302 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d00003550 con 0x7f8d2013d060 2024-09-16T04:06:42.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252013c560 con 0x7f25200a4580 2024-09-16T04:06:42.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215870909 0 0) 0x7f8d200a0130 con 0x7f8d200a1b10 2024-09-16T04:06:42.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): my global_id is 4304 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d20137d20 con 0x7f8d200a1b10 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877315941 0 0) 0x7f8d2013ca90 con 0x7f8d200a2410 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): my global_id is 4288 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d200a0130 con 0x7f8d200a2410 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 481236441 0 0) 0x7f8d00003550 con 0x7f8d2013d060 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d2013ca90 con 0x7f8d2013d060 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d18001c30 con 0x7f8d2013d060 2024-09-16T04:06:42.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): dump: 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1661659098 0 0) 0x7f8d200a0130 con 0x7f8d200a2410 2024-09-16T04:06:42.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3913239979 0 0) 0x7f252006d860 con 0x7f252013d3c0 2024-09-16T04:06:42.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2504003520 con 0x7f252013d3c0 2024-09-16T04:06:42.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.751+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f25100017d0 con 0x7f25200a4580 2024-09-16T04:06:42.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2518001b30 con 0x7f252013d3c0 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 560376784 0 0) 0x7f252013cff0 con 0x7f25200a3a90 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252006d860 con 0x7f25200a3a90 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f251c003830 con 0x7f25200a3a90 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.789 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 950659176 0 0) 0x7f252013c560 con 0x7f25200a4580 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 >> v1:172.21.15.148:6790/0 conn(0x7f25200a3a90 legacy=0x7f252009f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 >> v1:172.21.15.171:6789/0 conn(0x7f252013d3c0 legacy=0x7f25201d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 monclient: found mon.a 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252013c840 con 0x7f25200a4580 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2515ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 5 monclient: authenticate success, global_id 4299 2024-09-16T04:06:42.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 10 monclient: _renew_subs 2024-09-16T04:06:42.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f252013cb20 con 0x7f25200a4580 2024-09-16T04:06:42.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 10 monclient: _renew_subs 2024-09-16T04:06:42.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f25201d9280 con 0x7f25200a4580 2024-09-16T04:06:42.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d00003550 con 0x7f8d200a2410 2024-09-16T04:06:42.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d14001c50 con 0x7f8d200a2410 2024-09-16T04:06:42.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.752+0000 7f8d0effd640 10 monclient(hunting): dump: 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1643167936 0 0) 0x7f8d00003550 con 0x7f8d200a2410 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 >> v1:172.21.15.148:6790/0 conn(0x7f8d200a1b10 legacy=0x7f8d201cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 >> v1:172.21.15.148:6789/0 conn(0x7f8d2013d060 legacy=0x7f8d201d4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 monclient: found mon.b 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f2527435640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f2527435640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f2527435640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25200008d0 con 0x7f25200a4580 2024-09-16T04:06:42.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d20137a40 con 0x7f8d200a2410 2024-09-16T04:06:42.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 5 monclient: authenticate success, global_id 4288 2024-09-16T04:06:42.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 10 monclient: _renew_subs 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2009f060 con 0x7f8d200a2410 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 10 monclient: _renew_subs 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d24d42640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d2009daf0 con 0x7f8d200a2410 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d140038a0 con 0x7f8d200a2410 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d14003bf0 con 0x7f8d200a2410 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d0effd640 10 monclient: dump: 2024-09-16T04:06:42.798 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.753+0000 7f8d1cd99640 4 set_mon_vals no callback set 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.754+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8d14003d90 con 0x7f8d200a2410 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f8d14045db0 con 0x7f8d200a2410 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f8d1e59c640 10 monclient: get_auth_request con 0x7f8d00038e80 auth_method 0 2024-09-16T04:06:42.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f8d0cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f8d0cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f8d0cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d201368f0 con 0x7f8d200a2410 2024-09-16T04:06:42.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2510001ce0 con 0x7f25200a4580 2024-09-16T04:06:42.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f2515ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2510004980 con 0x7f25200a4580 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f2515ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f2515ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f2515ffb640 10 monclient: dump: 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.755+0000 7f25177fe640 4 set_mon_vals no callback set 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.756+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2510004ba0 con 0x7f25200a4580 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.756+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f25100469f0 con 0x7f25200a4580 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.756+0000 7f25251d1640 10 monclient: get_auth_request con 0x7f25040391d0 auth_method 0 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.760+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2510015ae0 con 0x7f25200a4580 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.760+0000 7f2515ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.760+0000 7f2515ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.761+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8d14014d50 con 0x7f8d200a2410 2024-09-16T04:06:42.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.761+0000 7f8d0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.761+0000 7f8d0effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.782+0000 7fc772ee6640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.782+0000 7fc772ee6640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.782+0000 7fc772ee6640 10 monclient: monmap: 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.783787+0000 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.783787+0000 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.783+0000 7fc772ee6640 1 Processor -- start 2024-09-16T04:06:42.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.783+0000 7f88ad638640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.783+0000 7f88ad638640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.783+0000 7f88ad638640 10 monclient: monmap: 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.784637+0000 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.784637+0000 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.806 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.783+0000 7fc772ee6640 1 -- start start 2024-09-16T04:06:42.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.783+0000 7fc772ee6640 10 monclient: init 2024-09-16T04:06:42.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7f88ad638640 1 Processor -- start 2024-09-16T04:06:42.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7f88ad638640 1 -- start start 2024-09-16T04:06:42.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7f88ad638640 10 monclient: init 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fc76c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fc76c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fc76c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0a0e00 con 0x7fc76c0a2410 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0a10e0 con 0x7fc76c13d060 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0a4970 con 0x7fc76c0a1b10 2024-09-16T04:06:42.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc772ee6640 10 monclient(hunting): authenticate will time out at 823.409701s 2024-09-16T04:06:42.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc770c82640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc76c0a2410 0x7fc76c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45492/0 (socket says 172.21.15.148:45492) 2024-09-16T04:06:42.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.784+0000 7fc770c82640 1 -- 172.21.15.148:0/3606230520 learned_addr learned my addr 172.21.15.148:0/3606230520 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:42.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176921329 0 0) 0x7fc76c0a0e00 con 0x7fc76c0a2410 2024-09-16T04:06:42.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): my global_id is 4305 2024-09-16T04:06:42.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc75c003520 con 0x7fc76c0a2410 2024-09-16T04:06:42.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient(hunting): picked mon.noname-a con 0x7f88a80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient(hunting): picked mon.noname-c con 0x7f88a80a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a77fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88a813d060 0x7f88a813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35570/0 (socket says 172.21.15.148:35570) 2024-09-16T04:06:42.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a77fe640 1 -- 172.21.15.148:0/2278678259 learned_addr learned my addr 172.21.15.148:0/2278678259 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a67fc640 1 --1- 172.21.15.148:0/2278678259 >> v1:172.21.15.171:6789/0 conn(0x7f88a80a1b10 0x7f88a813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49514/0 (socket says 172.21.15.148:49514) 2024-09-16T04:06:42.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient(hunting): picked mon.noname-b con 0x7f88a813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a80a0e00 con 0x7f88a80a2410 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a80a10e0 con 0x7f88a80a1b10 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463419709 0 0) 0x7fc76c0a4970 con 0x7fc76c0a1b10 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): my global_id is 4307 2024-09-16T04:06:42.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c0a0e00 con 0x7fc76c0a1b10 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2413730061 0 0) 0x7fc76c0a10e0 con 0x7fc76c13d060 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient(hunting): my global_id is 4291 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c0a4970 con 0x7fc76c13d060 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a80a4970 con 0x7f88a813d060 2024-09-16T04:06:42.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88ad638640 10 monclient(hunting): authenticate will time out at 823.410321s 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2174740039 0 0) 0x7fc75c003520 con 0x7fc76c0a2410 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 >> v1:172.21.15.148:6790/0 conn(0x7fc76c0a1b10 legacy=0x7fc76c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 >> v1:172.21.15.171:6789/0 conn(0x7fc76c13d060 legacy=0x7fc76c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:06:42.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c06dfd0 con 0x7fc76c0a2410 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc760001610 con 0x7fc76c0a2410 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc772ee6640 5 monclient: authenticate success, global_id 4305 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc772ee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489979631 0 0) 0x7f88a80a10e0 con 0x7f88a80a1b10 2024-09-16T04:06:42.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7fc76b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 10 monclient: dump: 2024-09-16T04:06:42.817 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.817 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.818 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 10 monclient(hunting): my global_id is 4294 2024-09-16T04:06:42.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8890003520 con 0x7f88a80a1b10 2024-09-16T04:06:42.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2943267862 0 0) 0x7f8890003520 con 0x7f88a80a1b10 2024-09-16T04:06:42.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc772ee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.785+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 >> v1:172.21.15.148:6790/0 conn(0x7f88a813d060 legacy=0x7f88a813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 >> v1:172.21.15.148:6789/0 conn(0x7f88a80a2410 legacy=0x7f88a80a27f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 1 monclient: found mon.noname-c 2024-09-16T04:06:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc760001870 con 0x7fc76c0a2410 2024-09-16T04:06:42.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a806dfd0 con 0x7f88a80a1b10 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f889c001610 con 0x7f88a80a1b10 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 1 -- 172.21.15.148:0/3606230520 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc760001c80 con 0x7fc76c0a2410 2024-09-16T04:06:42.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc76b7fe640 10 monclient: dump: 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.822 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: dump: 2024-09-16T04:06:42.823 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.823 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.823 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88ad638640 5 monclient: authenticate success, global_id 4294 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88ad638640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f889c001820 con 0x7f88a80a1b10 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 1 -- 172.21.15.148:0/2278678259 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f889c001c70 con 0x7f88a80a1b10 2024-09-16T04:06:42.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88a5ffb640 10 monclient: dump: 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88ad638640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7f88ad638640 4 set_mon_vals no callback set 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 10 monclient: shutdown 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 >> v1:172.21.15.171:6789/0 conn(0x7f88a80a1b10 legacy=0x7f88a813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc772ee6640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.786+0000 7fc772ee6640 4 set_mon_vals no callback set 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7fc772ee6640 10 monclient: shutdown 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7fc772ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7fc772ee6640 1 -- 172.21.15.148:0/3606230520 >> v1:172.21.15.148:6789/0 conn(0x7fc76c0a2410 legacy=0x7fc76c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 shutdown_connections 2024-09-16T04:06:42.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 >> 172.21.15.148:0/2278678259 conn(0x7f88a80a97b0 msgr2=0x7f88a80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7fc772ee6640 1 -- 172.21.15.148:0/3606230520 shutdown_connections 2024-09-16T04:06:42.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7fc772ee6640 1 -- 172.21.15.148:0/3606230520 >> 172.21.15.148:0/3606230520 conn(0x7fc76c0a97b0 msgr2=0x7fc76c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 shutdown_connections 2024-09-16T04:06:42.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7f88ad638640 1 -- 172.21.15.148:0/2278678259 wait complete. 2024-09-16T04:06:42.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.787+0000 7fc772ee6640 1 -- 172.21.15.148:0/3606230520 shutdown_connections 2024-09-16T04:06:42.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7fc772ee6640 1 -- 172.21.15.148:0/3606230520 wait complete. 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7f88ad638640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7f88ad638640 10 monclient: monmap: 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.829 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7fc772ee6640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7fc772ee6640 10 monclient: monmap: 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.830 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7fa16b8af640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7fa16b8af640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.788+0000 7fa16b8af640 10 monclient: monmap: 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.789878+0000 2024-09-16T04:06:42.831 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.789878+0000 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7f88ad638640 1 Processor -- start 2024-09-16T04:06:42.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7f88ad638640 1 -- start start 2024-09-16T04:06:42.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7f88ad638640 10 monclient: init 2024-09-16T04:06:42.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7fc772ee6640 1 Processor -- start 2024-09-16T04:06:42.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7fa16b8af640 1 Processor -- start 2024-09-16T04:06:42.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7fc772ee6640 1 -- start start 2024-09-16T04:06:42.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7fc772ee6640 10 monclient: init 2024-09-16T04:06:42.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.789+0000 7f88ad638640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:42.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 10 monclient(hunting): picked mon.noname-b con 0x7f88a80a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fa16b8af640 1 -- start start 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fa16b8af640 10 monclient: init 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 10 monclient(hunting): picked mon.noname-c con 0x7f88a80a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 10 monclient(hunting): picked mon.noname-a con 0x7f88a813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a81230b0 con 0x7f88a813d060 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a80a7b60 con 0x7f88a80a1b10 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a813ce10 con 0x7f88a80a2410 2024-09-16T04:06:42.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f89f4e96640 10 monclient: get_monmap_and_config 2024-09-16T04:06:42.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88ad638640 10 monclient(hunting): authenticate will time out at 823.415267s 2024-09-16T04:06:42.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88a6ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f88a80a1b10 0x7f88a813b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49528/0 (socket says 172.21.15.148:49528) 2024-09-16T04:06:42.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f88a6ffd640 1 -- 172.21.15.148:0/3204126912 learned_addr learned my addr 172.21.15.148:0/3204126912 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f89f4e96640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f89f4e96640 10 monclient: monmap: 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:42.791300+0000 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:42.791300+0000 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:42.837 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235121551 0 0) 0x7f88a80a7b60 con 0x7f88a80a1b10 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): my global_id is 4297 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8894003550 con 0x7f88a80a1b10 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fc772ee6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fc772ee6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fc772ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fc76c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fc772ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fc76c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1052611397 0 0) 0x7f88a813ce10 con 0x7f88a80a2410 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): my global_id is 4313 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88a80a7b60 con 0x7f88a80a2410 2024-09-16T04:06:42.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1349342019 0 0) 0x7f88a81230b0 con 0x7f88a813d060 2024-09-16T04:06:42.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fa16b8af640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 10 monclient(hunting): my global_id is 4308 2024-09-16T04:06:42.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88a813ce10 con 0x7f88a813d060 2024-09-16T04:06:42.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fa16b8af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:42.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.790+0000 7fc76bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc76c0a2410 0x7fc76c09b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45508/0 (socket says 172.21.15.148:45508) 2024-09-16T04:06:42.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc771483640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc76c13d060 0x7fc76c1d7fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35590/0 (socket says 172.21.15.148:35590) 2024-09-16T04:06:42.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 10 monclient(hunting): picked mon.noname-b con 0x7fa16409e830 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc770c82640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc76c0a1b10 0x7fc76c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49544/0 (socket says 172.21.15.148:49544) 2024-09-16T04:06:42.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16964b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa16409e830 0x7fa16409ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35596/0 (socket says 172.21.15.148:35596) 2024-09-16T04:06:42.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16964b640 1 -- 172.21.15.148:0/3693864287 learned_addr learned my addr 172.21.15.148:0/3693864287 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3900289884 0 0) 0x7f8894003550 con 0x7f88a80a1b10 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88a81230b0 con 0x7f88a80a1b10 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88980017f0 con 0x7f88a80a1b10 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.843 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc772ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fc76c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc772ee6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c09b880 con 0x7fc76c0a2410 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc772ee6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c09cd90 con 0x7fc76c0a1b10 2024-09-16T04:06:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc772ee6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c09e240 con 0x7fc76c13d060 2024-09-16T04:06:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc772ee6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc772ee6640 10 monclient(hunting): authenticate will time out at 823.416209s 2024-09-16T04:06:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc76bfff640 1 -- 172.21.15.148:0/1895110952 learned_addr learned my addr 172.21.15.148:0/1895110952 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3042372889 0 0) 0x7f88a813ce10 con 0x7f88a813d060 2024-09-16T04:06:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8894003550 con 0x7f88a813d060 2024-09-16T04:06:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1628295825 0 0) 0x7f88a80a7b60 con 0x7f88a80a2410 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88a813ce10 con 0x7f88a80a2410 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88a0001aa0 con 0x7f88a813d060 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.846 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f889c001b70 con 0x7f88a80a2410 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.848 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 10 monclient(hunting): picked mon.noname-a con 0x7fa1640a01b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3365170133 0 0) 0x7f88a81230b0 con 0x7f88a80a1b10 2024-09-16T04:06:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 10 monclient(hunting): picked mon.noname-c con 0x7fa16409bce0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 >> v1:172.21.15.148:6790/0 conn(0x7f88a80a2410 legacy=0x7f88a81cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16409cb10 con 0x7fa1640a01b0 2024-09-16T04:06:42.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 >> v1:172.21.15.148:6789/0 conn(0x7f88a813d060 legacy=0x7f88a81cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 monclient: found mon.b 2024-09-16T04:06:42.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16409f320 con 0x7fa16409bce0 2024-09-16T04:06:42.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f89f4e96640 1 Processor -- start 2024-09-16T04:06:42.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281325581 0 0) 0x7fc76c09b880 con 0x7fc76c0a2410 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a80a0130 con 0x7f88a80a1b10 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 5 monclient: authenticate success, global_id 4297 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8898001dc0 con 0x7f88a80a1b10 2024-09-16T04:06:42.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16409f4f0 con 0x7fa16409e830 2024-09-16T04:06:42.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 10 monclient: _renew_subs 2024-09-16T04:06:42.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88a8122dd0 con 0x7f88a80a1b10 2024-09-16T04:06:42.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 10 monclient: _renew_subs 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88ad638640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f88a8120530 con 0x7f88a80a1b10 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa16b8af640 10 monclient(hunting): authenticate will time out at 823.416914s 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc7697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc7697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc7697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc7697fa640 10 monclient(hunting): my global_id is 4311 2024-09-16T04:06:42.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc750003520 con 0x7fc76c0a2410 2024-09-16T04:06:42.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591647428 0 0) 0x7fa16409cb10 con 0x7fa1640a01b0 2024-09-16T04:06:42.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa15bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa15bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 290528032 0 0) 0x7fc76c09e240 con 0x7fc76c13d060 2024-09-16T04:06:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa15bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): my global_id is 4316 2024-09-16T04:06:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c09b880 con 0x7fc76c13d060 2024-09-16T04:06:42.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7fa15bfff640 10 monclient(hunting): my global_id is 4314 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa150003520 con 0x7fa1640a01b0 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002860400 0 0) 0x7fa16409f320 con 0x7fa16409bce0 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): my global_id is 4300 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa16409cb10 con 0x7fa16409bce0 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231315124 0 0) 0x7fa16409f4f0 con 0x7fa16409e830 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 880728712 0 0) 0x7fc76c09b880 con 0x7fc76c13d060 2024-09-16T04:06:42.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc76c09e240 con 0x7fc76c13d060 2024-09-16T04:06:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7f89f4e96640 1 -- start start 2024-09-16T04:06:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7f89f4e96640 10 monclient: init 2024-09-16T04:06:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc764001cd0 con 0x7fc76c13d060 2024-09-16T04:06:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:42.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): dump: 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3619811228 0 0) 0x7fc750003520 con 0x7fc76c0a2410 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc76c09b880 con 0x7fc76c0a2410 2024-09-16T04:06:42.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc7580018b0 con 0x7fc76c0a2410 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): dump: 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.860 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f88a57fa640 4 set_mon_vals no callback set 2024-09-16T04:06:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176473087 0 0) 0x7fc76c09cd90 con 0x7fc76c0a1b10 2024-09-16T04:06:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 10 monclient(hunting): my global_id is 4303 2024-09-16T04:06:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc750003520 con 0x7fc76c0a1b10 2024-09-16T04:06:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 10 monclient(hunting): my global_id is 4319 2024-09-16T04:06:42.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa16409f320 con 0x7fa16409e830 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.791+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8898004ac0 con 0x7f88a80a1b10 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7f888f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7f888f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3688983862 0 0) 0x7fa150003520 con 0x7fa1640a01b0 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7f888f7fe640 10 monclient: dump: 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.863 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 >> v1:172.21.15.148:6790/0 conn(0x7fa16409e830 legacy=0x7fa16409ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.792+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1109186938 0 0) 0x7fc76c09e240 con 0x7fc76c13d060 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 >> v1:172.21.15.171:6789/0 conn(0x7fc76c0a1b10 legacy=0x7fc76c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 >> v1:172.21.15.148:6789/0 conn(0x7fc76c0a2410 legacy=0x7fc76c09b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 1 monclient: found mon.c 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c13cb00 con 0x7fc76c13d060 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 >> v1:172.21.15.171:6789/0 conn(0x7fa16409bce0 legacy=0x7fa16413b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 monclient: found mon.noname-a 2024-09-16T04:06:42.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16413dd80 con 0x7fa1640a01b0 2024-09-16T04:06:42.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 5 monclient: authenticate success, global_id 4316 2024-09-16T04:06:42.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 10 monclient: _renew_subs 2024-09-16T04:06:42.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c09df60 con 0x7fc76c13d060 2024-09-16T04:06:42.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f888d7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f888d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f888d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa154001610 con 0x7fa1640a01b0 2024-09-16T04:06:42.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa16b8af640 5 monclient: authenticate success, global_id 4314 2024-09-16T04:06:42.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa16b8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a80008d0 con 0x7f88a80a1b10 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: dump: 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.870 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.871 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.871 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa16b8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc764003970 con 0x7fc76c13d060 2024-09-16T04:06:42.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc764003cc0 con 0x7fc76c13d060 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc7697fa640 10 monclient: dump: 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 10 monclient: _renew_subs 2024-09-16T04:06:42.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa154001870 con 0x7fa1640a01b0 2024-09-16T04:06:42.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:42.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 1 -- 172.21.15.148:0/3693864287 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa154001c00 con 0x7fa1640a01b0 2024-09-16T04:06:42.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa16b8af640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient(hunting): picked mon.noname-c con 0x7f89f0134990 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa16b8af640 4 set_mon_vals no callback set 2024-09-16T04:06:42.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient(hunting): picked mon.noname-a con 0x7f89f009e760 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient(hunting): picked mon.noname-b con 0x7f89f009ebb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f006c980 con 0x7f89f009e760 2024-09-16T04:06:42.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f009d9c0 con 0x7f89f0134990 2024-09-16T04:06:42.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f009dca0 con 0x7f89f009ebb0 2024-09-16T04:06:42.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7f89f4e96640 10 monclient(hunting): authenticate will time out at 823.418936s 2024-09-16T04:06:42.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc76c09cb10 con 0x7fc76c13d060 2024-09-16T04:06:42.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fa15bfff640 10 monclient: dump: 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.793+0000 7fc76affd640 4 set_mon_vals no callback set 2024-09-16T04:06:42.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7f89ee59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f89f0134990 0x7f89f0134d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49570/0 (socket says 172.21.15.148:49570) 2024-09-16T04:06:42.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7f89ee59c640 1 -- 172.21.15.148:0/250658335 learned_addr learned my addr 172.21.15.148:0/250658335 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f889800fd50 con 0x7f88a80a1b10 2024-09-16T04:06:42.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f8898046b10 con 0x7f88a80a1b10 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7fa16b8af640 10 monclient: shutdown 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7fa16b8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 >> v1:172.21.15.148:6789/0 conn(0x7fa1640a01b0 legacy=0x7fa164137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 shutdown_connections 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 >> 172.21.15.148:0/3693864287 conn(0x7fa1640a1dd0 msgr2=0x7fa1640a21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364738202 0 0) 0x7f89f006c980 con 0x7f89f009e760 2024-09-16T04:06:42.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc764003e60 con 0x7fc76c13d060 2024-09-16T04:06:42.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): my global_id is 4317 2024-09-16T04:06:42.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.794+0000 7f88a67fc640 10 monclient: get_auth_request con 0x7f8894039350 auth_method 0 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d8003520 con 0x7f89f009e760 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490939635 0 0) 0x7f89f009dca0 con 0x7f89f009ebb0 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient(hunting): my global_id is 4322 2024-09-16T04:06:42.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d8003880 con 0x7f89f009ebb0 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 shutdown_connections 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7fa16b8af640 1 -- 172.21.15.148:0/3693864287 wait complete. 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1107610740 0 0) 0x7f89d8003880 con 0x7f89f009ebb0 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 >> v1:172.21.15.171:6789/0 conn(0x7f89f0134990 legacy=0x7f89f0134d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 >> v1:172.21.15.148:6789/0 conn(0x7f89f009e760 legacy=0x7f89f01397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 monclient: found mon.noname-b 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f00a3590 con 0x7f89f009ebb0 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fc764046ec0 con 0x7fc76c13d060 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89e80019b0 con 0x7f89f009ebb0 2024-09-16T04:06:42.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: dump: 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89f4e96640 5 monclient: authenticate success, global_id 4322 2024-09-16T04:06:42.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89f4e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7fc770c82640 10 monclient: get_auth_request con 0x7fc7500390f0 auth_method 0 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89e8001c10 con 0x7f89f009ebb0 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 1 -- 172.21.15.148:0/250658335 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89e8002020 con 0x7f89f009ebb0 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89ed59a640 10 monclient: dump: 2024-09-16T04:06:42.885 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89f4e96640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:42.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7f89f4e96640 4 set_mon_vals no callback set 2024-09-16T04:06:42.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7fc772ee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.795+0000 7fc772ee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fc772ee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:42.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fa16b8af640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c0008d0 con 0x7fc76c13d060 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fa16b8af640 10 monclient: monmap: 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.888 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fa16b8af640 1 Processor -- start 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fa16b8af640 1 -- start start 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.796+0000 7fa16b8af640 10 monclient: init 2024-09-16T04:06:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 10 monclient: shutdown 2024-09-16T04:06:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 1 -- 172.21.15.148:0/250658335 >> v1:172.21.15.148:6790/0 conn(0x7f89f009ebb0 legacy=0x7f89f013cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 1 -- 172.21.15.148:0/250658335 shutdown_connections 2024-09-16T04:06:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 1 -- 172.21.15.148:0/250658335 >> 172.21.15.148:0/250658335 conn(0x7f89f00a2800 msgr2=0x7f89f00a2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 1 -- 172.21.15.148:0/250658335 shutdown_connections 2024-09-16T04:06:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7f89f4e96640 1 -- 172.21.15.148:0/250658335 wait complete. 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16b8af640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16b8af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16964b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa16409bce0 0x7fa16413c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45546/0 (socket says 172.21.15.148:45546) 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16964b640 1 -- 172.21.15.148:0/3079758771 learned_addr learned my addr 172.21.15.148:0/3079758771 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16b8af640 10 monclient(hunting): picked mon.noname-a con 0x7fa16409bce0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16b8af640 10 monclient(hunting): picked mon.noname-c con 0x7fa16409e830 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.797+0000 7fa16b8af640 10 monclient(hunting): picked mon.noname-b con 0x7fa1640a01b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1640a47b0 con 0x7fa16409bce0 2024-09-16T04:06:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa164132410 con 0x7fa1640a01b0 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7f89f4e96640 10 monclient: build_initial_monmap 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7f89f4e96640 10 monclient: monmap: 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:42.893 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3188047889 0 0) 0x7fa1600011d0 con 0x7fa16409bce0 2024-09-16T04:06:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1641338c0 con 0x7fa16409e830 2024-09-16T04:06:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa16b8af640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa16b8af640 10 monclient(hunting): authenticate will time out at 823.423593s 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 10 monclient(hunting): my global_id is 4320 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa148003520 con 0x7fa16409bce0 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7f89f4e96640 1 Processor -- start 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7f89f4e96640 1 -- start start 2024-09-16T04:06:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7f89f4e96640 10 monclient: init 2024-09-16T04:06:42.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2629292507 0 0) 0x7fa148003520 con 0x7fa16409bce0 2024-09-16T04:06:42.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1600011d0 con 0x7fa16409bce0 2024-09-16T04:06:42.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa1600014e0 con 0x7fa16409bce0 2024-09-16T04:06:42.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.798+0000 7fa159ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient(hunting): dump: 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f88980159d0 con 0x7f88a80a1b10 2024-09-16T04:06:42.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f888f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f888f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient(hunting): picked mon.noname-b con 0x7f89f009e760 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient(hunting): picked mon.noname-a con 0x7f89f009ebb0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient(hunting): picked mon.noname-c con 0x7f89f0134990 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f009b130 con 0x7f89f009ebb0 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f006e020 con 0x7f89f009e760 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f01cc700 con 0x7f89f0134990 2024-09-16T04:06:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89f4e96640 10 monclient(hunting): authenticate will time out at 823.424283s 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89eed9d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f89f0134990 0x7f89f01d3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35610/0 (socket says 172.21.15.148:35610) 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89eed9d640 1 -- 172.21.15.148:0/3151077772 learned_addr learned my addr 172.21.15.148:0/3151077772 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 619178848 0 0) 0x7f89f006e020 con 0x7f89f009e760 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): my global_id is 4309 2024-09-16T04:06:42.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d0003520 con 0x7f89f009e760 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188232192 0 0) 0x7f89f009b130 con 0x7f89f009ebb0 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 10 monclient(hunting): my global_id is 4323 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89f006e020 con 0x7f89f009ebb0 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 497123947 0 0) 0x7f89d0003520 con 0x7f89f009e760 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89f009b130 con 0x7f89f009e760 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89e00017a0 con 0x7f89f009e760 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): dump: 2024-09-16T04:06:42.903 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3645093589 0 0) 0x7f89f006e020 con 0x7f89f009ebb0 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89d0003520 con 0x7f89f009ebb0 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89e40018b0 con 0x7f89f009ebb0 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient(hunting): dump: 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2241258797 0 0) 0x7f89f009b130 con 0x7f89f009e760 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 >> v1:172.21.15.148:6790/0 conn(0x7f89f0134990 legacy=0x7f89f01d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 >> v1:172.21.15.148:6789/0 conn(0x7f89f009ebb0 legacy=0x7f89f01cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 monclient: found mon.b 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f009b410 con 0x7f89f009e760 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 5 monclient: authenticate success, global_id 4309 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 10 monclient: _renew_subs 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89f00a4430 con 0x7f89f009e760 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 10 monclient: _renew_subs 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89f0130da0 con 0x7f89f009e760 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89e0001cb0 con 0x7f89f009e760 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89e0004980 con 0x7f89f009e760 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89deffd640 10 monclient: dump: 2024-09-16T04:06:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7f89ecd99640 4 set_mon_vals no callback set 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 115228389 0 0) 0x7fa1600011d0 con 0x7fa16409bce0 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 >> v1:172.21.15.148:6790/0 conn(0x7fa16409e830 legacy=0x7fa1641cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 >> v1:172.21.15.171:6789/0 conn(0x7fa1640a01b0 legacy=0x7fa1641cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 1 monclient: found mon.a 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1641335e0 con 0x7fa16409bce0 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa16b8af640 5 monclient: authenticate success, global_id 4320 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa16b8af640 10 monclient: _renew_subs 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa16b8af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.799+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa164132190 con 0x7fa16409bce0 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7fa16b8af640 10 monclient: _renew_subs 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7fa16b8af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.800+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa16413ca40 con 0x7fa16409bce0 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa16b8af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa16b8af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa16b8af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1640008d0 con 0x7fa16409bce0 2024-09-16T04:06:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7f89f4e96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa160004a90 con 0x7fa16409bce0 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa159ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa160004de0 con 0x7fa16409bce0 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa159ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa159ffb640 10 monclient: dump: 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.911 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:42.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.801+0000 7fa15b7fe640 4 set_mon_vals no callback set 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.802+0000 7f89f4e96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.802+0000 7f89f4e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.802+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d4005080 con 0x7f89f009e760 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.802+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f89e0004b20 con 0x7f89f009e760 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.803+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f89e0046ba0 con 0x7f89f009e760 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.803+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa160004f80 con 0x7fa16409bce0 2024-09-16T04:06:42.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.803+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fa160046fa0 con 0x7fa16409bce0 2024-09-16T04:06:42.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.803+0000 7fa168e4a640 10 monclient: get_auth_request con 0x7fa148038f20 auth_method 0 2024-09-16T04:06:42.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.803+0000 7f89edd9b640 10 monclient: get_auth_request con 0x7f89d0038fe0 auth_method 0 2024-09-16T04:06:42.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.805+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc764015d20 con 0x7fc76c13d060 2024-09-16T04:06:42.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.805+0000 7fc7697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.805+0000 7fc7697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.807+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f89e0015a60 con 0x7f89f009e760 2024-09-16T04:06:42.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.807+0000 7f89deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.807+0000 7f89deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.808+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa1640a47b0 con 0x7fa16409bce0 2024-09-16T04:06:42.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.808+0000 7fa159ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:42.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.808+0000 7fa159ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:42.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.976+0000 7f888d7fa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:06:42.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.976+0000 7f888d7fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:06:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.976+0000 7f888d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.976+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f88a806f0c0 con 0x7f88a80a1b10 2024-09-16T04:06:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.977+0000 7f888f7fe640 1 -- 172.21.15.148:0/3204126912 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 555833778) 0x7f8898019620 con 0x7f88a80a1b10 2024-09-16T04:06:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.977+0000 7f888f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:06:42.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.977+0000 7f888f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:42.980 INFO:teuthology.orchestra.run.smithi148.stdout:47244640258 2024-09-16T04:06:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 >> v1:172.21.15.171:6812/7327273 conn(0x7f8894039350 legacy=0x7f889403b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 10 monclient: shutdown 2024-09-16T04:06:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 >> v1:172.21.15.171:6789/0 conn(0x7f88a80a1b10 legacy=0x7f88a813b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 shutdown_connections 2024-09-16T04:06:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 >> 172.21.15.148:0/3204126912 conn(0x7f88a80a97b0 msgr2=0x7f88a80720b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 shutdown_connections 2024-09-16T04:06:42.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.979+0000 7f888d7fa640 1 -- 172.21.15.148:0/3204126912 wait complete. 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa16b8af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa16b8af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa16b8af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa1640628c0 con 0x7fa16409bce0 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa159ffb640 1 -- 172.21.15.148:0/3079758771 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 847495621) 0x7fa160015e00 con 0x7fa16409bce0 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa159ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:06:42.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.991+0000 7fa159ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:42.994 INFO:teuthology.orchestra.run.smithi148.stdout:47244640259 2024-09-16T04:06:42.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.3 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.993+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 >> v1:172.21.15.171:6812/7327273 conn(0x7fa148038f20 legacy=0x7fa14803b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.993+0000 7fa16b8af640 10 monclient: shutdown 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.993+0000 7fa16b8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.993+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 >> v1:172.21.15.148:6789/0 conn(0x7fa16409bce0 legacy=0x7fa16413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.993+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 shutdown_connections 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.993+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 >> 172.21.15.148:0/3079758771 conn(0x7fa1640a1dd0 msgr2=0x7fa16409d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.994+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 shutdown_connections 2024-09-16T04:06:42.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:42.994+0000 7fa16b8af640 1 -- 172.21.15.148:0/3079758771 wait complete. 2024-09-16T04:06:43.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-09-16T04:06:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0cff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:06:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0cff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:06:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8d20062630 con 0x7f8d200a2410 2024-09-16T04:06:43.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0effd640 1 -- 172.21.15.148:0/2296134046 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 3756282479) 0x7f8d1401d200 con 0x7f8d200a2410 2024-09-16T04:06:43.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:06:43.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.063+0000 7f8d0effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:43.066 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 >> v1:172.21.15.171:6812/7327273 conn(0x7f8d00038e80 legacy=0x7f8d0003b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 10 monclient: shutdown 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 >> v1:172.21.15.171:6789/0 conn(0x7f8d200a2410 legacy=0x7f8d201d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 shutdown_connections 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 >> 172.21.15.148:0/2296134046 conn(0x7f8d200a97b0 msgr2=0x7f8d20135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 shutdown_connections 2024-09-16T04:06:43.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.066+0000 7f8d0cff9640 1 -- 172.21.15.148:0/2296134046 wait complete. 2024-09-16T04:06:43.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.073+0000 7f2527435640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:06:43.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.073+0000 7f2527435640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:06:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.073+0000 7f2527435640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.073+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f25200a4bc0 con 0x7f25200a4580 2024-09-16T04:06:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.074+0000 7f2515ffb640 1 -- 172.21.15.148:0/135693250 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 555833778) 0x7f25100153e0 con 0x7f25200a4580 2024-09-16T04:06:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.074+0000 7f2515ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:06:43.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.074+0000 7f2515ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:43.076 INFO:teuthology.orchestra.run.smithi148.stdout:47244640258 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 >> v1:172.21.15.171:6812/7327273 conn(0x7f25040391d0 legacy=0x7f250403b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 10 monclient: shutdown 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 >> v1:172.21.15.148:6789/0 conn(0x7f25200a4580 legacy=0x7f25201d4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 shutdown_connections 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 >> 172.21.15.148:0/135693250 conn(0x7f25200a9a40 msgr2=0x7f25200a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 shutdown_connections 2024-09-16T04:06:43.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.076+0000 7f2527435640 1 -- 172.21.15.148:0/135693250 wait complete. 2024-09-16T04:06:43.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-09-16T04:06:43.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.5 2024-09-16T04:06:43.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89f4e96640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:06:43.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89f4e96640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:06:43.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89f4e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:43.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f89d4005370 con 0x7f89f009e760 2024-09-16T04:06:43.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89deffd640 1 -- 172.21.15.148:0/3151077772 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 2777258794) 0x7f89e0015360 con 0x7f89f009e760 2024-09-16T04:06:43.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:06:43.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.100+0000 7f89deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:43.103 INFO:teuthology.orchestra.run.smithi148.stdout:55834574853 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 >> v1:172.21.15.171:6812/7327273 conn(0x7f89d0038fe0 legacy=0x7f89d003b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 10 monclient: shutdown 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 >> v1:172.21.15.171:6789/0 conn(0x7f89f009e760 legacy=0x7f89f01cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 shutdown_connections 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 >> 172.21.15.148:0/3151077772 conn(0x7f89f00a2800 msgr2=0x7f89f012b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:43.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 shutdown_connections 2024-09-16T04:06:43.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.103+0000 7f89f4e96640 1 -- 172.21.15.148:0/3151077772 wait complete. 2024-09-16T04:06:43.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.106+0000 7fc772ee6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:06:43.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.106+0000 7fc772ee6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:06:43.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.106+0000 7fc772ee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:43.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.106+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc76c09dc80 con 0x7fc76c13d060 2024-09-16T04:06:43.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.107+0000 7fc7697fa640 1 -- 172.21.15.148:0/1895110952 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 847495621) 0x7fc7640199d0 con 0x7fc76c13d060 2024-09-16T04:06:43.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.107+0000 7fc7697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:06:43.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.107+0000 7fc7697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:43.109 INFO:teuthology.orchestra.run.smithi148.stdout:47244640259 2024-09-16T04:06:43.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 >> v1:172.21.15.171:6812/7327273 conn(0x7fc7500390f0 legacy=0x7fc75003b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 10 monclient: shutdown 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 >> v1:172.21.15.148:6790/0 conn(0x7fc76c13d060 legacy=0x7fc76c1d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 shutdown_connections 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 >> 172.21.15.148:0/1895110952 conn(0x7fc76c0a97b0 msgr2=0x7fc76c13fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 shutdown_connections 2024-09-16T04:06:43.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:43.109+0000 7fc772ee6640 1 -- 172.21.15.148:0/1895110952 wait complete. 2024-09-16T04:06:43.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-09-16T04:06:43.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-09-16T04:06:43.996 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T04:06:44.007 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T04:06:44.081 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T04:06:44.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.105+0000 7fd0f2e8b640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.105+0000 7fd0f2e8b640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.105+0000 7fd0f2e8b640 10 monclient: monmap: 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.107065+0000 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.107065+0000 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.108 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.109 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.109 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.109 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.109 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.105+0000 7fd0f2e8b640 1 Processor -- start 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.105+0000 7fd0f2e8b640 1 -- start start 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.105+0000 7fd0f2e8b640 10 monclient: init 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ec0a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ec0a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ec13d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ec0a0590 con 0x7fd0ec0a4580 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ec0a1270 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ec0a4fd0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0f2e8b640 10 monclient(hunting): authenticate will time out at 824.732573s 2024-09-16T04:06:44.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0ebfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd0ec0a4580 0x7fd0ec13bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45562/0 (socket says 172.21.15.148:45562) 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0ebfff640 1 -- 172.21.15.148:0/2843826468 learned_addr learned my addr 172.21.15.148:0/2843826468 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 629100187 0 0) 0x7fd0d8001110 con 0x7fd0ec0a4580 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.106+0000 7fd0eb7fe640 10 monclient(hunting): my global_id is 4326 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0dc003520 con 0x7fd0ec0a4580 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184540939 0 0) 0x7fd0ec0a4fd0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): my global_id is 4325 2024-09-16T04:06:44.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0d8001110 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984355465 0 0) 0x7fd0ec0a1270 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient(hunting): my global_id is 4312 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0ec0a4fd0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 24273637 0 0) 0x7fd0dc003520 con 0x7fd0ec0a4580 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 >> v1:172.21.15.148:6790/0 conn(0x7fd0ec0a3a90 legacy=0x7fd0ec0a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 >> v1:172.21.15.171:6789/0 conn(0x7fd0ec13d3c0 legacy=0x7fd0ec13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 monclient: found mon.noname-a 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ec06e170 con 0x7fd0ec0a4580 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd0d80017e0 con 0x7fd0ec0a4580 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: dump: 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.120 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0f2e8b640 5 monclient: authenticate success, global_id 4326 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0f2e8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd0d8001b00 con 0x7fd0ec0a4580 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 1 -- 172.21.15.148:0/2843826468 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd0d8004a40 con 0x7fd0ec0a4580 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0f2e8b640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0f2e8b640 4 set_mon_vals no callback set 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.107+0000 7fd0eb7fe640 10 monclient: dump: 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.121 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.108+0000 7fd0f2e8b640 10 monclient: shutdown 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.108+0000 7fd0f2e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.108+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/2843826468 >> v1:172.21.15.148:6789/0 conn(0x7fd0ec0a4580 legacy=0x7fd0ec13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.108+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/2843826468 shutdown_connections 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.108+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/2843826468 >> 172.21.15.148:0/2843826468 conn(0x7fd0ec0a9a40 msgr2=0x7fd0ec0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.108+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/2843826468 shutdown_connections 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.109+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/2843826468 wait complete. 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.109+0000 7fd0f2e8b640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.109+0000 7fd0f2e8b640 10 monclient: monmap: 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.109+0000 7fd0f2e8b640 1 Processor -- start 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.109+0000 7fd0f2e8b640 1 -- start start 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.109+0000 7fd0f2e8b640 10 monclient: init 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ec0a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ec0a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ec13d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ec09ef10 con 0x7fd0ec0a4580 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f0c27640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd0ec0a3a90 0x7fd0ec09b7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35634/0 (socket says 172.21.15.148:35634) 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f2e8b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ec09dac0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.110+0000 7fd0f0c27640 1 -- 172.21.15.148:0/3382417255 learned_addr learned my addr 172.21.15.148:0/3382417255 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ec09f1f0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036014290 0 0) 0x7fd0ec09ef10 con 0x7fd0ec0a4580 2024-09-16T04:06:44.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 10 monclient(hunting): authenticate will time out at 824.737110s 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): my global_id is 4329 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0d0003520 con 0x7fd0ec0a4580 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769919213 0 0) 0x7fd0ec09dac0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): my global_id is 4315 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0ec09ef10 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823503586 0 0) 0x7fd0ec09f1f0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): my global_id is 4328 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0ec09dac0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 693460780 0 0) 0x7fd0ec09ef10 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0ec09f1f0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3993806014 0 0) 0x7fd0d0003520 con 0x7fd0ec0a4580 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0ec09ef10 con 0x7fd0ec0a4580 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd0e4001bb0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): dump: 2024-09-16T04:06:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd0d80017a0 con 0x7fd0ec0a4580 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): dump: 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4099754679 0 0) 0x7fd0ec09dac0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0d0003520 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd0e00012d0 con 0x7fd0ec0a3a90 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient(hunting): dump: 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2654016877 0 0) 0x7fd0ec09f1f0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 >> v1:172.21.15.148:6790/0 conn(0x7fd0ec0a3a90 legacy=0x7fd0ec09b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 >> v1:172.21.15.148:6789/0 conn(0x7fd0ec0a4580 legacy=0x7fd0ec09bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 monclient: found mon.b 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ec13d130 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 5 monclient: authenticate success, global_id 4315 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 10 monclient: _renew_subs 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ec0a0280 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 10 monclient: _renew_subs 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.111+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd0ec09c5d0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd0e4003970 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd0e4003cc0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 10 monclient: dump: 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0eaffd640 4 set_mon_vals no callback set 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.112+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd0e400ef50 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.113+0000 7fd0f2e8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.113+0000 7fd0f2e8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.113+0000 7fd0f2e8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.113+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0cc005080 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.113+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fd0e4045dd0 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.113+0000 7fd0f0c27640 10 monclient: get_auth_request con 0x7fd0d0039210 auth_method 0 2024-09-16T04:06:44.133 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T04:06:44.134 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T04:06:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.118+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd0e4014c90 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.118+0000 7fd0e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.118+0000 7fd0e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.158+0000 7fb50f017640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.158+0000 7fb50f017640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.158+0000 7fb50f017640 10 monclient: monmap: 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.160565+0000 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.160565+0000 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.162 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.163 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.159+0000 7fb50f017640 1 Processor -- start 2024-09-16T04:06:44.163 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.159+0000 7fb50f017640 1 -- start start 2024-09-16T04:06:44.163 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.159+0000 7fb50f017640 10 monclient: init 2024-09-16T04:06:44.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.160+0000 7fb50f017640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.160+0000 7fb50f017640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:44.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 10 monclient(hunting): picked mon.noname-b con 0x7fb5080a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 10 monclient(hunting): picked mon.noname-a con 0x7fb5080a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 10 monclient(hunting): picked mon.noname-c con 0x7fb50813d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50cdb3640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fb5080a3a90 0x7fb5080a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35640/0 (socket says 172.21.15.148:35640) 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb507fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb5080a4580 0x7fb50813bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45572/0 (socket says 172.21.15.148:45572) 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50cdb3640 1 -- 172.21.15.148:0/3811391657 learned_addr learned my addr 172.21.15.148:0/3811391657 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5080a0590 con 0x7fb5080a4580 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5080a1270 con 0x7fb50813d3c0 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5080a4fd0 con 0x7fb5080a3a90 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.161+0000 7fb50f017640 10 monclient(hunting): authenticate will time out at 824.787998s 2024-09-16T04:06:44.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825109495 0 0) 0x7fb4f4001110 con 0x7fb5080a4580 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): my global_id is 4332 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f8003520 con 0x7fb5080a4580 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2787758985 0 0) 0x7fb5080a1270 con 0x7fb50813d3c0 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): my global_id is 4318 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f4001110 con 0x7fb50813d3c0 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 701660226 0 0) 0x7fb5080a4fd0 con 0x7fb5080a3a90 2024-09-16T04:06:44.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient(hunting): my global_id is 4331 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5080a1270 con 0x7fb5080a3a90 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3526986520 0 0) 0x7fb4f8003520 con 0x7fb5080a4580 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 >> v1:172.21.15.148:6790/0 conn(0x7fb5080a3a90 legacy=0x7fb5080a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 >> v1:172.21.15.171:6789/0 conn(0x7fb50813d3c0 legacy=0x7fb50813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 monclient: found mon.noname-a 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50806e170 con 0x7fb5080a4580 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4f40017e0 con 0x7fb5080a4580 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: dump: 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.170 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb50f017640 5 monclient: authenticate success, global_id 4332 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb50f017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4f4001a40 con 0x7fb5080a4580 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 1 -- 172.21.15.148:0/3811391657 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4f40047e0 con 0x7fb5080a4580 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb5077fe640 10 monclient: dump: 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.171 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb50f017640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.162+0000 7fb50f017640 4 set_mon_vals no callback set 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.163+0000 7fb50f017640 10 monclient: shutdown 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.163+0000 7fb50f017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.163+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 >> v1:172.21.15.148:6789/0 conn(0x7fb5080a4580 legacy=0x7fb50813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.163+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 shutdown_connections 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.163+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 >> 172.21.15.148:0/3811391657 conn(0x7fb5080a9a40 msgr2=0x7fb5080a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.164+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 shutdown_connections 2024-09-16T04:06:44.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.164+0000 7fb50f017640 1 -- 172.21.15.148:0/3811391657 wait complete. 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.165+0000 7fb50f017640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.165+0000 7fb50f017640 10 monclient: monmap: 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.180 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.165+0000 7fb50f017640 1 Processor -- start 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.166+0000 7fb50f017640 1 -- start start 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.166+0000 7fb50f017640 10 monclient: init 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50f017640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50f017640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50f017640 10 monclient(hunting): picked mon.noname-a con 0x7fb5080a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50cdb3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb5080a3a90 0x7fb50809b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45578/0 (socket says 172.21.15.148:45578) 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50f017640 10 monclient(hunting): picked mon.noname-b con 0x7fb5080a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50cdb3640 1 -- 172.21.15.148:0/2571350380 learned_addr learned my addr 172.21.15.148:0/2571350380 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.167+0000 7fb50f017640 10 monclient(hunting): picked mon.noname-c con 0x7fb50813d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50809d7b0 con 0x7fb5080a3a90 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50809f1c0 con 0x7fb5080a4580 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5081409f0 con 0x7fb50813d3c0 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb50f017640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3825424144 0 0) 0x7fb50809d7b0 con 0x7fb5080a3a90 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb50f017640 10 monclient(hunting): authenticate will time out at 824.794596s 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): my global_id is 4335 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4ec003550 con 0x7fb5080a3a90 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2217428066 0 0) 0x7fb50809f1c0 con 0x7fb5080a4580 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): my global_id is 4321 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb50809d7b0 con 0x7fb5080a4580 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506879661 0 0) 0x7fb5081409f0 con 0x7fb50813d3c0 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 10 monclient(hunting): my global_id is 4334 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.168+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb50809f1c0 con 0x7fb50813d3c0 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 158466396 0 0) 0x7fb4ec003550 con 0x7fb5080a3a90 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5081409f0 con 0x7fb5080a3a90 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4fc001990 con 0x7fb5080a3a90 2024-09-16T04:06:44.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): dump: 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.184 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4054854372 0 0) 0x7fb50809d7b0 con 0x7fb5080a4580 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4ec003550 con 0x7fb5080a4580 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4f4001bf0 con 0x7fb5080a4580 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient(hunting): dump: 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.185 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3694092190 0 0) 0x7fb5081409f0 con 0x7fb5080a3a90 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 >> v1:172.21.15.148:6790/0 conn(0x7fb50813d3c0 legacy=0x7fb5081d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 >> v1:172.21.15.171:6789/0 conn(0x7fb5080a4580 legacy=0x7fb50809bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 monclient: found mon.a 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50813d110 con 0x7fb5080a3a90 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.186 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4fc001c90 con 0x7fb5080a3a90 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4fc004c30 con 0x7fb5080a3a90 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb50f017640 5 monclient: authenticate success, global_id 4335 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb50f017640 10 monclient: _renew_subs 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb50f017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb506ffd640 4 set_mon_vals no callback set 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.169+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5080a0280 con 0x7fb5080a3a90 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.170+0000 7fb50f017640 10 monclient: _renew_subs 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.170+0000 7fb50f017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.170+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb50809eee0 con 0x7fb5080a3a90 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.170+0000 7fb5057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.170+0000 7fb5057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.170+0000 7fb5057fa640 10 monclient: dump: 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.171+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb4fc010060 con 0x7fb5080a3a90 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.171+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fb4fc046e20 con 0x7fb5080a3a90 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.171+0000 7fb507fff640 10 monclient: get_auth_request con 0x7fb4ec039080 auth_method 0 2024-09-16T04:06:44.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.172+0000 7fb50f017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.172+0000 7fb50f017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.172+0000 7fb50f017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.172+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5080008d0 con 0x7fb5080a3a90 2024-09-16T04:06:44.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.181+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb4fc015c70 con 0x7fb5080a3a90 2024-09-16T04:06:44.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.181+0000 7fb5057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.181+0000 7fb5057fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.205+0000 7f8c51ccb640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.205+0000 7f8c51ccb640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.205+0000 7f8c51ccb640 10 monclient: monmap: 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.207145+0000 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.207145+0000 2024-09-16T04:06:44.208 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.209 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.209 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.205+0000 7f8c51ccb640 1 Processor -- start 2024-09-16T04:06:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 1 -- start start 2024-09-16T04:06:44.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient: init 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient(hunting): picked mon.noname-c con 0x7f8c4c0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient(hunting): picked mon.noname-a con 0x7f8c4c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c4affd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8c4c0a1b10 0x7f8c4c13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45582/0 (socket says 172.21.15.148:45582) 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c0a0e00 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c4affd640 1 -- 172.21.15.148:0/2734920827 learned_addr learned my addr 172.21.15.148:0/2734920827 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c0a10e0 con 0x7f8c4c0a2410 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c0a4970 con 0x7f8c4c13d060 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.206+0000 7f8c51ccb640 10 monclient(hunting): authenticate will time out at 824.832891s 2024-09-16T04:06:44.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3003060650 0 0) 0x7f8c4c0a0e00 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): my global_id is 4338 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c34003520 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108900360 0 0) 0x7f8c4c0a10e0 con 0x7f8c4c0a2410 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): my global_id is 4324 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c4c0a0e00 con 0x7f8c4c0a2410 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254781952 0 0) 0x7f8c4c0a4970 con 0x7f8c4c13d060 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient(hunting): my global_id is 4337 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c4c0a10e0 con 0x7f8c4c13d060 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2445269292 0 0) 0x7f8c34003520 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 >> v1:172.21.15.148:6790/0 conn(0x7f8c4c13d060 legacy=0x7f8c4c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 >> v1:172.21.15.171:6789/0 conn(0x7f8c4c0a2410 legacy=0x7f8c4c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 monclient: found mon.noname-a 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c06dfd0 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8c40001610 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: dump: 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.214 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c51ccb640 5 monclient: authenticate success, global_id 4338 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c51ccb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8c40001870 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c51ccb640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c51ccb640 4 set_mon_vals no callback set 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 1 -- 172.21.15.148:0/2734920827 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8c40001c40 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.207+0000 7f8c4a7fc640 10 monclient: dump: 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.215 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.216 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 10 monclient: shutdown 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 >> v1:172.21.15.148:6789/0 conn(0x7f8c4c0a1b10 legacy=0x7f8c4c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 shutdown_connections 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 >> 172.21.15.148:0/2734920827 conn(0x7f8c4c0a97b0 msgr2=0x7f8c4c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 shutdown_connections 2024-09-16T04:06:44.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.208+0000 7f8c51ccb640 1 -- 172.21.15.148:0/2734920827 wait complete. 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.209+0000 7f8c51ccb640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.209+0000 7f8c51ccb640 10 monclient: monmap: 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.209+0000 7f8c51ccb640 1 Processor -- start 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.209+0000 7f8c51ccb640 1 -- start start 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.209+0000 7f8c51ccb640 10 monclient: init 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient(hunting): picked mon.noname-c con 0x7f8c4c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient(hunting): picked mon.noname-a con 0x7f8c4c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c1399b0 con 0x7f8c4c13d060 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c137850 con 0x7f8c4c0a1b10 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c4affd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8c4c0a2410 0x7f8c4c137140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35670/0 (socket says 172.21.15.148:35670) 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c4affd640 1 -- 172.21.15.148:0/803168501 learned_addr learned my addr 172.21.15.148:0/803168501 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c0a0130 con 0x7f8c4c0a2410 2024-09-16T04:06:44.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c51ccb640 10 monclient(hunting): authenticate will time out at 824.836401s 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998627807 0 0) 0x7f8c4c1399b0 con 0x7f8c4c13d060 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): my global_id is 4341 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c38003520 con 0x7f8c4c13d060 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046414031 0 0) 0x7f8c4c0a0130 con 0x7f8c4c0a2410 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): my global_id is 4340 2024-09-16T04:06:44.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c4c1399b0 con 0x7f8c4c0a2410 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2878361676 0 0) 0x7f8c38003520 con 0x7f8c4c13d060 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c4c0a0130 con 0x7f8c4c13d060 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8c44001c30 con 0x7f8c4c13d060 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): dump: 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.227 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.210+0000 7f8c33fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1849416629 0 0) 0x7f8c4c1399b0 con 0x7f8c4c0a2410 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8c38003520 con 0x7f8c4c0a2410 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8c40001510 con 0x7f8c4c0a2410 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient(hunting): dump: 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1186530312 0 0) 0x7f8c4c0a0130 con 0x7f8c4c13d060 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 >> v1:172.21.15.148:6790/0 conn(0x7f8c4c0a2410 legacy=0x7f8c4c137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 >> v1:172.21.15.171:6789/0 conn(0x7f8c4c0a1b10 legacy=0x7f8c4c136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 monclient: found mon.a 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c13cc20 con 0x7f8c4c13d060 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 5 monclient: authenticate success, global_id 4341 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 10 monclient: _renew_subs 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c09f6d0 con 0x7f8c4c13d060 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 10 monclient: _renew_subs 2024-09-16T04:06:44.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.211+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8c4c138560 con 0x7f8c4c13d060 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8c44003970 con 0x7f8c4c13d060 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8c44003cc0 con 0x7f8c4c13d060 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 10 monclient: dump: 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.231 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c49ffb640 4 set_mon_vals no callback set 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.212+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8c4400f0f0 con 0x7f8c4c13d060 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.213+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f8c44045e10 con 0x7f8c4c13d060 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.213+0000 7f8c51ccb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.213+0000 7f8c51ccb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.213+0000 7f8c51ccb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.213+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2c005080 con 0x7f8c4c13d060 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.213+0000 7f8c4b7fe640 10 monclient: get_auth_request con 0x7f8c38039000 auth_method 0 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.218+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8c44014d30 con 0x7f8c4c13d060 2024-09-16T04:06:44.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.218+0000 7f8c33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.218+0000 7f8c33fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.245+0000 7ff2dfb45640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.245+0000 7ff2dfb45640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.245+0000 7ff2dfb45640 10 monclient: monmap: 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.247036+0000 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.247036+0000 2024-09-16T04:06:44.248 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.249 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.249 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.249 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.249 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.249 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.245+0000 7ff2dfb45640 1 Processor -- start 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 1 -- start start 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 10 monclient: init 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 10 monclient(hunting): picked mon.noname-a con 0x7ff2d80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 10 monclient(hunting): picked mon.noname-b con 0x7ff2d80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 10 monclient(hunting): picked mon.noname-c con 0x7ff2d813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dd8e1640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff2d80a2410 0x7ff2d80a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45594/0 (socket says 172.21.15.148:45594) 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dd8e1640 1 -- 172.21.15.148:0/956061806 learned_addr learned my addr 172.21.15.148:0/956061806 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dd0e0640 1 --1- 172.21.15.148:0/956061806 >> v1:172.21.15.148:6790/0 conn(0x7ff2d80a1b10 0x7ff2d813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35682/0 (socket says 172.21.15.148:35682) 2024-09-16T04:06:44.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.246+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d80a0e00 con 0x7ff2d80a2410 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d80a10e0 con 0x7ff2d813d060 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d80a4970 con 0x7ff2d80a1b10 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dfb45640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3631055447 0 0) 0x7ff2d80a0e00 con 0x7ff2d80a2410 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dfb45640 10 monclient(hunting): authenticate will time out at 824.873282s 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): my global_id is 4344 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c4003520 con 0x7ff2d80a2410 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393066189 0 0) 0x7ff2d80a4970 con 0x7ff2d80a1b10 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient(hunting): my global_id is 4343 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d80a0e00 con 0x7ff2d80a1b10 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2954603576 0 0) 0x7ff2c4003520 con 0x7ff2d80a2410 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 >> v1:172.21.15.148:6790/0 conn(0x7ff2d80a1b10 legacy=0x7ff2d813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 >> v1:172.21.15.171:6789/0 conn(0x7ff2d813d060 legacy=0x7ff2d813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 monclient: found mon.noname-a 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d806dfd0 con 0x7ff2d80a2410 2024-09-16T04:06:44.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d4001610 con 0x7ff2d80a2410 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dc8df640 10 monclient: dump: 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.257 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dfb45640 5 monclient: authenticate success, global_id 4344 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.247+0000 7ff2dfb45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2d4001870 con 0x7ff2d80a2410 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dc8df640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 4 set_mon_vals no callback set 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dc8df640 1 -- 172.21.15.148:0/956061806 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d4001c40 con 0x7ff2d80a2410 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dc8df640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dc8df640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dc8df640 10 monclient: dump: 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.258 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 10 monclient: shutdown 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 >> v1:172.21.15.148:6789/0 conn(0x7ff2d80a2410 legacy=0x7ff2d80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 shutdown_connections 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 >> 172.21.15.148:0/956061806 conn(0x7ff2d80a97b0 msgr2=0x7ff2d80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 shutdown_connections 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7f420c76c640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7f420c76c640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.248+0000 7f420c76c640 10 monclient: monmap: 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.250894+0000 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.250894+0000 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.249+0000 7ff2dfb45640 1 -- 172.21.15.148:0/956061806 wait complete. 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.249+0000 7ff2dfb45640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.249+0000 7ff2dfb45640 10 monclient: monmap: 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.262 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.249+0000 7f420c76c640 1 Processor -- start 2024-09-16T04:06:44.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.249+0000 7f420c76c640 1 -- start start 2024-09-16T04:06:44.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.249+0000 7f420c76c640 10 monclient: init 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 1 Processor -- start 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 1 -- start start 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient: init 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient(hunting): picked mon.noname-c con 0x7ff2d80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient(hunting): picked mon.noname-a con 0x7ff2d80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient(hunting): picked mon.noname-b con 0x7ff2d813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient(hunting): picked mon.noname-c con 0x7f42040a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient(hunting): picked mon.noname-a con 0x7f42040a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient(hunting): picked mon.noname-b con 0x7f420413d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42040a0590 con 0x7f42040a4580 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42040a1270 con 0x7f42040a3a90 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42040a4fd0 con 0x7f420413d3c0 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7f420c76c640 10 monclient(hunting): authenticate will time out at 824.876926s 2024-09-16T04:06:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d81408b0 con 0x7ff2d80a2410 2024-09-16T04:06:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d81377c0 con 0x7ff2d813d060 2024-09-16T04:06:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d813c9f0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dfb45640 10 monclient(hunting): authenticate will time out at 824.876758s 2024-09-16T04:06:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dd8e1640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff2d80a1b10 0x7ff2d809c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35688/0 (socket says 172.21.15.148:35688) 2024-09-16T04:06:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.250+0000 7ff2dd8e1640 1 -- 172.21.15.148:0/883230349 learned_addr learned my addr 172.21.15.148:0/883230349 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f420ad09640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f420413d3c0 0x7f420413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35692/0 (socket says 172.21.15.148:35692) 2024-09-16T04:06:44.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f420ad09640 1 -- 172.21.15.148:0/1112968995 learned_addr learned my addr 172.21.15.148:0/1112968995 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905850869 0 0) 0x7ff2d81377c0 con 0x7ff2d813d060 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): my global_id is 4330 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2bc003520 con 0x7ff2d813d060 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997031540 0 0) 0x7ff2d813c9f0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): my global_id is 4349 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d81377c0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919421000 0 0) 0x7f42040a4fd0 con 0x7f420413d3c0 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757805941 0 0) 0x7ff2d80a0e00 con 0x7ff2d80a2410 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 10 monclient(hunting): my global_id is 4350 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2d813c9f0 con 0x7ff2d80a2410 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 10 monclient(hunting): my global_id is 4346 2024-09-16T04:06:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3114005509 0 0) 0x7ff2bc003520 con 0x7ff2d813d060 2024-09-16T04:06:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2d80a0e00 con 0x7ff2d813d060 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4189774337 0 0) 0x7ff2d81377c0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2bc003520 con 0x7ff2d80a1b10 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d0001c30 con 0x7ff2d813d060 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d4001a20 con 0x7ff2d80a1b10 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41f0003520 con 0x7f420413d3c0 2024-09-16T04:06:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f420a508640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42040a3a90 0x7f42040a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:49666/0 (socket says 172.21.15.148:49666) 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 533525384 0 0) 0x7ff2d813c9f0 con 0x7ff2d80a2410 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2d81377c0 con 0x7ff2d80a2410 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2c80018d0 con 0x7ff2d80a2410 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient(hunting): dump: 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2760736353 0 0) 0x7ff2bc003520 con 0x7ff2d80a1b10 2024-09-16T04:06:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.251+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 767690955 0 0) 0x7f41f0003520 con 0x7f420413d3c0 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 >> v1:172.21.15.171:6789/0 conn(0x7ff2d813d060 legacy=0x7ff2d809d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 >> v1:172.21.15.148:6789/0 conn(0x7ff2d80a2410 legacy=0x7ff2d809cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 monclient: found mon.c 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d81374e0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 >> v1:172.21.15.171:6789/0 conn(0x7f42040a3a90 legacy=0x7f42040a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2d4004c40 con 0x7ff2d80a1b10 2024-09-16T04:06:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 >> v1:172.21.15.148:6789/0 conn(0x7f42040a4580 legacy=0x7f420413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2dfb45640 5 monclient: authenticate success, global_id 4349 2024-09-16T04:06:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2dfb45640 10 monclient: _renew_subs 2024-09-16T04:06:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2dfb45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d81371d0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7ff2cffff640 4 set_mon_vals no callback set 2024-09-16T04:06:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 monclient: found mon.noname-b 2024-09-16T04:06:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420406e170 con 0x7f420413d3c0 2024-09-16T04:06:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41fc001990 con 0x7f420413d3c0 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: dump: 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41fc001c60 con 0x7f420413d3c0 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 1 -- 172.21.15.148:0/1112968995 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41fc002030 con 0x7f420413d3c0 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.252+0000 7f4209506640 10 monclient: dump: 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d4004df0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2ce7fc640 10 monclient: dump: 2024-09-16T04:06:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.285 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7f420c76c640 5 monclient: authenticate success, global_id 4346 2024-09-16T04:06:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7f420c76c640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7f420c76c640 4 set_mon_vals no callback set 2024-09-16T04:06:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2dfb45640 10 monclient: _renew_subs 2024-09-16T04:06:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2dfb45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.253+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2d81d4d30 con 0x7ff2d80a1b10 2024-09-16T04:06:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff2d4010270 con 0x7ff2d80a1b10 2024-09-16T04:06:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7ff2d40470b0 con 0x7ff2d80a1b10 2024-09-16T04:06:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 10 monclient: shutdown 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 1 -- 172.21.15.148:0/1112968995 >> v1:172.21.15.148:6790/0 conn(0x7f420413d3c0 legacy=0x7f420413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 1 -- 172.21.15.148:0/1112968995 shutdown_connections 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 1 -- 172.21.15.148:0/1112968995 >> 172.21.15.148:0/1112968995 conn(0x7f42040a9a40 msgr2=0x7f42040a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 1 -- 172.21.15.148:0/1112968995 shutdown_connections 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7ff2dd0e0640 10 monclient: get_auth_request con 0x7ff2bc039320 auth_method 0 2024-09-16T04:06:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.254+0000 7f420c76c640 1 -- 172.21.15.148:0/1112968995 wait complete. 2024-09-16T04:06:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.255+0000 7ff2dfb45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.255+0000 7ff2dfb45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.255+0000 7ff2dfb45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.255+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d8137a00 con 0x7ff2d80a1b10 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.255+0000 7f420c76c640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.255+0000 7f420c76c640 10 monclient: monmap: 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.294 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.256+0000 7f420c76c640 1 Processor -- start 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.256+0000 7f420c76c640 1 -- start start 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.256+0000 7f420c76c640 10 monclient: init 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.256+0000 7f420c76c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.256+0000 7f420c76c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.256+0000 7f420c76c640 10 monclient(hunting): picked mon.noname-a con 0x7f42040a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 10 monclient(hunting): picked mon.noname-c con 0x7f42040a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 10 monclient(hunting): picked mon.noname-b con 0x7f420413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420409ecd0 con 0x7f42040a3a90 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42041c8d10 con 0x7f420413d3c0 2024-09-16T04:06:44.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420409da40 con 0x7f42040a4580 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420c76c640 10 monclient(hunting): authenticate will time out at 824.883072s 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420a508640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f42040a3a90 0x7f42041c7ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45624/0 (socket says 172.21.15.148:45624) 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f420a508640 1 -- 172.21.15.148:0/3229824219 learned_addr learned my addr 172.21.15.148:0/3229824219 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1226453941 0 0) 0x7f42041c8d10 con 0x7f420413d3c0 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): my global_id is 4336 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41e8003520 con 0x7f420413d3c0 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124230856 0 0) 0x7f420409ecd0 con 0x7f42040a3a90 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 10 monclient(hunting): my global_id is 4353 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42041c8d10 con 0x7f42040a3a90 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 280677646 0 0) 0x7f42041c8d10 con 0x7f42040a3a90 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.257+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f420409ecd0 con 0x7f42040a3a90 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1855150762 0 0) 0x7f41e8003520 con 0x7f420413d3c0 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42041c8d10 con 0x7f420413d3c0 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41fc001890 con 0x7f420413d3c0 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): dump: 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.297 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41f4001860 con 0x7f42040a3a90 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient(hunting): dump: 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.298 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1079458392 0 0) 0x7f420409ecd0 con 0x7f42040a3a90 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 >> v1:172.21.15.148:6790/0 conn(0x7f42040a4580 legacy=0x7f42041c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 >> v1:172.21.15.171:6789/0 conn(0x7f420413d3c0 legacy=0x7f42041d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 monclient: found mon.a 2024-09-16T04:06:44.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420409e9f0 con 0x7f42040a3a90 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f420c76c640 5 monclient: authenticate success, global_id 4353 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f420c76c640 10 monclient: _renew_subs 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41f4001c50 con 0x7f42040a3a90 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f420c76c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f420409efb0 con 0x7f42040a3a90 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f4208d05640 4 set_mon_vals no callback set 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41f4004a90 con 0x7f42040a3a90 2024-09-16T04:06:44.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.258+0000 7f41faffd640 10 monclient: dump: 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.301 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.302 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.302 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.259+0000 7f420c76c640 10 monclient: _renew_subs 2024-09-16T04:06:44.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.259+0000 7f420c76c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.259+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42041d50e0 con 0x7f42040a3a90 2024-09-16T04:06:44.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff2d4016020 con 0x7ff2d80a1b10 2024-09-16T04:06:44.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7ff2ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7ff2ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7f420c76c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7f420c76c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7f420c76c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420409d5e0 con 0x7f42040a3a90 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.260+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f41f4004d00 con 0x7f42040a3a90 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.262+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f41f4046e70 con 0x7f42040a3a90 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.262+0000 7f4209d07640 10 monclient: get_auth_request con 0x7f41e8038f40 auth_method 0 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.270+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f41f4019950 con 0x7f42040a3a90 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.270+0000 7f41faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.270+0000 7f41faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.274+0000 7f01d4e8d640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.275+0000 7f01d4e8d640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.275+0000 7f01d4e8d640 10 monclient: monmap: 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.276944+0000 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.276944+0000 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.317 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.275+0000 7f01d4e8d640 1 Processor -- start 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.275+0000 7f01d4e8d640 1 -- start start 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.275+0000 7f01d4e8d640 10 monclient: init 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f01d00a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f01d00a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f01d013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d00a0e00 con 0x7f01d013d060 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d00a10e0 con 0x7f01d00a1b10 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d00a4970 con 0x7f01d00a2410 2024-09-16T04:06:44.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01d4e8d640 10 monclient(hunting): authenticate will time out at 824.902637s 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01ce59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f01d00a2410 0x7f01d00a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35722/0 (socket says 172.21.15.148:35722) 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.276+0000 7f01ce59c640 1 -- 172.21.15.148:0/1783484933 learned_addr learned my addr 172.21.15.148:0/1783484933 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1804537931 0 0) 0x7f01d00a0e00 con 0x7f01d013d060 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): my global_id is 4356 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b8003520 con 0x7f01d013d060 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3623095594 0 0) 0x7f01d00a4970 con 0x7f01d00a2410 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient(hunting): my global_id is 4352 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d00a0e00 con 0x7f01d00a2410 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2781236701 0 0) 0x7f01b8003520 con 0x7f01d013d060 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 >> v1:172.21.15.148:6790/0 conn(0x7f01d00a2410 legacy=0x7f01d00a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 >> v1:172.21.15.171:6789/0 conn(0x7f01d00a1b10 legacy=0x7f01d013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 monclient: found mon.noname-a 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d006dfd0 con 0x7f01d013d060 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01c8001990 con 0x7f01d013d060 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: dump: 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.321 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01d4e8d640 5 monclient: authenticate success, global_id 4356 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01d4e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01c8001cb0 con 0x7f01d013d060 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01d4e8d640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01d4e8d640 4 set_mon_vals no callback set 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 1 -- 172.21.15.148:0/1783484933 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01c8002080 con 0x7f01d013d060 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.277+0000 7f01cd59a640 10 monclient: dump: 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.322 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 10 monclient: shutdown 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 1 -- 172.21.15.148:0/1783484933 >> v1:172.21.15.148:6789/0 conn(0x7f01d013d060 legacy=0x7f01d013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 1 -- 172.21.15.148:0/1783484933 shutdown_connections 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 1 -- 172.21.15.148:0/1783484933 >> 172.21.15.148:0/1783484933 conn(0x7f01d00a97b0 msgr2=0x7f01d00a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 1 -- 172.21.15.148:0/1783484933 shutdown_connections 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.278+0000 7f01d4e8d640 1 -- 172.21.15.148:0/1783484933 wait complete. 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.279+0000 7f01d4e8d640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.279+0000 7f01d4e8d640 10 monclient: monmap: 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.324 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 1 Processor -- start 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 1 -- start start 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient: init 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f01d00a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f01d00a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f01d013d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d009c770 con 0x7f01d00a2410 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d009da90 con 0x7f01d00a1b10 2024-09-16T04:06:44.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d009e150 con 0x7f01d013d060 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01d4e8d640 10 monclient(hunting): authenticate will time out at 824.906762s 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01cdd9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f01d00a2410 0x7f01d009c060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45648/0 (socket says 172.21.15.148:45648) 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.280+0000 7f01cdd9b640 1 -- 172.21.15.148:0/4071243140 learned_addr learned my addr 172.21.15.148:0/4071243140 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4271355552 0 0) 0x7f01d009c770 con 0x7f01d00a2410 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): my global_id is 4359 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b0003520 con 0x7f01d00a2410 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3295261880 0 0) 0x7f01d009da90 con 0x7f01d00a1b10 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): my global_id is 4339 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d009c770 con 0x7f01d00a1b10 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935295389 0 0) 0x7f01d009e150 con 0x7f01d013d060 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 10 monclient(hunting): my global_id is 4355 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d009da90 con 0x7f01d013d060 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3262269977 0 0) 0x7f01d009c770 con 0x7f01d00a1b10 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.281+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01d009e150 con 0x7f01d00a1b10 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01c00018d0 con 0x7f01d00a1b10 2024-09-16T04:06:44.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): dump: 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.328 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 38355783 0 0) 0x7f01d009da90 con 0x7f01d013d060 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01d009c770 con 0x7f01d013d060 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01c80017b0 con 0x7f01d013d060 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): dump: 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.329 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4227909255 0 0) 0x7f01b0003520 con 0x7f01d00a2410 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01d009da90 con 0x7f01d00a2410 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01c4003f80 con 0x7f01d00a2410 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient(hunting): dump: 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.330 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3550170427 0 0) 0x7f01d009c770 con 0x7f01d013d060 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 >> v1:172.21.15.171:6789/0 conn(0x7f01d00a1b10 legacy=0x7f01d009b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 >> v1:172.21.15.148:6789/0 conn(0x7f01d00a2410 legacy=0x7f01d009c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 monclient: found mon.c 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d009f320 con 0x7f01d013d060 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 5 monclient: authenticate success, global_id 4355 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 10 monclient: _renew_subs 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01d009ded0 con 0x7f01d013d060 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 10 monclient: _renew_subs 2024-09-16T04:06:44.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f01d013c4a0 con 0x7f01d013d060 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01c8001cb0 con 0x7f01d013d060 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.282+0000 7f01ccd99640 4 set_mon_vals no callback set 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01c8003db0 con 0x7f01d013d060 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01beffd640 10 monclient: dump: 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.333 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f01c800f1e0 con 0x7f01d013d060 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01d4e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01d4e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01d4e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b4005080 con 0x7f01d013d060 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f01c8045cd0 con 0x7f01d013d060 2024-09-16T04:06:44.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.284+0000 7f01ce59c640 10 monclient: get_auth_request con 0x7f01b0039140 auth_method 0 2024-09-16T04:06:44.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.289+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f01c8014cb0 con 0x7f01d013d060 2024-09-16T04:06:44.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.289+0000 7f01beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.289+0000 7f01beffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.446+0000 7ff2dfb45640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:06:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.446+0000 7ff2dfb45640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:06:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.446+0000 7ff2dfb45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.446+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff2d8062630 con 0x7ff2d80a1b10 2024-09-16T04:06:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.447+0000 7ff2ce7fc640 1 -- 172.21.15.148:0/883230349 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 1444633209) 0x7ff2d4015920 con 0x7ff2d80a1b10 2024-09-16T04:06:44.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.447+0000 7ff2ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T04:06:44.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.447+0000 7ff2ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:44.451 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 >> v1:172.21.15.171:6812/7327273 conn(0x7ff2bc039320 legacy=0x7ff2bc03b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 10 monclient: shutdown 2024-09-16T04:06:44.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 >> v1:172.21.15.148:6790/0 conn(0x7ff2d80a1b10 legacy=0x7ff2d809c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 shutdown_connections 2024-09-16T04:06:44.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 >> 172.21.15.148:0/883230349 conn(0x7ff2d80a97b0 msgr2=0x7ff2d80a82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 shutdown_connections 2024-09-16T04:06:44.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.449+0000 7ff2dfb45640 1 -- 172.21.15.148:0/883230349 wait complete. 2024-09-16T04:06:44.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-16T04:06:44.463 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01d4e8d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01d4e8d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01d4e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f01b4005370 con 0x7f01d013d060 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01beffd640 1 -- 172.21.15.148:0/4071243140 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 3756282479) 0x7f01c801d160 con 0x7f01d013d060 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.469+0000 7f01beffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:44.472 INFO:teuthology.orchestra.run.smithi148.stdout:55834574854 2024-09-16T04:06:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 >> v1:172.21.15.171:6812/7327273 conn(0x7f01b0039140 legacy=0x7f01b003b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 10 monclient: shutdown 2024-09-16T04:06:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 >> v1:172.21.15.148:6790/0 conn(0x7f01d013d060 legacy=0x7f01d01cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 shutdown_connections 2024-09-16T04:06:44.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 >> 172.21.15.148:0/4071243140 conn(0x7f01d00a97b0 msgr2=0x7f01d013fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 shutdown_connections 2024-09-16T04:06:44.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.471+0000 7f01d4e8d640 1 -- 172.21.15.148:0/4071243140 wait complete. 2024-09-16T04:06:44.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0f2e8b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:06:44.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0f2e8b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:06:44.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0f2e8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:06:44.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd0cc005370 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0e97fa640 1 -- 172.21.15.148:0/3382417255 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1444633209) 0x7fd0e4014590 con 0x7fd0ec13d3c0 2024-09-16T04:06:44.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T04:06:44.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.480+0000 7fd0e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:44.485 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:44.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-09-16T04:06:44.486 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 >> v1:172.21.15.171:6812/7327273 conn(0x7fd0d0039210 legacy=0x7fd0d003b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 10 monclient: shutdown 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 >> v1:172.21.15.171:6789/0 conn(0x7fd0ec13d3c0 legacy=0x7fd0ec1d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 shutdown_connections 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 >> 172.21.15.148:0/3382417255 conn(0x7fd0ec0a9a40 msgr2=0x7fd0ec13fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 shutdown_connections 2024-09-16T04:06:44.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.484+0000 7fd0f2e8b640 1 -- 172.21.15.148:0/3382417255 wait complete. 2024-09-16T04:06:44.501 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-16T04:06:44.502 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:44.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb50f017640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:06:44.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb50f017640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:06:44.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb50f017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb5080a4bc0 con 0x7fb5080a3a90 2024-09-16T04:06:44.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb5057fa640 1 -- 172.21.15.148:0/2571350380 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 1444633209) 0x7fb4fc015570 con 0x7fb5080a3a90 2024-09-16T04:06:44.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb5057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T04:06:44.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.530+0000 7fb5057fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:44.534 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 >> v1:172.21.15.171:6812/7327273 conn(0x7fb4ec039080 legacy=0x7fb4ec03b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 10 monclient: shutdown 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 >> v1:172.21.15.148:6789/0 conn(0x7fb5080a3a90 legacy=0x7fb50809b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 shutdown_connections 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 >> 172.21.15.148:0/2571350380 conn(0x7fb5080a9a40 msgr2=0x7fb508140020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 shutdown_connections 2024-09-16T04:06:44.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.532+0000 7fb50f017640 1 -- 172.21.15.148:0/2571350380 wait complete. 2024-09-16T04:06:44.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.542+0000 7f8c51ccb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:06:44.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.542+0000 7f8c51ccb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:06:44.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.542+0000 7f8c51ccb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.542+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8c2c005370 con 0x7f8c4c13d060 2024-09-16T04:06:44.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-16T04:06:44.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.543+0000 7f8c33fff640 1 -- 172.21.15.148:0/803168501 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 3427191320) 0x7f8c44014630 con 0x7f8c4c13d060 2024-09-16T04:06:44.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.543+0000 7f8c33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T04:06:44.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.543+0000 7f8c33fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:44.547 INFO:teuthology.orchestra.run.smithi148.stdout:55834574855 2024-09-16T04:06:44.547 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:44.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 >> v1:172.21.15.171:6812/7327273 conn(0x7f8c38039000 legacy=0x7f8c3803b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 10 monclient: shutdown 2024-09-16T04:06:44.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 >> v1:172.21.15.148:6789/0 conn(0x7f8c4c13d060 legacy=0x7f8c4c1d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 shutdown_connections 2024-09-16T04:06:44.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 >> 172.21.15.148:0/803168501 conn(0x7f8c4c0a97b0 msgr2=0x7f8c4c13faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 shutdown_connections 2024-09-16T04:06:44.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.545+0000 7f8c51ccb640 1 -- 172.21.15.148:0/803168501 wait complete. 2024-09-16T04:06:44.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-09-16T04:06:44.559 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:44.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.579+0000 7f420c76c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:06:44.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.579+0000 7f420c76c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:06:44.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.579+0000 7f420c76c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:44.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.579+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4204062790 con 0x7f42040a3a90 2024-09-16T04:06:44.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.580+0000 7f41faffd640 1 -- 172.21.15.148:0/3229824219 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 1444633209) 0x7f41f401e210 con 0x7f42040a3a90 2024-09-16T04:06:44.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.580+0000 7f41faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T04:06:44.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.580+0000 7f41faffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:44.585 INFO:teuthology.orchestra.run.smithi148.stdout:47244640260 2024-09-16T04:06:44.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 >> v1:172.21.15.171:6812/7327273 conn(0x7f41e8038f40 legacy=0x7f41e803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 10 monclient: shutdown 2024-09-16T04:06:44.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 >> v1:172.21.15.148:6789/0 conn(0x7f42040a3a90 legacy=0x7f42041c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 shutdown_connections 2024-09-16T04:06:44.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 >> 172.21.15.148:0/3229824219 conn(0x7f42040a9a40 msgr2=0x7f420413fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 shutdown_connections 2024-09-16T04:06:44.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.582+0000 7f420c76c640 1 -- 172.21.15.148:0/3229824219 wait complete. 2024-09-16T04:06:44.596 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-09-16T04:06:44.596 DEBUG:teuthology.parallel:result is None 2024-09-16T04:06:44.596 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:06:44.596 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:06:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.731+0000 7f123940e640 10 monclient: get_monmap_and_config 2024-09-16T04:06:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.731+0000 7f123940e640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.731+0000 7f123940e640 10 monclient: monmap: 2024-09-16T04:06:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.734 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:44.732979+0000 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:44.732979+0000 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.731+0000 7f123940e640 1 Processor -- start 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.731+0000 7f123940e640 1 -- start start 2024-09-16T04:06:44.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.731+0000 7f123940e640 10 monclient: init 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient(hunting): picked mon.noname-b con 0x7f12340a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient(hunting): picked mon.noname-a con 0x7f12340a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient(hunting): picked mon.noname-c con 0x7f123413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a0590 con 0x7f12340a4580 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a1270 con 0x7f123413d3c0 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12340a4fd0 con 0x7f12340a3a90 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 10 monclient(hunting): authenticate will time out at 825.358312s 2024-09-16T04:06:44.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1232ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f12340a3a90 0x7f12340a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35728/0 (socket says 172.21.15.148:35728) 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1232ffd640 1 -- 172.21.15.148:0/3035465194 learned_addr learned my addr 172.21.15.148:0/3035465194 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f12327fc640 1 --1- 172.21.15.148:0/3035465194 >> v1:172.21.15.148:6789/0 conn(0x7f12340a4580 0x7f123413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45664/0 (socket says 172.21.15.148:45664) 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 807222026 0 0) 0x7f12340a4fd0 con 0x7f12340a3a90 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): my global_id is 4358 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f121c003520 con 0x7f12340a3a90 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4042976173 0 0) 0x7f1228001040 con 0x7f12340a4580 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient(hunting): my global_id is 4362 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12340a4fd0 con 0x7f12340a4580 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1566574321 0 0) 0x7f121c003520 con 0x7f12340a3a90 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 >> v1:172.21.15.171:6789/0 conn(0x7f123413d3c0 legacy=0x7f123413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 >> v1:172.21.15.148:6789/0 conn(0x7f12340a4580 legacy=0x7f123413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 monclient: found mon.noname-b 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123406e170 con 0x7f12340a3a90 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 5 monclient: authenticate success, global_id 4358 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1224001990 con 0x7f12340a3a90 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f1231ffb640 10 monclient: dump: 2024-09-16T04:06:44.745 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.746 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.747 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.732+0000 7f123940e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1224001c70 con 0x7f12340a3a90 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f1231ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f1231ffb640 1 -- 172.21.15.148:0/3035465194 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1224002080 con 0x7f12340a3a90 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f1231ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f1231ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f1231ffb640 10 monclient: dump: 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.748 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 4 set_mon_vals no callback set 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 10 monclient: shutdown 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 1 -- 172.21.15.148:0/3035465194 >> v1:172.21.15.148:6790/0 conn(0x7f12340a3a90 legacy=0x7f12340a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 1 -- 172.21.15.148:0/3035465194 shutdown_connections 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 1 -- 172.21.15.148:0/3035465194 >> 172.21.15.148:0/3035465194 conn(0x7f12340a9a40 msgr2=0x7f12340a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 1 -- 172.21.15.148:0/3035465194 shutdown_connections 2024-09-16T04:06:44.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.733+0000 7f123940e640 1 -- 172.21.15.148:0/3035465194 wait complete. 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.734+0000 7f123940e640 10 monclient: build_initial_monmap 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.734+0000 7f123940e640 10 monclient: monmap: 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.734+0000 7f123940e640 1 Processor -- start 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.734+0000 7f123940e640 1 -- start start 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.734+0000 7f123940e640 10 monclient: init 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient(hunting): picked mon.noname-c con 0x7f12340a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient(hunting): picked mon.noname-a con 0x7f12340a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient(hunting): picked mon.noname-b con 0x7f123413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123409e5b0 con 0x7f12340a4580 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1234138230 con 0x7f123413d3c0 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12341387f0 con 0x7f12340a3a90 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient(hunting): authenticate will time out at 825.361170s 2024-09-16T04:06:44.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f1232ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f12340a3a90 0x7f123409ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35732/0 (socket says 172.21.15.148:35732) 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f1232ffd640 1 -- 172.21.15.148:0/4035844333 learned_addr learned my addr 172.21.15.148:0/4035844333 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f12327fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f12340a4580 0x7f123409d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45674/0 (socket says 172.21.15.148:45674) 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447382950 0 0) 0x7f12341387f0 con 0x7f12340a3a90 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): my global_id is 4361 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1220003520 con 0x7f12340a3a90 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2342471797 0 0) 0x7f123409e5b0 con 0x7f12340a4580 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): my global_id is 4365 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12341387f0 con 0x7f12340a4580 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3202468597 0 0) 0x7f1220003520 con 0x7f12340a3a90 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f123409e5b0 con 0x7f12340a3a90 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1224001fc0 con 0x7f12340a3a90 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.753 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2223602860 0 0) 0x7f12341387f0 con 0x7f12340a4580 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1220003520 con 0x7f12340a4580 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1228001290 con 0x7f12340a4580 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): dump: 2024-09-16T04:06:44.754 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773719761 0 0) 0x7f1234138230 con 0x7f123413d3c0 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:44.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient(hunting): my global_id is 4345 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12341387f0 con 0x7f123413d3c0 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2049328437 0 0) 0x7f123409e5b0 con 0x7f12340a3a90 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 >> v1:172.21.15.171:6789/0 conn(0x7f123413d3c0 legacy=0x7f12341d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 >> v1:172.21.15.148:6789/0 conn(0x7f12340a4580 legacy=0x7f123409d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 monclient: found mon.c 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123409d840 con 0x7f12340a3a90 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:44.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f121b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 5 monclient: authenticate success, global_id 4361 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient: _renew_subs 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f123409f110 con 0x7f12340a3a90 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient: _renew_subs 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.735+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1234138510 con 0x7f12340a3a90 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1224001c70 con 0x7f12340a3a90 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1224005040 con 0x7f12340a3a90 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:44.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f12317fa640 4 set_mon_vals no callback set 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 10 monclient: dump: 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.736+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1224010470 con 0x7f12340a3a90 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.737+0000 7f123940e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.737+0000 7f123940e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.737+0000 7f123940e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:44.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.737+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1214005080 con 0x7f12340a3a90 2024-09-16T04:06:44.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.737+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f12240470b0 con 0x7f12340a3a90 2024-09-16T04:06:44.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.737+0000 7f12327fc640 10 monclient: get_auth_request con 0x7f1220039110 auth_method 0 2024-09-16T04:06:44.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.741+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1224016070 con 0x7f12340a3a90 2024-09-16T04:06:44.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.741+0000 7f121b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:44.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.741+0000 7f121b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:44.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.903+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1214000b90 con 0x7f1220039110 2024-09-16T04:06:44.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.906+0000 7f121b7fe640 1 -- 172.21.15.148:0/4035844333 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 1800975122) 0x7f1214000b90 con 0x7f1220039110 2024-09-16T04:06:44.908 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:44.909 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:06:44.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 >> v1:172.21.15.171:6812/7327273 conn(0x7f1220039110 legacy=0x7f122003b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 10 monclient: shutdown 2024-09-16T04:06:44.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:44.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 >> v1:172.21.15.148:6790/0 conn(0x7f12340a3a90 legacy=0x7f123409ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:44.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 shutdown_connections 2024-09-16T04:06:44.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 >> 172.21.15.148:0/4035844333 conn(0x7f12340a9a40 msgr2=0x7f1234130960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:44.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 shutdown_connections 2024-09-16T04:06:44.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:44.907+0000 7f123940e640 1 -- 172.21.15.148:0/4035844333 wait complete. 2024-09-16T04:06:44.921 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T04:06:43.268683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163220,"kb_used_data":1024,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067820,"statfs":{"total":579820584960,"available":579653447680,"internally_reserved":0,"allocated":1048576,"data_stored":1349908,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.930332"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584889+0000","last_change":"2024-09-16T04:06:41.585087+0000","last_active":"2024-09-16T04:06:41.584889+0000","last_peered":"2024-09-16T04:06:41.584889+0000","last_clean":"2024-09-16T04:06:41.584889+0000","last_became_active":"2024-09-16T04:06:40.097037+0000","last_became_peered":"2024-09-16T04:06:40.097037+0000","last_unstale":"2024-09-16T04:06:41.584889+0000","last_undegraded":"2024-09-16T04:06:41.584889+0000","last_fullsized":"2024-09-16T04:06:41.584889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:02.401977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371006+0000","last_change":"2024-09-16T04:06:41.371006+0000","last_active":"2024-09-16T04:06:41.371006+0000","last_peered":"2024-09-16T04:06:41.371006+0000","last_clean":"2024-09-16T04:06:41.371006+0000","last_became_active":"2024-09-16T04:06:39.358537+0000","last_became_peered":"2024-09-16T04:06:39.358537+0000","last_unstale":"2024-09-16T04:06:41.371006+0000","last_undegraded":"2024-09-16T04:06:41.371006+0000","last_fullsized":"2024-09-16T04:06:41.371006+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:29.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.585112+0000","last_change":"2024-09-16T04:06:41.585281+0000","last_active":"2024-09-16T04:06:41.585112+0000","last_peered":"2024-09-16T04:06:41.585112+0000","last_clean":"2024-09-16T04:06:41.585112+0000","last_became_active":"2024-09-16T04:06:39.350857+0000","last_became_peered":"2024-09-16T04:06:39.350857+0000","last_unstale":"2024-09-16T04:06:41.585112+0000","last_undegraded":"2024-09-16T04:06:41.585112+0000","last_fullsized":"2024-09-16T04:06:41.585112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:47.038041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371026+0000","last_change":"2024-09-16T04:06:41.371026+0000","last_active":"2024-09-16T04:06:41.371026+0000","last_peered":"2024-09-16T04:06:41.371026+0000","last_clean":"2024-09-16T04:06:41.371026+0000","last_became_active":"2024-09-16T04:06:39.357940+0000","last_became_peered":"2024-09-16T04:06:39.357940+0000","last_unstale":"2024-09-16T04:06:41.371026+0000","last_undegraded":"2024-09-16T04:06:41.371026+0000","last_fullsized":"2024-09-16T04:06:41.371026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:07:44.187612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.366120+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.366120+0000","last_peered":"2024-09-16T04:06:41.366120+0000","last_clean":"2024-09-16T04:06:41.366120+0000","last_became_active":"2024-09-16T04:06:39.361009+0000","last_became_peered":"2024-09-16T04:06:39.361009+0000","last_unstale":"2024-09-16T04:06:41.366120+0000","last_undegraded":"2024-09-16T04:06:41.366120+0000","last_fullsized":"2024-09-16T04:06:41.366120+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:31.304607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.367607+0000","last_change":"2024-09-16T04:06:41.367886+0000","last_active":"2024-09-16T04:06:41.367607+0000","last_peered":"2024-09-16T04:06:41.367607+0000","last_clean":"2024-09-16T04:06:41.367607+0000","last_became_active":"2024-09-16T04:06:39.360920+0000","last_became_peered":"2024-09-16T04:06:39.360920+0000","last_unstale":"2024-09-16T04:06:41.367607+0000","last_undegraded":"2024-09-16T04:06:41.367607+0000","last_fullsized":"2024-09-16T04:06:41.367607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:20.390772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584837+0000","last_change":"2024-09-16T04:06:41.585052+0000","last_active":"2024-09-16T04:06:41.584837+0000","last_peered":"2024-09-16T04:06:41.584837+0000","last_clean":"2024-09-16T04:06:41.584837+0000","last_became_active":"2024-09-16T04:06:39.361108+0000","last_became_peered":"2024-09-16T04:06:39.361108+0000","last_unstale":"2024-09-16T04:06:41.584837+0000","last_undegraded":"2024-09-16T04:06:41.584837+0000","last_fullsized":"2024-09-16T04:06:41.584837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:20:22.999464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.365824+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.365824+0000","last_peered":"2024-09-16T04:06:41.365824+0000","last_clean":"2024-09-16T04:06:41.365824+0000","last_became_active":"2024-09-16T04:06:39.355555+0000","last_became_peered":"2024-09-16T04:06:39.355555+0000","last_unstale":"2024-09-16T04:06:41.365824+0000","last_undegraded":"2024-09-16T04:06:41.365824+0000","last_fullsized":"2024-09-16T04:06:41.365824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:52.167894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584588+0000","last_change":"2024-09-16T04:06:33.211282+0000","last_active":"2024-09-16T04:06:41.584588+0000","last_peered":"2024-09-16T04:06:41.584588+0000","last_clean":"2024-09-16T04:06:41.584588+0000","last_became_active":"2024-09-16T04:06:33.209033+0000","last_became_peered":"2024-09-16T04:06:33.209033+0000","last_unstale":"2024-09-16T04:06:41.584588+0000","last_undegraded":"2024-09-16T04:06:41.584588+0000","last_fullsized":"2024-09-16T04:06:41.584588+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:51:10.779025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:06:44.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.017+0000 7fa0f65cf640 10 monclient: get_monmap_and_config 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.017+0000 7fa0f65cf640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.017+0000 7fa0f65cf640 10 monclient: monmap: 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:45.019000+0000 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:45.019000+0000 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.020 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:45.021 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:45.021 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.017+0000 7fa0f65cf640 1 Processor -- start 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.017+0000 7fa0f65cf640 1 -- start start 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.017+0000 7fa0f65cf640 10 monclient: init 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f0133910 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f006c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f013d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f009b0d0 con 0x7fa0f006c430 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f009b340 con 0x7fa0f0133910 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f009b620 con 0x7fa0f013d0e0 2024-09-16T04:06:45.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 10 monclient(hunting): authenticate will time out at 825.644330s 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f4b6c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa0f013d0e0 0x7fa0f013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35740/0 (socket says 172.21.15.148:35740) 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f4b6c640 1 -- 172.21.15.148:0/3186732945 learned_addr learned my addr 172.21.15.148:0/3186732945 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0ef7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa0f006c430 0x7fa0f013b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45680/0 (socket says 172.21.15.148:45680) 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310510384 0 0) 0x7fa0f009b620 con 0x7fa0f013d0e0 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): my global_id is 4364 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0d8003520 con 0x7fa0f013d0e0 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473172499 0 0) 0x7fa0f009b0d0 con 0x7fa0f006c430 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient(hunting): my global_id is 4368 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f009b620 con 0x7fa0f006c430 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 669835968 0 0) 0x7fa0d8003520 con 0x7fa0f013d0e0 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 >> v1:172.21.15.171:6789/0 conn(0x7fa0f0133910 legacy=0x7fa0f0133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 >> v1:172.21.15.148:6789/0 conn(0x7fa0f006c430 legacy=0x7fa0f013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 monclient: found mon.noname-b 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f00633c0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0e8001990 con 0x7fa0f013d0e0 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 5 monclient: authenticate success, global_id 4364 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0f65cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.018+0000 7fa0eeffd640 10 monclient: dump: 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.031 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa0e8001bf0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0eeffd640 1 -- 172.21.15.148:0/3186732945 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0e8001fc0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 4 set_mon_vals no callback set 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0eeffd640 10 monclient: dump: 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.032 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 10 monclient: shutdown 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3186732945 >> v1:172.21.15.148:6790/0 conn(0x7fa0f013d0e0 legacy=0x7fa0f013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3186732945 shutdown_connections 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3186732945 >> 172.21.15.148:0/3186732945 conn(0x7fa0f0062630 msgr2=0x7fa0f0062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3186732945 shutdown_connections 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.019+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3186732945 wait complete. 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.020+0000 7fa0f65cf640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.020+0000 7fa0f65cf640 10 monclient: monmap: 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:45.034 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.020+0000 7fa0f65cf640 1 Processor -- start 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.020+0000 7fa0f65cf640 1 -- start start 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.020+0000 7fa0f65cf640 10 monclient: init 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.020+0000 7fa0f65cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f006c430 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f0133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f013d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f013ad40 con 0x7fa0f0133910 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f0138230 con 0x7fa0f006c430 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f00a4dc0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient(hunting): authenticate will time out at 825.647095s 2024-09-16T04:06:45.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ef7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa0f0133910 0x7fa0f0137b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45690/0 (socket says 172.21.15.148:45690) 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f4b6c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa0f013d0e0 0x7fa0f01d3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35754/0 (socket says 172.21.15.148:35754) 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ef7fe640 1 -- 172.21.15.148:0/3364176909 learned_addr learned my addr 172.21.15.148:0/3364176909 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2622614120 0 0) 0x7fa0f013ad40 con 0x7fa0f0133910 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): my global_id is 4371 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0dc003520 con 0x7fa0f0133910 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805293980 0 0) 0x7fa0f00a4dc0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): my global_id is 4367 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f013ad40 con 0x7fa0f013d0e0 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988730572 0 0) 0x7fa0f0138230 con 0x7fa0f006c430 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): my global_id is 4351 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f00a4dc0 con 0x7fa0f006c430 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2130269953 0 0) 0x7fa0f013ad40 con 0x7fa0f013d0e0 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0f0138230 con 0x7fa0f013d0e0 2024-09-16T04:06:45.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0e8001da0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): dump: 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.038 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 687124311 0 0) 0x7fa0dc003520 con 0x7fa0f0133910 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0f013ad40 con 0x7fa0f0133910 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0e4001960 con 0x7fa0f0133910 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): dump: 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.039 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2938385176 0 0) 0x7fa0f00a4dc0 con 0x7fa0f006c430 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0dc003520 con 0x7fa0f006c430 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0e0001830 con 0x7fa0f006c430 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient(hunting): dump: 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.040 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.041 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3309039779 0 0) 0x7fa0f0138230 con 0x7fa0f013d0e0 2024-09-16T04:06:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 >> v1:172.21.15.171:6789/0 conn(0x7fa0f006c430 legacy=0x7fa0f0137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 >> v1:172.21.15.148:6789/0 conn(0x7fa0f0133910 legacy=0x7fa0f0137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 monclient: found mon.c 2024-09-16T04:06:45.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f013b300 con 0x7fa0f013d0e0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 5 monclient: authenticate success, global_id 4367 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient: _renew_subs 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f01398f0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient: _renew_subs 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.021+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa0f013cc60 con 0x7fa0f013d0e0 2024-09-16T04:06:45.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa0e8001bb0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0e8004f70 con 0x7fa0f013d0e0 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 10 monclient: dump: 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.046 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ee7fc640 4 set_mon_vals no callback set 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa0e80103a0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0f65cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0f65cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0f65cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.022+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d0005080 con 0x7fa0f013d0e0 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.023+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fa0e8046d90 con 0x7fa0f013d0e0 2024-09-16T04:06:45.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.023+0000 7fa0effff640 10 monclient: get_auth_request con 0x7fa0dc0391b0 auth_method 0 2024-09-16T04:06:45.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.027+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa0e8015dd0 con 0x7fa0f013d0e0 2024-09-16T04:06:45.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.027+0000 7fa0ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.027+0000 7fa0ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:45.195 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.192+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0d0000b90 con 0x7fa0dc0391b0 2024-09-16T04:06:45.196 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.193+0000 7fa0ecff9640 1 -- 172.21.15.148:0/3364176909 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 1800975122) 0x7fa0d0000b90 con 0x7fa0dc0391b0 2024-09-16T04:06:45.197 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:45.197 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:06:45.198 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.195+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 >> v1:172.21.15.171:6812/7327273 conn(0x7fa0dc0391b0 legacy=0x7fa0dc03b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.195+0000 7fa0f65cf640 10 monclient: shutdown 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.195+0000 7fa0f65cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.195+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 >> v1:172.21.15.148:6790/0 conn(0x7fa0f013d0e0 legacy=0x7fa0f01d3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.195+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 shutdown_connections 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.195+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 >> 172.21.15.148:0/3364176909 conn(0x7fa0f0062630 msgr2=0x7fa0f013d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.196+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 shutdown_connections 2024-09-16T04:06:45.199 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.196+0000 7fa0f65cf640 1 -- 172.21.15.148:0/3364176909 wait complete. 2024-09-16T04:06:45.209 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T04:06:43.268683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163220,"kb_used_data":1024,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067820,"statfs":{"total":579820584960,"available":579653447680,"internally_reserved":0,"allocated":1048576,"data_stored":1349908,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.930332"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584889+0000","last_change":"2024-09-16T04:06:41.585087+0000","last_active":"2024-09-16T04:06:41.584889+0000","last_peered":"2024-09-16T04:06:41.584889+0000","last_clean":"2024-09-16T04:06:41.584889+0000","last_became_active":"2024-09-16T04:06:40.097037+0000","last_became_peered":"2024-09-16T04:06:40.097037+0000","last_unstale":"2024-09-16T04:06:41.584889+0000","last_undegraded":"2024-09-16T04:06:41.584889+0000","last_fullsized":"2024-09-16T04:06:41.584889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:02.401977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371006+0000","last_change":"2024-09-16T04:06:41.371006+0000","last_active":"2024-09-16T04:06:41.371006+0000","last_peered":"2024-09-16T04:06:41.371006+0000","last_clean":"2024-09-16T04:06:41.371006+0000","last_became_active":"2024-09-16T04:06:39.358537+0000","last_became_peered":"2024-09-16T04:06:39.358537+0000","last_unstale":"2024-09-16T04:06:41.371006+0000","last_undegraded":"2024-09-16T04:06:41.371006+0000","last_fullsized":"2024-09-16T04:06:41.371006+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:29.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.585112+0000","last_change":"2024-09-16T04:06:41.585281+0000","last_active":"2024-09-16T04:06:41.585112+0000","last_peered":"2024-09-16T04:06:41.585112+0000","last_clean":"2024-09-16T04:06:41.585112+0000","last_became_active":"2024-09-16T04:06:39.350857+0000","last_became_peered":"2024-09-16T04:06:39.350857+0000","last_unstale":"2024-09-16T04:06:41.585112+0000","last_undegraded":"2024-09-16T04:06:41.585112+0000","last_fullsized":"2024-09-16T04:06:41.585112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:47.038041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371026+0000","last_change":"2024-09-16T04:06:41.371026+0000","last_active":"2024-09-16T04:06:41.371026+0000","last_peered":"2024-09-16T04:06:41.371026+0000","last_clean":"2024-09-16T04:06:41.371026+0000","last_became_active":"2024-09-16T04:06:39.357940+0000","last_became_peered":"2024-09-16T04:06:39.357940+0000","last_unstale":"2024-09-16T04:06:41.371026+0000","last_undegraded":"2024-09-16T04:06:41.371026+0000","last_fullsized":"2024-09-16T04:06:41.371026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:07:44.187612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.366120+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.366120+0000","last_peered":"2024-09-16T04:06:41.366120+0000","last_clean":"2024-09-16T04:06:41.366120+0000","last_became_active":"2024-09-16T04:06:39.361009+0000","last_became_peered":"2024-09-16T04:06:39.361009+0000","last_unstale":"2024-09-16T04:06:41.366120+0000","last_undegraded":"2024-09-16T04:06:41.366120+0000","last_fullsized":"2024-09-16T04:06:41.366120+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:31.304607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.367607+0000","last_change":"2024-09-16T04:06:41.367886+0000","last_active":"2024-09-16T04:06:41.367607+0000","last_peered":"2024-09-16T04:06:41.367607+0000","last_clean":"2024-09-16T04:06:41.367607+0000","last_became_active":"2024-09-16T04:06:39.360920+0000","last_became_peered":"2024-09-16T04:06:39.360920+0000","last_unstale":"2024-09-16T04:06:41.367607+0000","last_undegraded":"2024-09-16T04:06:41.367607+0000","last_fullsized":"2024-09-16T04:06:41.367607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:20.390772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584837+0000","last_change":"2024-09-16T04:06:41.585052+0000","last_active":"2024-09-16T04:06:41.584837+0000","last_peered":"2024-09-16T04:06:41.584837+0000","last_clean":"2024-09-16T04:06:41.584837+0000","last_became_active":"2024-09-16T04:06:39.361108+0000","last_became_peered":"2024-09-16T04:06:39.361108+0000","last_unstale":"2024-09-16T04:06:41.584837+0000","last_undegraded":"2024-09-16T04:06:41.584837+0000","last_fullsized":"2024-09-16T04:06:41.584837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:20:22.999464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.365824+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.365824+0000","last_peered":"2024-09-16T04:06:41.365824+0000","last_clean":"2024-09-16T04:06:41.365824+0000","last_became_active":"2024-09-16T04:06:39.355555+0000","last_became_peered":"2024-09-16T04:06:39.355555+0000","last_unstale":"2024-09-16T04:06:41.365824+0000","last_undegraded":"2024-09-16T04:06:41.365824+0000","last_fullsized":"2024-09-16T04:06:41.365824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:52.167894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584588+0000","last_change":"2024-09-16T04:06:33.211282+0000","last_active":"2024-09-16T04:06:41.584588+0000","last_peered":"2024-09-16T04:06:41.584588+0000","last_clean":"2024-09-16T04:06:41.584588+0000","last_became_active":"2024-09-16T04:06:33.209033+0000","last_became_peered":"2024-09-16T04:06:33.209033+0000","last_unstale":"2024-09-16T04:06:41.584588+0000","last_undegraded":"2024-09-16T04:06:41.584588+0000","last_fullsized":"2024-09-16T04:06:41.584588+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:51:10.779025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:06:45.210 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:06:45.210 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T04:06:45.210 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T04:06:45.210 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.304+0000 7fbad802e640 10 monclient: get_monmap_and_config 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.304+0000 7fbad802e640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.304+0000 7fbad802e640 10 monclient: monmap: 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:45.306093+0000 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:45.306093+0000 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.307 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.308 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.308 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:45.308 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:45.308 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.304+0000 7fbad802e640 1 Processor -- start 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.304+0000 7fbad802e640 1 -- start start 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.304+0000 7fbad802e640 10 monclient: init 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient(hunting): picked mon.noname-a con 0x7fbad00a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient(hunting): picked mon.noname-b con 0x7fbad009b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient(hunting): picked mon.noname-c con 0x7fbad009b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad009e040 con 0x7fbad00a0e90 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad009d660 con 0x7fbad009b5c0 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad009c0d0 con 0x7fbad009b170 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad802e640 10 monclient(hunting): authenticate will time out at 825.931430s 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad5dca640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbad00a0e90 0x7fbad009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45706/0 (socket says 172.21.15.148:45706) 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad55c9640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fbad009b170 0x7fbad01383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35770/0 (socket says 172.21.15.148:35770) 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad5dca640 1 -- 172.21.15.148:0/2829911572 learned_addr learned my addr 172.21.15.148:0/2829911572 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414473042 0 0) 0x7fbac4001110 con 0x7fbad00a0e90 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): my global_id is 4374 2024-09-16T04:06:45.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbabc003520 con 0x7fbad00a0e90 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2535233317 0 0) 0x7fbad009c0d0 con 0x7fbad009b170 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): my global_id is 4370 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbac4001110 con 0x7fbad009b170 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316312778 0 0) 0x7fbac8001490 con 0x7fbad009b5c0 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 10 monclient(hunting): my global_id is 4354 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.305+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad009c0d0 con 0x7fbad009b5c0 2024-09-16T04:06:45.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2451889702 0 0) 0x7fbac4001110 con 0x7fbad009b170 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 >> v1:172.21.15.171:6789/0 conn(0x7fbad009b5c0 legacy=0x7fbad013bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 >> v1:172.21.15.148:6789/0 conn(0x7fbad00a0e90 legacy=0x7fbad009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 monclient: found mon.noname-b 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad009d380 con 0x7fbad009b170 2024-09-16T04:06:45.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 5 monclient: authenticate success, global_id 4370 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbacc001570 con 0x7fbad009b170 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: dump: 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.320 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbacc001860 con 0x7fbad009b170 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 4 set_mon_vals no callback set 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 1 -- 172.21.15.148:0/2829911572 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbacc001c70 con 0x7fbad009b170 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:45.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad4dc8640 10 monclient: dump: 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 10 monclient: shutdown 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 1 -- 172.21.15.148:0/2829911572 >> v1:172.21.15.148:6790/0 conn(0x7fbad009b170 legacy=0x7fbad01383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 1 -- 172.21.15.148:0/2829911572 shutdown_connections 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 1 -- 172.21.15.148:0/2829911572 >> 172.21.15.148:0/2829911572 conn(0x7fbad00a5640 msgr2=0x7fbad00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 1 -- 172.21.15.148:0/2829911572 shutdown_connections 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.306+0000 7fbad802e640 1 -- 172.21.15.148:0/2829911572 wait complete. 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.307+0000 7fbad802e640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.307+0000 7fbad802e640 10 monclient: monmap: 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.323 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.307+0000 7fbad802e640 1 Processor -- start 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.307+0000 7fbad802e640 1 -- start start 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.307+0000 7fbad802e640 10 monclient: init 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient(hunting): picked mon.noname-c con 0x7fbad009b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient(hunting): picked mon.noname-a con 0x7fbad009b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient(hunting): picked mon.noname-b con 0x7fbad00a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad01c3d20 con 0x7fbad009b5c0 2024-09-16T04:06:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad01a84b0 con 0x7fbad00a0e90 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad01aaf70 con 0x7fbad009b170 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient(hunting): authenticate will time out at 825.934125s 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad5dca640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fbad009b170 0x7fbad01c3610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35780/0 (socket says 172.21.15.148:35780) 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad55c9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbad009b5c0 0x7fbad01ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45710/0 (socket says 172.21.15.148:45710) 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad5dca640 1 -- 172.21.15.148:0/2163582562 learned_addr learned my addr 172.21.15.148:0/2163582562 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232965629 0 0) 0x7fbad01aaf70 con 0x7fbad009b170 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): my global_id is 4373 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbab4003520 con 0x7fbad009b170 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3449264658 0 0) 0x7fbad01c3d20 con 0x7fbad009b5c0 2024-09-16T04:06:45.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): my global_id is 4377 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad01aaf70 con 0x7fbad009b5c0 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1363387634 0 0) 0x7fbab4003520 con 0x7fbad009b170 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbad01c3d20 con 0x7fbad009b170 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbac4001400 con 0x7fbad009b170 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): dump: 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.326 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1449284401 0 0) 0x7fbad01aaf70 con 0x7fbad009b5c0 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbab4003520 con 0x7fbad009b5c0 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbacc001530 con 0x7fbad009b5c0 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): dump: 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475113022 0 0) 0x7fbad01a84b0 con 0x7fbad00a0e90 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient(hunting): my global_id is 4357 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad01aaf70 con 0x7fbad00a0e90 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 945344045 0 0) 0x7fbad01c3d20 con 0x7fbad009b170 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 >> v1:172.21.15.171:6789/0 conn(0x7fbad00a0e90 legacy=0x7fbad01d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 >> v1:172.21.15.148:6789/0 conn(0x7fbad009b5c0 legacy=0x7fbad01ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 monclient: found mon.c 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad0072d30 con 0x7fbad009b170 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 5 monclient: authenticate success, global_id 4373 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient: _renew_subs 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbad01a8ca0 con 0x7fbad009b170 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient: _renew_subs 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbad01a9a20 con 0x7fbad009b170 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbac40047e0 con 0x7fbad009b170 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbac4004bb0 con 0x7fbad009b170 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac3fff640 4 set_mon_vals no callback set 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.308+0000 7fbac27fc640 10 monclient: dump: 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.309+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbac4010060 con 0x7fbad009b170 2024-09-16T04:06:45.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.309+0000 7fbad802e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.309+0000 7fbad802e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.309+0000 7fbad802e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.309+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab8005080 con 0x7fbad009b170 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.310+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7fbac4046db0 con 0x7fbad009b170 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.310+0000 7fbad55c9640 10 monclient: get_auth_request con 0x7fbab4039150 auth_method 0 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.314+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbad009e040 con 0x7fbad009b170 2024-09-16T04:06:45.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.314+0000 7fbac27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.314+0000 7fbac27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbad802e640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbad802e640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbad802e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbab8005370 con 0x7fbad009b170 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbac27fc640 1 -- 172.21.15.148:0/2163582562 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fbac4015d30 con 0x7fbad009b170 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbac27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.517+0000 7fbac27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:45.521 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:45.522 INFO:teuthology.orchestra.run.smithi148.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T04:06:45.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 >> v1:172.21.15.171:6812/7327273 conn(0x7fbab4039150 legacy=0x7fbab403b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 10 monclient: shutdown 2024-09-16T04:06:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 >> v1:172.21.15.148:6790/0 conn(0x7fbad009b170 legacy=0x7fbad01c3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 shutdown_connections 2024-09-16T04:06:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 >> 172.21.15.148:0/2163582562 conn(0x7fbad00a5640 msgr2=0x7fbad0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 shutdown_connections 2024-09-16T04:06:45.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.519+0000 7fbad802e640 1 -- 172.21.15.148:0/2163582562 wait complete. 2024-09-16T04:06:45.533 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T04:06:45.533 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T04:06:45.546 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T04:06:45.546 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.640+0000 7f2f5a7c4640 10 monclient: get_monmap_and_config 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.640+0000 7f2f5a7c4640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.640+0000 7f2f5a7c4640 10 monclient: monmap: 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:45.642131+0000 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:45.642131+0000 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.644 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.644 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:45.644 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:45.644 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.640+0000 7f2f5a7c4640 1 Processor -- start 2024-09-16T04:06:45.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.640+0000 7f2f5a7c4640 1 -- start start 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.640+0000 7f2f5a7c4640 10 monclient: init 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540733a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f54073780 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f5413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f540a3410 con 0x7f2f5413d010 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f540a3190 con 0x7f2f54073780 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f5409e4b0 con 0x7f2f540733a0 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f5a7c4640 10 monclient(hunting): authenticate will time out at 826.267500s 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f58d61640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2f5413d010 0x7f2f5413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45714/0 (socket says 172.21.15.148:45714) 2024-09-16T04:06:45.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f58d61640 1 -- 172.21.15.148:0/1248284150 learned_addr learned my addr 172.21.15.148:0/1248284150 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f53fff640 1 --1- 172.21.15.148:0/1248284150 >> v1:172.21.15.148:6790/0 conn(0x7f2f540733a0 0x7f2f5409ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35796/0 (socket says 172.21.15.148:35796) 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 301951868 0 0) 0x7f2f540a3410 con 0x7f2f5413d010 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): my global_id is 4380 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f3c003520 con 0x7f2f5413d010 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2549178772 0 0) 0x7f2f5409e4b0 con 0x7f2f540733a0 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 10 monclient(hunting): my global_id is 4376 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.641+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f540a3410 con 0x7f2f540733a0 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3668357328 0 0) 0x7f2f3c003520 con 0x7f2f5413d010 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 >> v1:172.21.15.148:6790/0 conn(0x7f2f540733a0 legacy=0x7f2f5409ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 >> v1:172.21.15.171:6789/0 conn(0x7f2f54073780 legacy=0x7f2f5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 monclient: found mon.noname-a 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5406e700 con 0x7f2f5413d010 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 5 monclient: authenticate success, global_id 4380 2024-09-16T04:06:45.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f4c001990 con 0x7f2f5413d010 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: dump: 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.654 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f4c001b80 con 0x7f2f5413d010 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 1 -- 172.21.15.148:0/1248284150 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f4c001f90 con 0x7f2f5413d010 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 4 set_mon_vals no callback set 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f52ffd640 10 monclient: dump: 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.655 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 10 monclient: shutdown 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1248284150 >> v1:172.21.15.148:6789/0 conn(0x7f2f5413d010 legacy=0x7f2f5413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1248284150 shutdown_connections 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1248284150 >> 172.21.15.148:0/1248284150 conn(0x7f2f540a78b0 msgr2=0x7f2f540a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1248284150 shutdown_connections 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.642+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1248284150 wait complete. 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.643+0000 7f2f5a7c4640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.643+0000 7f2f5a7c4640 10 monclient: monmap: 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.643+0000 7f2f5a7c4640 1 Processor -- start 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.643+0000 7f2f5a7c4640 1 -- start start 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.643+0000 7f2f5a7c4640 10 monclient: init 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540733a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient(hunting): picked mon.noname-a con 0x7f2f54073780 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5413d010 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f54122590 con 0x7f2f54073780 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f541240a0 con 0x7f2f540733a0 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f540a4c10 con 0x7f2f5413d010 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f5a7c4640 10 monclient(hunting): authenticate will time out at 826.270474s 2024-09-16T04:06:45.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f537fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2f54073780 0x7f2f54121e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45716/0 (socket says 172.21.15.148:45716) 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f537fe640 1 -- 172.21.15.148:0/1135662369 learned_addr learned my addr 172.21.15.148:0/1135662369 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755682575 0 0) 0x7f2f540a4c10 con 0x7f2f5413d010 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): my global_id is 4379 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f40003520 con 0x7f2f5413d010 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029007059 0 0) 0x7f2f54122590 con 0x7f2f54073780 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): my global_id is 4383 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f540a4c10 con 0x7f2f54073780 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743582952 0 0) 0x7f2f541240a0 con 0x7f2f540733a0 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 10 monclient(hunting): my global_id is 4363 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f54122590 con 0x7f2f540733a0 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1864329727 0 0) 0x7f2f40003520 con 0x7f2f5413d010 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f541240a0 con 0x7f2f5413d010 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.644+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2619230165 0 0) 0x7f2f540a4c10 con 0x7f2f54073780 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f40003520 con 0x7f2f54073780 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f4c001ed0 con 0x7f2f5413d010 2024-09-16T04:06:45.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f480018f0 con 0x7f2f54073780 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.662 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1467005035 0 0) 0x7f2f54122590 con 0x7f2f540733a0 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f540a4c10 con 0x7f2f540733a0 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f44001310 con 0x7f2f540733a0 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient(hunting): dump: 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.663 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3213300956 0 0) 0x7f2f541240a0 con 0x7f2f5413d010 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 >> v1:172.21.15.171:6789/0 conn(0x7f2f540733a0 legacy=0x7f2f54121770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 >> v1:172.21.15.148:6789/0 conn(0x7f2f54073780 legacy=0x7f2f54121e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 monclient: found mon.c 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f540741d0 con 0x7f2f5413d010 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 5 monclient: authenticate success, global_id 4379 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 10 monclient: _renew_subs 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f540a5060 con 0x7f2f5413d010 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 10 monclient: _renew_subs 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f54123dc0 con 0x7f2f5413d010 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f4c004f70 con 0x7f2f5413d010 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f4c005120 con 0x7f2f5413d010 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f527fc640 4 set_mon_vals no callback set 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.645+0000 7f2f50ff9640 10 monclient: dump: 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2f4c010550 con 0x7f2f5413d010 2024-09-16T04:06:45.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f5a7c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f5a7c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f5a7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f34005080 con 0x7f2f5413d010 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (unknown 3531634111 0 0) 0x7f2f4c046ef0 con 0x7f2f5413d010 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.646+0000 7f2f53fff640 10 monclient: get_auth_request con 0x7f2f40039140 auth_method 0 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.650+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2f4c015f50 con 0x7f2f5413d010 2024-09-16T04:06:45.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.650+0000 7f2f50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.650+0000 7f2f50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:45.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.816+0000 7f2f5a7c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.816+0000 7f2f5a7c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.816+0000 7f2f5a7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.816+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f34005370 con 0x7f2f5413d010 2024-09-16T04:06:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.817+0000 7f2f50ff9640 1 -- 172.21.15.148:0/1135662369 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10953 (unknown 3338201251 0 1957694494) 0x7f2f4c01e400 con 0x7f2f5413d010 2024-09-16T04:06:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.817+0000 7f2f50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:06:45.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.817+0000 7f2f50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:45.821 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:45.821 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":18,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:06:41.354592+0000","last_up_change":"2024-09-16T04:06:32.301387+0000","last_in_change":"2024-09-16T04:06:22.826228+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:06:31.297815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:06:37.900873+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":985869260}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":985869260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":985869260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":985869260}]},"public_addr":"172.21.15.148:6808/985869260","cluster_addr":"172.21.15.148:6809/985869260","heartbeat_back_addr":"172.21.15.148:6811/985869260","heartbeat_front_addr":"172.21.15.148:6810/985869260","state":["exists","up"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":3044422561}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":3044422561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":3044422561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":3044422561}]},"public_addr":"172.21.15.148:6800/3044422561","cluster_addr":"172.21.15.148:6801/3044422561","heartbeat_back_addr":"172.21.15.148:6803/3044422561","heartbeat_front_addr":"172.21.15.148:6802/3044422561","state":["exists","up"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":262588452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":262588452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":262588452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":262588452}]},"public_addr":"172.21.15.148:6804/262588452","cluster_addr":"172.21.15.148:6805/262588452","heartbeat_back_addr":"172.21.15.148:6807/262588452","heartbeat_front_addr":"172.21.15.148:6806/262588452","state":["exists","up"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":772155526}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":772155526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":772155526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":772155526}]},"public_addr":"172.21.15.171:6804/772155526","cluster_addr":"172.21.15.171:6805/772155526","heartbeat_back_addr":"172.21.15.171:6807/772155526","heartbeat_front_addr":"172.21.15.171:6806/772155526","state":["exists","up"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":4274747803}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":4274747803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":4274747803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":4274747803}]},"public_addr":"172.21.15.171:6808/4274747803","cluster_addr":"172.21.15.171:6809/4274747803","heartbeat_back_addr":"172.21.15.171:6811/4274747803","heartbeat_front_addr":"172.21.15.171:6810/4274747803","state":["exists","up"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":466524840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":466524840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":466524840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":466524840}]},"public_addr":"172.21.15.171:6800/466524840","cluster_addr":"172.21.15.171:6801/466524840","heartbeat_back_addr":"172.21.15.171:6803/466524840","heartbeat_front_addr":"172.21.15.171:6802/466524840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.060468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:28.867753+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140301+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.136223+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.080093+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:06:45.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.819+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 >> v1:172.21.15.171:6812/7327273 conn(0x7f2f40039140 legacy=0x7f2f4003b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.819+0000 7f2f5a7c4640 10 monclient: shutdown 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.819+0000 7f2f5a7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.819+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 >> v1:172.21.15.148:6790/0 conn(0x7f2f5413d010 legacy=0x7f2f541d3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.819+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 shutdown_connections 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.819+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 >> 172.21.15.148:0/1135662369 conn(0x7f2f540a78b0 msgr2=0x7f2f5409d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.820+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 shutdown_connections 2024-09-16T04:06:45.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.820+0000 7f2f5a7c4640 1 -- 172.21.15.148:0/1135662369 wait complete. 2024-09-16T04:06:45.833 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T04:06:31.297815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T04:06:37.900873+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T04:06:45.833 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.932+0000 7f4748dcf640 10 monclient: get_monmap_and_config 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.932+0000 7f4748dcf640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.932+0000 7f4748dcf640 10 monclient: monmap: 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:45.934009+0000 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:45.934009+0000 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:45.936 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.932+0000 7f4748dcf640 1 Processor -- start 2024-09-16T04:06:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.932+0000 7f4748dcf640 1 -- start start 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.932+0000 7f4748dcf640 10 monclient: init 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f47440a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f47440a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f474413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47440a0e00 con 0x7f474413d060 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47440a10e0 con 0x7f47440a2410 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47440a4970 con 0x7f47440a1b10 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 10 monclient(hunting): authenticate will time out at 826.559360s 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f473bfff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f47440a1b10 0x7f474413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35816/0 (socket says 172.21.15.148:35816) 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4742d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f474413d060 0x7f474413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45720/0 (socket says 172.21.15.148:45720) 2024-09-16T04:06:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f473bfff640 1 -- 172.21.15.148:0/2104080433 learned_addr learned my addr 172.21.15.148:0/2104080433 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131068721 0 0) 0x7f47440a0e00 con 0x7f474413d060 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): my global_id is 4386 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4730003520 con 0x7f474413d060 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2050622366 0 0) 0x7f47440a4970 con 0x7f47440a1b10 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient(hunting): my global_id is 4382 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47440a0e00 con 0x7f47440a1b10 2024-09-16T04:06:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 707888834 0 0) 0x7f4730003520 con 0x7f474413d060 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 >> v1:172.21.15.148:6790/0 conn(0x7f47440a1b10 legacy=0x7f474413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 >> v1:172.21.15.171:6789/0 conn(0x7f47440a2410 legacy=0x7f47440a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 monclient: found mon.noname-a 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474406dfd0 con 0x7f474413d060 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f473c001990 con 0x7f474413d060 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 5 monclient: authenticate success, global_id 4386 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4741d9b640 10 monclient: dump: 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.933+0000 7f4748dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f473c001c70 con 0x7f474413d060 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4741d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4741d9b640 1 -- 172.21.15.148:0/2104080433 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f473c002080 con 0x7f474413d060 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 4 set_mon_vals no callback set 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4741d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4741d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4741d9b640 10 monclient: dump: 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 10 monclient: shutdown 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 1 -- 172.21.15.148:0/2104080433 >> v1:172.21.15.148:6789/0 conn(0x7f474413d060 legacy=0x7f474413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 1 -- 172.21.15.148:0/2104080433 shutdown_connections 2024-09-16T04:06:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 1 -- 172.21.15.148:0/2104080433 >> 172.21.15.148:0/2104080433 conn(0x7f47440a97b0 msgr2=0x7f47440a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 1 -- 172.21.15.148:0/2104080433 shutdown_connections 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.934+0000 7f4748dcf640 1 -- 172.21.15.148:0/2104080433 wait complete. 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.935+0000 7f4748dcf640 10 monclient: build_initial_monmap 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.935+0000 7f4748dcf640 10 monclient: monmap: 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:45.949 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.935+0000 7f4748dcf640 1 Processor -- start 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.935+0000 7f4748dcf640 1 -- start start 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.935+0000 7f4748dcf640 10 monclient: init 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f47440a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f47440a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f474413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4744137850 con 0x7f47440a1b10 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47440a0130 con 0x7f47440a2410 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47441399b0 con 0x7f474413d060 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:45.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient(hunting): authenticate will time out at 826.562186s 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f474259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f47440a1b10 0x7f4744136a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45726/0 (socket says 172.21.15.148:45726) 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f474259c640 1 -- 172.21.15.148:0/503400362 learned_addr learned my addr 172.21.15.148:0/503400362 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1090746439 0 0) 0x7f4744137850 con 0x7f47440a1b10 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): my global_id is 4389 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4724003520 con 0x7f47440a1b10 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4092940122 0 0) 0x7f47441399b0 con 0x7f474413d060 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): my global_id is 4385 2024-09-16T04:06:45.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4744137850 con 0x7f474413d060 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325962644 0 0) 0x7f47440a0130 con 0x7f47440a2410 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): my global_id is 4369 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47441399b0 con 0x7f47440a2410 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 672954577 0 0) 0x7f4724003520 con 0x7f47440a1b10 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47440a0130 con 0x7f47440a1b10 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 416306141 0 0) 0x7f4744137850 con 0x7f474413d060 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4724003520 con 0x7f474413d060 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4734001930 con 0x7f47440a1b10 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): dump: 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:45.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f473c001e20 con 0x7f474413d060 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient(hunting): dump: 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3887122353 0 0) 0x7f47440a0130 con 0x7f47440a1b10 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 >> v1:172.21.15.148:6790/0 conn(0x7f474413d060 legacy=0x7f47441d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 >> v1:172.21.15.171:6789/0 conn(0x7f47440a2410 legacy=0x7f4744137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 monclient: found mon.a 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4744138560 con 0x7f47440a1b10 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient: _finish_auth 0 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 5 monclient: authenticate success, global_id 4389 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient: _renew_subs 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:45.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f474413c690 con 0x7f47440a1b10 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient: _renew_subs 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f474409ebe0 con 0x7f47440a1b10 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4734001cb0 con 0x7f47440a1b10 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.936+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4734003a60 con 0x7f47440a1b10 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.937+0000 7f473affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.937+0000 7f473affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.937+0000 7f473affd640 10 monclient: dump: 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:45.956 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.937+0000 7f474159a640 4 set_mon_vals no callback set 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.937+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4734003c00 con 0x7f47440a1b10 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.937+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f4734046cc0 con 0x7f47440a1b10 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.938+0000 7f4748dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.938+0000 7f4748dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.938+0000 7f4748dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.938+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4728005080 con 0x7f47440a1b10 2024-09-16T04:06:45.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.938+0000 7f473bfff640 10 monclient: get_auth_request con 0x7f4724039100 auth_method 0 2024-09-16T04:06:45.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.942+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4734015ba0 con 0x7f47440a1b10 2024-09-16T04:06:45.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.942+0000 7f473affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:45.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:45.942+0000 7f473affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:46.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.106+0000 7f4748dcf640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.106+0000 7f4748dcf640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.106+0000 7f4748dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.106+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4728005b80 con 0x7f47440a1b10 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.106+0000 7f473affd640 1 -- 172.21.15.148:0/503400362 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (unknown 4170632370 0 2170607528) 0x7f4734019850 con 0x7f47440a1b10 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.107+0000 7f473affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.107+0000 7f473affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:46.110 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 1 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 >> v1:172.21.15.171:6812/7327273 conn(0x7f4724039100 legacy=0x7f472403b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 10 monclient: shutdown 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 >> v1:172.21.15.148:6789/0 conn(0x7f47440a1b10 legacy=0x7f4744136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 shutdown_connections 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 >> 172.21.15.148:0/503400362 conn(0x7f47440a97b0 msgr2=0x7f47440a82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 shutdown_connections 2024-09-16T04:06:46.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.109+0000 7f4748dcf640 1 -- 172.21.15.148:0/503400362 wait complete. 2024-09-16T04:06:46.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T04:06:46.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.220+0000 7efce430c640 10 monclient: get_monmap_and_config 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.220+0000 7efce430c640 10 monclient: build_initial_monmap 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.220+0000 7efce430c640 10 monclient: monmap: 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:46.222736+0000 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:46.222736+0000 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:46.224 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.225 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:46.225 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:46.225 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:46.225 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.221+0000 7efce430c640 1 Processor -- start 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.221+0000 7efce430c640 1 -- start start 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.221+0000 7efce430c640 10 monclient: init 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient(hunting): picked mon.noname-c con 0x7efcdc0733a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient(hunting): picked mon.noname-b con 0x7efcdc073780 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient(hunting): picked mon.noname-a con 0x7efcdc13d0b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc0a3360 con 0x7efcdc13d0b0 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc06e760 con 0x7efcdc0733a0 2024-09-16T04:06:46.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc06db00 con 0x7efcdc073780 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient(hunting): authenticate will time out at 826.848161s 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce28a9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efcdc13d0b0 0x7efcdc13f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45738/0 (socket says 172.21.15.148:45738) 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce28a9640 1 -- 172.21.15.148:0/4275578843 learned_addr learned my addr 172.21.15.148:0/4275578843 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce18a7640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efcdc073780 0x7efcdc13b990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35834/0 (socket says 172.21.15.148:35834) 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814028349 0 0) 0x7efcdc06db00 con 0x7efcdc073780 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): my global_id is 4388 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcc8003520 con 0x7efcdc073780 2024-09-16T04:06:46.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573315126 0 0) 0x7efcd4001490 con 0x7efcdc13d0b0 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): my global_id is 4392 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcdc06db00 con 0x7efcdc13d0b0 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545528753 0 0) 0x7efcdc06e760 con 0x7efcdc0733a0 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient(hunting): my global_id is 4372 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcd4001490 con 0x7efcdc0733a0 2024-09-16T04:06:46.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2173001203 0 0) 0x7efcc8003520 con 0x7efcdc073780 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 >> v1:172.21.15.171:6789/0 conn(0x7efcdc0733a0 legacy=0x7efcdc09ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 >> v1:172.21.15.148:6789/0 conn(0x7efcdc13d0b0 legacy=0x7efcdc13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: _finish_hunting 0 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 monclient: found mon.noname-b 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc09e570 con 0x7efcdc073780 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: _finish_auth 0 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: _check_auth_tickets 2024-09-16T04:06:46.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 5 monclient: authenticate success, global_id 4388 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efccc001610 con 0x7efcdc073780 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: dump: 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.237 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efccc0018c0 con 0x7efcdc073780 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 1 -- 172.21.15.148:0/4275578843 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efccc001cd0 con 0x7efcdc073780 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce430c640 4 set_mon_vals no callback set 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.222+0000 7efce10a6640 10 monclient: dump: 2024-09-16T04:06:46.238 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 10 monclient: shutdown 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 1 -- 172.21.15.148:0/4275578843 >> v1:172.21.15.148:6790/0 conn(0x7efcdc073780 legacy=0x7efcdc13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 1 -- 172.21.15.148:0/4275578843 shutdown_connections 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 1 -- 172.21.15.148:0/4275578843 >> 172.21.15.148:0/4275578843 conn(0x7efcdc06bb90 msgr2=0x7efcdc06bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 1 -- 172.21.15.148:0/4275578843 shutdown_connections 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.223+0000 7efce430c640 1 -- 172.21.15.148:0/4275578843 wait complete. 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient: build_initial_monmap 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient: monmap: 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:46.240 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 1 Processor -- start 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 1 -- start start 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient: init 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient(hunting): picked mon.noname-c con 0x7efcdc0733a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient(hunting): picked mon.noname-a con 0x7efcdc073780 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:46.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient(hunting): picked mon.noname-b con 0x7efcdc13d0b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc123e50 con 0x7efcdc073780 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc124410 con 0x7efcdc13d0b0 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcdc09e110 con 0x7efcdc0733a0 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce430c640 10 monclient(hunting): authenticate will time out at 826.850941s 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce20a8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efcdc0733a0 0x7efcdc121810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35846/0 (socket says 172.21.15.148:35846) 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce20a8640 1 -- 172.21.15.148:0/408966380 learned_addr learned my addr 172.21.15.148:0/408966380 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.224+0000 7efce18a7640 1 --1- 172.21.15.148:0/408966380 >> v1:172.21.15.148:6789/0 conn(0x7efcdc073780 0x7efcdc121f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45744/0 (socket says 172.21.15.148:45744) 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1677464200 0 0) 0x7efcdc09e110 con 0x7efcdc0733a0 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): my global_id is 4391 2024-09-16T04:06:46.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcc0003520 con 0x7efcdc0733a0 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257775891 0 0) 0x7efcdc123e50 con 0x7efcdc073780 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): my global_id is 4395 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcdc09e110 con 0x7efcdc073780 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755741784 0 0) 0x7efcdc124410 con 0x7efcdc13d0b0 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): my global_id is 4375 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcdc123e50 con 0x7efcdc13d0b0 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3515423948 0 0) 0x7efcc0003520 con 0x7efcdc0733a0 2024-09-16T04:06:46.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcdc124410 con 0x7efcdc0733a0 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4251233240 0 0) 0x7efcdc09e110 con 0x7efcdc073780 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcc0003520 con 0x7efcdc073780 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcd80019d0 con 0x7efcdc0733a0 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.244 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efccc001a70 con 0x7efcdc073780 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.245 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.246 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3035236575 0 0) 0x7efcdc123e50 con 0x7efcdc13d0b0 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcdc09e110 con 0x7efcdc13d0b0 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcd4001800 con 0x7efcdc13d0b0 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.247 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3873749311 0 0) 0x7efcdc124410 con 0x7efcdc0733a0 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 >> v1:172.21.15.171:6789/0 conn(0x7efcdc13d0b0 legacy=0x7efcdc1d3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 >> v1:172.21.15.148:6789/0 conn(0x7efcdc073780 legacy=0x7efcdc121f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 monclient: found mon.c 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:46.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc0a4cb0 con 0x7efcdc0733a0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 5 monclient: authenticate success, global_id 4391 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 10 monclient: _renew_subs 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efcdc123b70 con 0x7efcdc0733a0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 10 monclient: _renew_subs 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efcdc122630 con 0x7efcdc0733a0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efcd80038c0 con 0x7efcdc0733a0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcd8003a70 con 0x7efcdc0733a0 2024-09-16T04:06:46.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efce08a5640 4 set_mon_vals no callback set 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.225+0000 7efcd2ffd640 10 monclient: dump: 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.250 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.226+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7efcd8003a70 con 0x7efcdc0733a0 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.226+0000 7efce430c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.226+0000 7efce430c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.226+0000 7efce430c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.226+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc4005080 con 0x7efcdc0733a0 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.227+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7efcd8046af0 con 0x7efcdc0733a0 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.227+0000 7efce18a7640 10 monclient: get_auth_request con 0x7efcc00391e0 auth_method 0 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.231+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7efcd80159d0 con 0x7efcdc0733a0 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.231+0000 7efcd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.231+0000 7efcd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:46.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efce430c640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T04:06:46.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efce430c640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T04:06:46.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efce430c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:46.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7efcc4005b80 con 0x7efcdc0733a0 2024-09-16T04:06:46.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efcd2ffd640 1 -- 172.21.15.148:0/408966380 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (unknown 1398289234 0 265718887) 0x7efcd8019680 con 0x7efcdc0733a0 2024-09-16T04:06:46.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efcd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T04:06:46.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.398+0000 7efcd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:46.402 INFO:teuthology.orchestra.run.smithi148.stdout:pg_num: 8 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.400+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 >> v1:172.21.15.171:6812/7327273 conn(0x7efcc00391e0 legacy=0x7efcc003b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.400+0000 7efce430c640 10 monclient: shutdown 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.400+0000 7efce430c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.400+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 >> v1:172.21.15.148:6790/0 conn(0x7efcdc0733a0 legacy=0x7efcdc121810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.400+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 shutdown_connections 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.400+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 >> 172.21.15.148:0/408966380 conn(0x7efcdc06bb90 msgr2=0x7efcdc06c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.401+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 shutdown_connections 2024-09-16T04:06:46.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.401+0000 7efce430c640 1 -- 172.21.15.148:0/408966380 wait complete. 2024-09-16T04:06:46.414 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T04:06:46.414 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.510+0000 7f4fa45f6640 10 monclient: get_monmap_and_config 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.510+0000 7f4fa45f6640 10 monclient: build_initial_monmap 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.510+0000 7f4fa45f6640 10 monclient: monmap: 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:46.511982+0000 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:46.511982+0000 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:46.513 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.514 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:46.514 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:46.514 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:46.514 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.510+0000 7f4fa45f6640 1 Processor -- start 2024-09-16T04:06:46.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.510+0000 7f4fa45f6640 1 -- start start 2024-09-16T04:06:46.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.510+0000 7f4fa45f6640 10 monclient: init 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c0a0e00 con 0x7f4f9c0a2410 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c0a10e0 con 0x7f4f9c0a1b10 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c0a4970 con 0x7f4f9c13d060 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:46.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa45f6640 10 monclient(hunting): authenticate will time out at 827.137506s 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa2392640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4f9c0a2410 0x7f4f9c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45754/0 (socket says 172.21.15.148:45754) 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa2392640 1 -- 172.21.15.148:0/1221727643 learned_addr learned my addr 172.21.15.148:0/1221727643 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 478793272 0 0) 0x7f4f9c0a4970 con 0x7f4f9c13d060 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): my global_id is 4394 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f88003520 con 0x7f4f9c13d060 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936596049 0 0) 0x7f4f9c0a0e00 con 0x7f4f9c0a2410 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 10 monclient(hunting): my global_id is 4398 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.511+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9c0a4970 con 0x7f4f9c0a2410 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4213158239 0 0) 0x7f4f9c0a4970 con 0x7f4f9c0a2410 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 >> v1:172.21.15.148:6790/0 conn(0x7f4f9c13d060 legacy=0x7f4f9c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 >> v1:172.21.15.171:6789/0 conn(0x7f4f9c0a1b10 legacy=0x7f4f9c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: _finish_hunting 0 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 monclient: found mon.noname-a 2024-09-16T04:06:46.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c06dfd0 con 0x7f4f9c0a2410 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: _finish_auth 0 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: _check_auth_tickets 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 5 monclient: authenticate success, global_id 4398 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f98001610 con 0x7f4f9c0a2410 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: dump: 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.527 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f98001870 con 0x7f4f9c0a2410 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 1 -- 172.21.15.148:0/1221727643 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f98001c80 con 0x7f4f9c0a2410 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa1390640 10 monclient: dump: 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.528 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 4 set_mon_vals no callback set 2024-09-16T04:06:46.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 10 monclient: shutdown 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 1 -- 172.21.15.148:0/1221727643 >> v1:172.21.15.148:6789/0 conn(0x7f4f9c0a2410 legacy=0x7f4f9c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 1 -- 172.21.15.148:0/1221727643 shutdown_connections 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 1 -- 172.21.15.148:0/1221727643 >> 172.21.15.148:0/1221727643 conn(0x7f4f9c0a97b0 msgr2=0x7f4f9c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 1 -- 172.21.15.148:0/1221727643 shutdown_connections 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.512+0000 7f4fa45f6640 1 -- 172.21.15.148:0/1221727643 wait complete. 2024-09-16T04:06:46.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.513+0000 7f4fa45f6640 10 monclient: build_initial_monmap 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.513+0000 7f4fa45f6640 10 monclient: monmap: 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:46.531 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.532 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:46.532 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:46.532 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:46.532 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.513+0000 7f4fa45f6640 1 Processor -- start 2024-09-16T04:06:46.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.513+0000 7f4fa45f6640 1 -- start start 2024-09-16T04:06:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.513+0000 7f4fa45f6640 10 monclient: init 2024-09-16T04:06:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:46.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:46.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c137770 con 0x7f4f9c0a2410 2024-09-16T04:06:46.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c137a50 con 0x7f4f9c0a1b10 2024-09-16T04:06:46.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c1cbaf0 con 0x7f4f9c13d060 2024-09-16T04:06:46.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:46.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa45f6640 10 monclient(hunting): authenticate will time out at 827.140249s 2024-09-16T04:06:46.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa1b91640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4f9c0a2410 0x7f4f9c1d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45760/0 (socket says 172.21.15.148:45760) 2024-09-16T04:06:46.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa1b91640 1 -- 172.21.15.148:0/3795763925 learned_addr learned my addr 172.21.15.148:0/3795763925 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:46.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4fa2b93640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f4f9c13d060 0x7f4f9c1d3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35860/0 (socket says 172.21.15.148:35860) 2024-09-16T04:06:46.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006854524 0 0) 0x7f4f9c137770 con 0x7f4f9c0a2410 2024-09-16T04:06:46.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 4401 2024-09-16T04:06:46.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f80003520 con 0x7f4f9c0a2410 2024-09-16T04:06:46.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530294159 0 0) 0x7f4f9c1cbaf0 con 0x7f4f9c13d060 2024-09-16T04:06:46.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 4397 2024-09-16T04:06:46.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9c137770 con 0x7f4f9c13d060 2024-09-16T04:06:46.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717087957 0 0) 0x7f4f9c137a50 con 0x7f4f9c0a1b10 2024-09-16T04:06:46.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 4381 2024-09-16T04:06:46.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9c1cbaf0 con 0x7f4f9c0a1b10 2024-09-16T04:06:46.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3463911739 0 0) 0x7f4f80003520 con 0x7f4f9c0a2410 2024-09-16T04:06:46.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f9c137a50 con 0x7f4f9c0a2410 2024-09-16T04:06:46.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f8c0019b0 con 0x7f4f9c0a2410 2024-09-16T04:06:46.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:46.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.540 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.540 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.540 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.540 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.540 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.540 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.541 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.541 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:46.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2366681579 0 0) 0x7f4f9c137770 con 0x7f4f9c13d060 2024-09-16T04:06:46.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f80003520 con 0x7f4f9c13d060 2024-09-16T04:06:46.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f94001db0 con 0x7f4f9c13d060 2024-09-16T04:06:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.542 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.543 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.544 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 250698239 0 0) 0x7f4f9c137a50 con 0x7f4f9c0a2410 2024-09-16T04:06:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 >> v1:172.21.15.148:6790/0 conn(0x7f4f9c13d060 legacy=0x7f4f9c1d3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 >> v1:172.21.15.171:6789/0 conn(0x7f4f9c0a1b10 legacy=0x7f4f9c13bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:46.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 monclient: found mon.a 2024-09-16T04:06:46.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:46.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c0a0130 con 0x7f4f9c0a2410 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.514+0000 7f4f92ffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 5 monclient: authenticate success, global_id 4401 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 10 monclient: _renew_subs 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9c09f2b0 con 0x7f4f9c0a2410 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 10 monclient: _renew_subs 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f9c09f4f0 con 0x7f4f9c0a2410 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f8c0038c0 con 0x7f4f9c0a2410 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f8c003c90 con 0x7f4f9c0a2410 2024-09-16T04:06:46.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4fa0b8f640 4 set_mon_vals no callback set 2024-09-16T04:06:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 10 monclient: dump: 2024-09-16T04:06:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.548 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.549 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.515+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4f8c003e30 con 0x7f4f9c0a2410 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.516+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f4f8c046cc0 con 0x7f4f9c0a2410 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.516+0000 7f4fa45f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.516+0000 7f4fa45f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.516+0000 7f4fa45f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.516+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f84005080 con 0x7f4f9c0a2410 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.516+0000 7f4fa2392640 10 monclient: get_auth_request con 0x7f4f80039010 auth_method 0 2024-09-16T04:06:46.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.520+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4f8c015d40 con 0x7f4f9c0a2410 2024-09-16T04:06:46.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.520+0000 7f4f92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.520+0000 7f4f92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:46.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.685+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f84000b90 con 0x7f4f80039010 2024-09-16T04:06:46.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.686+0000 7f4f92ffd640 1 -- 172.21.15.148:0/3795763925 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34167 (unknown 2965378022 0 2170131451) 0x7f4f84000b90 con 0x7f4f80039010 2024-09-16T04:06:46.694 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:46.694 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:06:46.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 >> v1:172.21.15.171:6812/7327273 conn(0x7f4f80039010 legacy=0x7f4f8003b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 10 monclient: shutdown 2024-09-16T04:06:46.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 >> v1:172.21.15.148:6789/0 conn(0x7f4f9c0a2410 legacy=0x7f4f9c1d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 shutdown_connections 2024-09-16T04:06:46.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 >> 172.21.15.148:0/3795763925 conn(0x7f4f9c0a97b0 msgr2=0x7f4f9c13f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.688+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 shutdown_connections 2024-09-16T04:06:46.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.689+0000 7f4fa45f6640 1 -- 172.21.15.148:0/3795763925 wait complete. 2024-09-16T04:06:46.702 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T04:06:45.930530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163220,"kb_used_data":1024,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067820,"statfs":{"total":579820584960,"available":579653447680,"internally_reserved":0,"allocated":1048576,"data_stored":1349908,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.585548"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584889+0000","last_change":"2024-09-16T04:06:41.585087+0000","last_active":"2024-09-16T04:06:41.584889+0000","last_peered":"2024-09-16T04:06:41.584889+0000","last_clean":"2024-09-16T04:06:41.584889+0000","last_became_active":"2024-09-16T04:06:40.097037+0000","last_became_peered":"2024-09-16T04:06:40.097037+0000","last_unstale":"2024-09-16T04:06:41.584889+0000","last_undegraded":"2024-09-16T04:06:41.584889+0000","last_fullsized":"2024-09-16T04:06:41.584889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:02.401977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371006+0000","last_change":"2024-09-16T04:06:41.371006+0000","last_active":"2024-09-16T04:06:41.371006+0000","last_peered":"2024-09-16T04:06:41.371006+0000","last_clean":"2024-09-16T04:06:41.371006+0000","last_became_active":"2024-09-16T04:06:39.358537+0000","last_became_peered":"2024-09-16T04:06:39.358537+0000","last_unstale":"2024-09-16T04:06:41.371006+0000","last_undegraded":"2024-09-16T04:06:41.371006+0000","last_fullsized":"2024-09-16T04:06:41.371006+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:29.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.585112+0000","last_change":"2024-09-16T04:06:41.585281+0000","last_active":"2024-09-16T04:06:41.585112+0000","last_peered":"2024-09-16T04:06:41.585112+0000","last_clean":"2024-09-16T04:06:41.585112+0000","last_became_active":"2024-09-16T04:06:39.350857+0000","last_became_peered":"2024-09-16T04:06:39.350857+0000","last_unstale":"2024-09-16T04:06:41.585112+0000","last_undegraded":"2024-09-16T04:06:41.585112+0000","last_fullsized":"2024-09-16T04:06:41.585112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:47.038041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371026+0000","last_change":"2024-09-16T04:06:41.371026+0000","last_active":"2024-09-16T04:06:41.371026+0000","last_peered":"2024-09-16T04:06:41.371026+0000","last_clean":"2024-09-16T04:06:41.371026+0000","last_became_active":"2024-09-16T04:06:39.357940+0000","last_became_peered":"2024-09-16T04:06:39.357940+0000","last_unstale":"2024-09-16T04:06:41.371026+0000","last_undegraded":"2024-09-16T04:06:41.371026+0000","last_fullsized":"2024-09-16T04:06:41.371026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:07:44.187612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.366120+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.366120+0000","last_peered":"2024-09-16T04:06:41.366120+0000","last_clean":"2024-09-16T04:06:41.366120+0000","last_became_active":"2024-09-16T04:06:39.361009+0000","last_became_peered":"2024-09-16T04:06:39.361009+0000","last_unstale":"2024-09-16T04:06:41.366120+0000","last_undegraded":"2024-09-16T04:06:41.366120+0000","last_fullsized":"2024-09-16T04:06:41.366120+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:31.304607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.367607+0000","last_change":"2024-09-16T04:06:41.367886+0000","last_active":"2024-09-16T04:06:41.367607+0000","last_peered":"2024-09-16T04:06:41.367607+0000","last_clean":"2024-09-16T04:06:41.367607+0000","last_became_active":"2024-09-16T04:06:39.360920+0000","last_became_peered":"2024-09-16T04:06:39.360920+0000","last_unstale":"2024-09-16T04:06:41.367607+0000","last_undegraded":"2024-09-16T04:06:41.367607+0000","last_fullsized":"2024-09-16T04:06:41.367607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:20.390772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584837+0000","last_change":"2024-09-16T04:06:41.585052+0000","last_active":"2024-09-16T04:06:41.584837+0000","last_peered":"2024-09-16T04:06:41.584837+0000","last_clean":"2024-09-16T04:06:41.584837+0000","last_became_active":"2024-09-16T04:06:39.361108+0000","last_became_peered":"2024-09-16T04:06:39.361108+0000","last_unstale":"2024-09-16T04:06:41.584837+0000","last_undegraded":"2024-09-16T04:06:41.584837+0000","last_fullsized":"2024-09-16T04:06:41.584837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:20:22.999464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.365824+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.365824+0000","last_peered":"2024-09-16T04:06:41.365824+0000","last_clean":"2024-09-16T04:06:41.365824+0000","last_became_active":"2024-09-16T04:06:39.355555+0000","last_became_peered":"2024-09-16T04:06:39.355555+0000","last_unstale":"2024-09-16T04:06:41.365824+0000","last_undegraded":"2024-09-16T04:06:41.365824+0000","last_fullsized":"2024-09-16T04:06:41.365824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:52.167894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584588+0000","last_change":"2024-09-16T04:06:33.211282+0000","last_active":"2024-09-16T04:06:41.584588+0000","last_peered":"2024-09-16T04:06:41.584588+0000","last_clean":"2024-09-16T04:06:41.584588+0000","last_became_active":"2024-09-16T04:06:33.209033+0000","last_became_peered":"2024-09-16T04:06:33.209033+0000","last_unstale":"2024-09-16T04:06:41.584588+0000","last_undegraded":"2024-09-16T04:06:41.584588+0000","last_fullsized":"2024-09-16T04:06:41.584588+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:51:10.779025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:06:46.704 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:06:46.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.799+0000 7fb69474b640 10 monclient: get_monmap_and_config 2024-09-16T04:06:46.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.799+0000 7fb69474b640 10 monclient: build_initial_monmap 2024-09-16T04:06:46.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.799+0000 7fb69474b640 10 monclient: monmap: 2024-09-16T04:06:46.802 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:46.802 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.803 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:46.800930+0000 2024-09-16T04:06:46.803 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:46.800930+0000 2024-09-16T04:06:46.803 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:46.803 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.803 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:46.804 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:46.804 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:46.804 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.799+0000 7fb69474b640 1 Processor -- start 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.799+0000 7fb69474b640 1 -- start start 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.799+0000 7fb69474b640 10 monclient: init 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient(hunting): picked mon.noname-a con 0x7fb68c09ebf0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient(hunting): picked mon.noname-c con 0x7fb68c09efd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient(hunting): picked mon.noname-b con 0x7fb68c09f3d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c09cbf0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c09ced0 con 0x7fb68c09efd0 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c09d1b0 con 0x7fb68c09f3d0 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 10 monclient(hunting): authenticate will time out at 827.426362s 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6924e7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb68c09ebf0 0x7fb68c136bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45762/0 (socket says 172.21.15.148:45762) 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6924e7640 1 -- 172.21.15.148:0/2354888129 learned_addr learned my addr 172.21.15.148:0/2354888129 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4125691236 0 0) 0x7fb68c09cbf0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): my global_id is 4404 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb678003520 con 0x7fb68c09ebf0 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810946581 0 0) 0x7fb68c09d1b0 con 0x7fb68c09f3d0 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): my global_id is 4400 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb68c09cbf0 con 0x7fb68c09f3d0 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867540055 0 0) 0x7fb68c09ced0 con 0x7fb68c09efd0 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient(hunting): my global_id is 4384 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb68c09d1b0 con 0x7fb68c09efd0 2024-09-16T04:06:46.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4258570771 0 0) 0x7fb678003520 con 0x7fb68c09ebf0 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 >> v1:172.21.15.148:6790/0 conn(0x7fb68c09f3d0 legacy=0x7fb68c13f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 >> v1:172.21.15.171:6789/0 conn(0x7fb68c09efd0 legacy=0x7fb68c13bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient: _finish_hunting 0 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 monclient: found mon.noname-a 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c137300 con 0x7fb68c09ebf0 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient: _finish_auth 0 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 10 monclient: _check_auth_tickets 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb69474b640 5 monclient: authenticate success, global_id 4404 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.800+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb688001610 con 0x7fb68c09ebf0 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:46.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: dump: 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6880018c0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 4 set_mon_vals no callback set 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 1 -- 172.21.15.148:0/2354888129 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb688001c90 con 0x7fb68c09ebf0 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb6914e5640 10 monclient: dump: 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.821 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.822 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 10 monclient: shutdown 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 1 -- 172.21.15.148:0/2354888129 >> v1:172.21.15.148:6789/0 conn(0x7fb68c09ebf0 legacy=0x7fb68c136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 1 -- 172.21.15.148:0/2354888129 shutdown_connections 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 1 -- 172.21.15.148:0/2354888129 >> 172.21.15.148:0/2354888129 conn(0x7fb68c0a5860 msgr2=0x7fb68c0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 1 -- 172.21.15.148:0/2354888129 shutdown_connections 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.801+0000 7fb69474b640 1 -- 172.21.15.148:0/2354888129 wait complete. 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient: build_initial_monmap 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient: monmap: 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:46.824 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 1 Processor -- start 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 1 -- start start 2024-09-16T04:06:46.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient: init 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient(hunting): picked mon.noname-a con 0x7fb68c09ebf0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.802+0000 7fb69474b640 10 monclient(hunting): picked mon.noname-c con 0x7fb68c09efd0 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient(hunting): picked mon.noname-b con 0x7fb68c09f3d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c12ab40 con 0x7fb68c09ebf0 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c12b100 con 0x7fb68c09f3d0 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68c13f370 con 0x7fb68c09efd0 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient(hunting): authenticate will time out at 827.429057s 2024-09-16T04:06:46.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb6924e7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb68c09ebf0 0x7fb68c098670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45766/0 (socket says 172.21.15.148:45766) 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb6924e7640 1 -- 172.21.15.148:0/2532931201 learned_addr learned my addr 172.21.15.148:0/2532931201 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936505138 0 0) 0x7fb68c12ab40 con 0x7fb68c09ebf0 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): my global_id is 4407 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb670003550 con 0x7fb68c09ebf0 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2834574442 0 0) 0x7fb68c13f370 con 0x7fb68c09efd0 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): my global_id is 4403 2024-09-16T04:06:46.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb68c12ab40 con 0x7fb68c09efd0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2701338217 0 0) 0x7fb670003550 con 0x7fb68c09ebf0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb68c13f370 con 0x7fb68c09ebf0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307405008 0 0) 0x7fb68c12b100 con 0x7fb68c09f3d0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): my global_id is 4387 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb670003550 con 0x7fb68c09f3d0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb688001c10 con 0x7fb68c09ebf0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:46.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3367199596 0 0) 0x7fb68c12ab40 con 0x7fb68c09efd0 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb68c12b100 con 0x7fb68c09efd0 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb67c001850 con 0x7fb68c09efd0 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient(hunting): dump: 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.830 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1719678161 0 0) 0x7fb68c13f370 con 0x7fb68c09ebf0 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 >> v1:172.21.15.148:6790/0 conn(0x7fb68c09efd0 legacy=0x7fb68c1d0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 >> v1:172.21.15.171:6789/0 conn(0x7fb68c09f3d0 legacy=0x7fb68c1d44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 monclient: found mon.a 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c13fa50 con 0x7fb68c09ebf0 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:46.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 5 monclient: authenticate success, global_id 4407 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient: _renew_subs 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb68c13f7b0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient: _renew_subs 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb68c098f60 con 0x7fb68c09ebf0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb688004ba0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb688004f70 con 0x7fb68c09ebf0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:46.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb682ffd640 10 monclient: dump: 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.803+0000 7fb690ce4640 4 set_mon_vals no callback set 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.804+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb688005210 con 0x7fb68c09ebf0 2024-09-16T04:06:46.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.804+0000 7fb69474b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.804+0000 7fb69474b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.804+0000 7fb69474b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.804+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb68c12aee0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.804+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7fb688046de0 con 0x7fb68c09ebf0 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.805+0000 7fb691ce6640 10 monclient: get_auth_request con 0x7fb670039060 auth_method 0 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.809+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb688015d80 con 0x7fb68c09ebf0 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.809+0000 7fb682ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:46.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.809+0000 7fb682ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:46.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.972+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb68c12a570 con 0x7fb670039060 2024-09-16T04:06:46.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.973+0000 7fb682ffd640 1 -- 172.21.15.148:0/2532931201 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34167 (unknown 2965378022 0 2170131451) 0x7fb68c12a570 con 0x7fb670039060 2024-09-16T04:06:46.977 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:06:46.977 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.975+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 >> v1:172.21.15.171:6812/7327273 conn(0x7fb670039060 legacy=0x7fb67003b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.975+0000 7fb69474b640 10 monclient: shutdown 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.975+0000 7fb69474b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.975+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 >> v1:172.21.15.148:6789/0 conn(0x7fb68c09ebf0 legacy=0x7fb68c098670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.975+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 shutdown_connections 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.975+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 >> 172.21.15.148:0/2532931201 conn(0x7fb68c0a5860 msgr2=0x7fb68c0a26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.976+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 shutdown_connections 2024-09-16T04:06:46.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:46.976+0000 7fb69474b640 1 -- 172.21.15.148:0/2532931201 wait complete. 2024-09-16T04:06:46.989 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T04:06:45.930530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163220,"kb_used_data":1024,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067820,"statfs":{"total":579820584960,"available":579653447680,"internally_reserved":0,"allocated":1048576,"data_stored":1349908,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.585548"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584889+0000","last_change":"2024-09-16T04:06:41.585087+0000","last_active":"2024-09-16T04:06:41.584889+0000","last_peered":"2024-09-16T04:06:41.584889+0000","last_clean":"2024-09-16T04:06:41.584889+0000","last_became_active":"2024-09-16T04:06:40.097037+0000","last_became_peered":"2024-09-16T04:06:40.097037+0000","last_unstale":"2024-09-16T04:06:41.584889+0000","last_undegraded":"2024-09-16T04:06:41.584889+0000","last_fullsized":"2024-09-16T04:06:41.584889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:02.401977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371006+0000","last_change":"2024-09-16T04:06:41.371006+0000","last_active":"2024-09-16T04:06:41.371006+0000","last_peered":"2024-09-16T04:06:41.371006+0000","last_clean":"2024-09-16T04:06:41.371006+0000","last_became_active":"2024-09-16T04:06:39.358537+0000","last_became_peered":"2024-09-16T04:06:39.358537+0000","last_unstale":"2024-09-16T04:06:41.371006+0000","last_undegraded":"2024-09-16T04:06:41.371006+0000","last_fullsized":"2024-09-16T04:06:41.371006+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:29.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.585112+0000","last_change":"2024-09-16T04:06:41.585281+0000","last_active":"2024-09-16T04:06:41.585112+0000","last_peered":"2024-09-16T04:06:41.585112+0000","last_clean":"2024-09-16T04:06:41.585112+0000","last_became_active":"2024-09-16T04:06:39.350857+0000","last_became_peered":"2024-09-16T04:06:39.350857+0000","last_unstale":"2024-09-16T04:06:41.585112+0000","last_undegraded":"2024-09-16T04:06:41.585112+0000","last_fullsized":"2024-09-16T04:06:41.585112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:47.038041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.371026+0000","last_change":"2024-09-16T04:06:41.371026+0000","last_active":"2024-09-16T04:06:41.371026+0000","last_peered":"2024-09-16T04:06:41.371026+0000","last_clean":"2024-09-16T04:06:41.371026+0000","last_became_active":"2024-09-16T04:06:39.357940+0000","last_became_peered":"2024-09-16T04:06:39.357940+0000","last_unstale":"2024-09-16T04:06:41.371026+0000","last_undegraded":"2024-09-16T04:06:41.371026+0000","last_fullsized":"2024-09-16T04:06:41.371026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:07:44.187612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.366120+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.366120+0000","last_peered":"2024-09-16T04:06:41.366120+0000","last_clean":"2024-09-16T04:06:41.366120+0000","last_became_active":"2024-09-16T04:06:39.361009+0000","last_became_peered":"2024-09-16T04:06:39.361009+0000","last_unstale":"2024-09-16T04:06:41.366120+0000","last_undegraded":"2024-09-16T04:06:41.366120+0000","last_fullsized":"2024-09-16T04:06:41.366120+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:31.304607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.367607+0000","last_change":"2024-09-16T04:06:41.367886+0000","last_active":"2024-09-16T04:06:41.367607+0000","last_peered":"2024-09-16T04:06:41.367607+0000","last_clean":"2024-09-16T04:06:41.367607+0000","last_became_active":"2024-09-16T04:06:39.360920+0000","last_became_peered":"2024-09-16T04:06:39.360920+0000","last_unstale":"2024-09-16T04:06:41.367607+0000","last_undegraded":"2024-09-16T04:06:41.367607+0000","last_fullsized":"2024-09-16T04:06:41.367607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:20.390772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584837+0000","last_change":"2024-09-16T04:06:41.585052+0000","last_active":"2024-09-16T04:06:41.584837+0000","last_peered":"2024-09-16T04:06:41.584837+0000","last_clean":"2024-09-16T04:06:41.584837+0000","last_became_active":"2024-09-16T04:06:39.361108+0000","last_became_peered":"2024-09-16T04:06:39.361108+0000","last_unstale":"2024-09-16T04:06:41.584837+0000","last_undegraded":"2024-09-16T04:06:41.584837+0000","last_fullsized":"2024-09-16T04:06:41.584837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:20:22.999464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.365824+0000","last_change":"2024-09-16T04:06:41.366120+0000","last_active":"2024-09-16T04:06:41.365824+0000","last_peered":"2024-09-16T04:06:41.365824+0000","last_clean":"2024-09-16T04:06:41.365824+0000","last_became_active":"2024-09-16T04:06:39.355555+0000","last_became_peered":"2024-09-16T04:06:39.355555+0000","last_unstale":"2024-09-16T04:06:41.365824+0000","last_undegraded":"2024-09-16T04:06:41.365824+0000","last_fullsized":"2024-09-16T04:06:41.365824+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:52.167894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T04:06:41.584588+0000","last_change":"2024-09-16T04:06:33.211282+0000","last_active":"2024-09-16T04:06:41.584588+0000","last_peered":"2024-09-16T04:06:41.584588+0000","last_clean":"2024-09-16T04:06:41.584588+0000","last_became_active":"2024-09-16T04:06:33.209033+0000","last_became_peered":"2024-09-16T04:06:33.209033+0000","last_unstale":"2024-09-16T04:06:41.584588+0000","last_undegraded":"2024-09-16T04:06:41.584588+0000","last_fullsized":"2024-09-16T04:06:41.584588+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:51:10.779025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":48688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":48669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":577597,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:06:46.990 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T04:06:46.990 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T04:06:46.990 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T04:06:47.004 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T04:06:47.005 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T04:06:47.005 DEBUG:teuthology.orchestra.run.smithi171:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T04:06:47.005 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T04:06:47.006 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726459606, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T04:06:47.006 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:06:47.006 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:06:47.025 INFO:teuthology.orchestra.run.smithi171.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T04:06:47.026 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T04:06:47.084 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T04:06:47.084 DEBUG:teuthology.orchestra.run.smithi171:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T04:06:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient: get_monmap_and_config 2024-09-16T04:06:47.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient: monmap: 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:47.102633+0000 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:47.102633+0000 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 1 Processor -- start 2024-09-16T04:06:47.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 1 -- start start 2024-09-16T04:06:47.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient: init 2024-09-16T04:06:47.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ec0a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ec09b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ec09b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec09e040 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec09d660 con 0x7fa3ec09b170 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec09c0d0 con 0x7fa3ec0a0e90 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.101+0000 7fa3f3da6640 10 monclient(hunting): authenticate will time out at 827.727980s 2024-09-16T04:06:47.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f2343640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa3ec09b5c0 0x7fa3ec13bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45776/0 (socket says 172.21.15.148:45776) 2024-09-16T04:06:47.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f1b42640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa3ec0a0e90 0x7fa3ec09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35894/0 (socket says 172.21.15.148:35894) 2024-09-16T04:06:47.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f2343640 1 -- 172.21.15.148:0/2218798905 learned_addr learned my addr 172.21.15.148:0/2218798905 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324923935 0 0) 0x7fa3ec09c0d0 con 0x7fa3ec0a0e90 2024-09-16T04:06:47.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): my global_id is 4406 2024-09-16T04:06:47.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d8003520 con 0x7fa3ec0a0e90 2024-09-16T04:06:47.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767858771 0 0) 0x7fa3e4001490 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): my global_id is 4410 2024-09-16T04:06:47.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3ec09c0d0 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1454025828 0 0) 0x7fa3dc001110 con 0x7fa3ec09b170 2024-09-16T04:06:47.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient(hunting): my global_id is 4390 2024-09-16T04:06:47.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3e4001490 con 0x7fa3ec09b170 2024-09-16T04:06:47.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 511058662 0 0) 0x7fa3ec09c0d0 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 >> v1:172.21.15.148:6790/0 conn(0x7fa3ec0a0e90 legacy=0x7fa3ec09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 >> v1:172.21.15.171:6789/0 conn(0x7fa3ec09b170 legacy=0x7fa3ec1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 monclient: found mon.noname-a 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec09d380 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3e4001b40 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f3da6640 5 monclient: authenticate success, global_id 4410 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f3da6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: dump: 2024-09-16T04:06:47.125 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f3da6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3e4001e30 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f3da6640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 1 -- 172.21.15.148:0/2218798905 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3e4004c20 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f3da6640 4 set_mon_vals no callback set 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.102+0000 7fa3f0b40640 10 monclient: dump: 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.127 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 10 monclient: shutdown 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2218798905 >> v1:172.21.15.148:6789/0 conn(0x7fa3ec09b5c0 legacy=0x7fa3ec13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2218798905 shutdown_connections 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2218798905 >> 172.21.15.148:0/2218798905 conn(0x7fa3ec0a5640 msgr2=0x7fa3ec0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2218798905 shutdown_connections 2024-09-16T04:06:47.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2218798905 wait complete. 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.103+0000 7fa3f3da6640 10 monclient: monmap: 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 1 Processor -- start 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 1 -- start start 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient: init 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient(hunting): picked mon.noname-c con 0x7fa3ec09b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient(hunting): picked mon.noname-a con 0x7fa3ec09b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient(hunting): picked mon.noname-b con 0x7fa3ec0a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec1c1970 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec1a6100 con 0x7fa3ec0a0e90 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec1a8bc0 con 0x7fa3ec09b170 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f3da6640 10 monclient(hunting): authenticate will time out at 827.730707s 2024-09-16T04:06:47.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f1b42640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa3ec09b170 0x7fa3ec1c1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35906/0 (socket says 172.21.15.148:35906) 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f1b42640 1 -- 172.21.15.148:0/2500598882 learned_addr learned my addr 172.21.15.148:0/2500598882 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3f1341640 1 --1- 172.21.15.148:0/2500598882 >> v1:172.21.15.148:6789/0 conn(0x7fa3ec09b5c0 0x7fa3ec1cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45780/0 (socket says 172.21.15.148:45780) 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225696227 0 0) 0x7fa3ec1c1970 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): my global_id is 4413 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d0003520 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502522789 0 0) 0x7fa3ec1a8bc0 con 0x7fa3ec09b170 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 10 monclient(hunting): my global_id is 4409 2024-09-16T04:06:47.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.104+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3ec1c1970 con 0x7fa3ec09b170 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 334899067 0 0) 0x7fa3ec1c1970 con 0x7fa3ec09b170 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3ec1a8bc0 con 0x7fa3ec09b170 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 553979327 0 0) 0x7fa3d0003520 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3ec1c1970 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1080556433 0 0) 0x7fa3ec1a6100 con 0x7fa3ec0a0e90 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): my global_id is 4393 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3d0003520 con 0x7fa3ec0a0e90 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3e8001a30 con 0x7fa3ec09b170 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): dump: 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3dc001a80 con 0x7fa3ec09b5c0 2024-09-16T04:06:47.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient(hunting): dump: 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1175632150 0 0) 0x7fa3ec1a8bc0 con 0x7fa3ec09b170 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 >> v1:172.21.15.171:6789/0 conn(0x7fa3ec0a0e90 legacy=0x7fa3ec1cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 >> v1:172.21.15.148:6789/0 conn(0x7fa3ec09b5c0 legacy=0x7fa3ec1cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 monclient: found mon.c 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec0a4290 con 0x7fa3ec09b170 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 5 monclient: authenticate success, global_id 4409 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 10 monclient: _renew_subs 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ec1a68f0 con 0x7fa3ec09b170 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 10 monclient: _renew_subs 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3ec1a7670 con 0x7fa3ec09b170 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3e8001cb0 con 0x7fa3ec09b170 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3e8004c30 con 0x7fa3ec09b170 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e3fff640 4 set_mon_vals no callback set 2024-09-16T04:06:47.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.105+0000 7fa3e27fc640 10 monclient: dump: 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.137 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa3e8010060 con 0x7fa3ec09b170 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3f3da6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3f3da6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3f3da6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d4005080 con 0x7fa3ec09b170 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7fa3e8046e10 con 0x7fa3ec09b170 2024-09-16T04:06:47.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.106+0000 7fa3f1341640 10 monclient: get_auth_request con 0x7fa3d0039070 auth_method 0 2024-09-16T04:06:47.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.110+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa3e8015d90 con 0x7fa3ec09b170 2024-09-16T04:06:47.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.110+0000 7fa3e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.110+0000 7fa3e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:47.155 INFO:tasks.workunit:timeout=3h 2024-09-16T04:06:47.155 INFO:tasks.workunit:cleanup=True 2024-09-16T04:06:47.156 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T04:06:47.214 INFO:tasks.workunit.client.0.smithi171.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.345+0000 7fa3f3da6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.345+0000 7fa3f3da6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.345+0000 7fa3f3da6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.345+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa3d4005370 con 0x7fa3ec09b170 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.346+0000 7fa3e27fc640 1 -- 172.21.15.148:0/2500598882 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3860552270) 0x7fa3e8015690 con 0x7fa3ec09b170 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.346+0000 7fa3e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.346+0000 7fa3e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:47.349 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:06:47.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 >> v1:172.21.15.171:6812/7327273 conn(0x7fa3d0039070 legacy=0x7fa3d003b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 10 monclient: shutdown 2024-09-16T04:06:47.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 >> v1:172.21.15.148:6790/0 conn(0x7fa3ec09b170 legacy=0x7fa3ec1c1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 shutdown_connections 2024-09-16T04:06:47.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 >> 172.21.15.148:0/2500598882 conn(0x7fa3ec0a5640 msgr2=0x7fa3ec0715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 shutdown_connections 2024-09-16T04:06:47.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.348+0000 7fa3f3da6640 1 -- 172.21.15.148:0/2500598882 wait complete. 2024-09-16T04:06:47.361 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:06:47.361 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:06:47.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:06:47.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.459+0000 7f70003d8640 10 monclient: get_monmap_and_config 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.459+0000 7f70003d8640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.459+0000 7f70003d8640 10 monclient: monmap: 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:47.460205+0000 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:47.460205+0000 2024-09-16T04:06:47.461 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.462 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.462 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.462 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:47.462 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:47.462 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.459+0000 7f70003d8640 1 Processor -- start 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.459+0000 7f70003d8640 1 -- start start 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.459+0000 7f70003d8640 10 monclient: init 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff80a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff80a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff80a0e00 con 0x7f6ff813d060 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff80a10e0 con 0x7f6ff80a1b10 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff80a4970 con 0x7f6ff80a2410 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f70003d8640 10 monclient(hunting): authenticate will time out at 828.085631s 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffe975640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6ff813d060 0x7f6ff813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45792/0 (socket says 172.21.15.148:45792) 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffe174640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6ff80a2410 0x7f6ff80a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35916/0 (socket says 172.21.15.148:35916) 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffe975640 1 -- 172.21.15.148:0/1735840293 learned_addr learned my addr 172.21.15.148:0/1735840293 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2459040070 0 0) 0x7f6ff80a4970 con 0x7f6ff80a2410 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffd172640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffd172640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffd172640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffd172640 10 monclient(hunting): my global_id is 4412 2024-09-16T04:06:47.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.460+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe4003520 con 0x7f6ff80a2410 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 307142456 0 0) 0x7f6ff80a0e00 con 0x7f6ff813d060 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): my global_id is 4416 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff80a4970 con 0x7f6ff813d060 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783443007 0 0) 0x7f6ff80a10e0 con 0x7f6ff80a1b10 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient(hunting): my global_id is 4396 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff80a0e00 con 0x7f6ff80a1b10 2024-09-16T04:06:47.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2149920226 0 0) 0x7f6fe4003520 con 0x7f6ff80a2410 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 >> v1:172.21.15.171:6789/0 conn(0x7f6ff80a1b10 legacy=0x7f6ff813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 >> v1:172.21.15.148:6789/0 conn(0x7f6ff813d060 legacy=0x7f6ff813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 monclient: found mon.noname-b 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff806dfd0 con 0x7f6ff80a2410 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 5 monclient: authenticate success, global_id 4412 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff4001540 con 0x7f6ff80a2410 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: dump: 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ff40017e0 con 0x7f6ff80a2410 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 1 -- 172.21.15.148:0/1735840293 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff4001bf0 con 0x7f6ff80a2410 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 4 set_mon_vals no callback set 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f6ffd172640 10 monclient: dump: 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.477 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 10 monclient: shutdown 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 1 -- 172.21.15.148:0/1735840293 >> v1:172.21.15.148:6790/0 conn(0x7f6ff80a2410 legacy=0x7f6ff80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 1 -- 172.21.15.148:0/1735840293 shutdown_connections 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 1 -- 172.21.15.148:0/1735840293 >> 172.21.15.148:0/1735840293 conn(0x7f6ff80a97b0 msgr2=0x7f6ff80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 1 -- 172.21.15.148:0/1735840293 shutdown_connections 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.461+0000 7f70003d8640 1 -- 172.21.15.148:0/1735840293 wait complete. 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.462+0000 7f70003d8640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.478 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.462+0000 7f70003d8640 10 monclient: monmap: 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.462+0000 7f70003d8640 1 Processor -- start 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.462+0000 7f70003d8640 1 -- start start 2024-09-16T04:06:47.479 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.462+0000 7f70003d8640 10 monclient: init 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff80a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff809d7b0 con 0x7f6ff80a2410 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff809f150 con 0x7f6ff80a1b10 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff813cb20 con 0x7f6ff813d060 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f70003d8640 10 monclient(hunting): authenticate will time out at 828.088356s 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6ffd973640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6ff80a2410 0x7f6ff809d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45802/0 (socket says 172.21.15.148:45802) 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6ffe975640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6ff813d060 0x7f6ff81d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35924/0 (socket says 172.21.15.148:35924) 2024-09-16T04:06:47.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6ffd973640 1 -- 172.21.15.148:0/3640919466 learned_addr learned my addr 172.21.15.148:0/3640919466 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573296468 0 0) 0x7f6ff813cb20 con 0x7f6ff813d060 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): my global_id is 4415 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fdc003520 con 0x7f6ff813d060 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507855868 0 0) 0x7f6ff809d7b0 con 0x7f6ff80a2410 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): my global_id is 4419 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff813cb20 con 0x7f6ff80a2410 2024-09-16T04:06:47.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4033543839 0 0) 0x7f6ff809f150 con 0x7f6ff80a1b10 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): my global_id is 4399 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff809d7b0 con 0x7f6ff80a1b10 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 68264020 0 0) 0x7f6fdc003520 con 0x7f6ff813d060 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff809f150 con 0x7f6ff813d060 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff0001610 con 0x7f6ff813d060 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): dump: 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.482 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3948317978 0 0) 0x7f6ff813cb20 con 0x7f6ff80a2410 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fdc003520 con 0x7f6ff80a2410 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fe8001870 con 0x7f6ff80a2410 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): dump: 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 838269562 0 0) 0x7f6ff809d7b0 con 0x7f6ff80a1b10 2024-09-16T04:06:47.484 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ff813cb20 con 0x7f6ff80a1b10 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff4001b30 con 0x7f6ff80a1b10 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.463+0000 7f6feeffd640 10 monclient(hunting): dump: 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.485 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1132597343 0 0) 0x7f6ff809f150 con 0x7f6ff813d060 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.171:6789/0 conn(0x7f6ff80a1b10 legacy=0x7f6ff809c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.148:6789/0 conn(0x7f6ff80a2410 legacy=0x7f6ff809d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 monclient: found mon.c 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff80a0130 con 0x7f6ff813d060 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 5 monclient: authenticate success, global_id 4415 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 10 monclient: _renew_subs 2024-09-16T04:06:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff8137660 con 0x7f6ff813d060 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 10 monclient: _renew_subs 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ff809eed0 con 0x7f6ff813d060 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ff0004b60 con 0x7f6ff813d060 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff0004eb0 con 0x7f6ff813d060 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6ffc971640 4 set_mon_vals no callback set 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f6feeffd640 10 monclient: dump: 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.487 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.464+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fe00065c0 con 0x7f6fe0002860 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6ff0010140 con 0x7f6ff813d060 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f6ff0046e60 con 0x7f6ff813d060 2024-09-16T04:06:47.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480332511 0 0) 0x7f6fe00065c0 con 0x7f6fe0002860 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient(hunting): my global_id is 4422 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff809f150 con 0x7f6fe0002860 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6ffd973640 10 monclient: get_auth_request con 0x7f6fdc0391c0 auth_method 0 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2500706127 0 0) 0x7f6ff809f150 con 0x7f6fe0002860 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe00065c0 con 0x7f6fe0002860 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff4004a20 con 0x7f6fe0002860 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 10 monclient: dump: 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.489 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4048297541 0 0) 0x7f6fe00065c0 con 0x7f6fe0002860 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.465+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fe000a150 con 0x7f6fe0002860 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.466+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6ff4004ab0 con 0x7f6fe0002860 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.466+0000 7f6feeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.466+0000 7f6feeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:47.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.466+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.148:6789/0 conn(0x7f6fe0002860 legacy=0x7f6fe0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.466+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.148:6789/0 conn(0x7f6fe0002860 legacy=0x7f6fe0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:47.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f70003d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fe0009e40 con 0x7f6fe0009100 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2054494496 0 0) 0x7f6fe0009e40 con 0x7f6fe0009100 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient(hunting): my global_id is 4425 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe00065c0 con 0x7f6fe0009100 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3885829834 0 0) 0x7f6fe00065c0 con 0x7f6fe0009100 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe0009e40 con 0x7f6fe0009100 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ff4005080 con 0x7f6fe0009100 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient: dump: 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 166337582 0 0) 0x7f6fe0009e40 con 0x7f6fe0009100 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fe000c3d0 con 0x7f6fe0009100 2024-09-16T04:06:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4145 (unknown 0 0 458118968) 0x7f6ff4005300 con 0x7f6fe0009100 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.148:6789/0 conn(0x7f6fe0009100 legacy=0x7f6fe0009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.472+0000 7f6feeffd640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.148:6789/0 conn(0x7f6fe0009100 legacy=0x7f6fe0009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 8, 2024-09-16T04:06:47.495 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 31551, 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 26, 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.496 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.497 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:06:47.498 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:06:47.499 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.500 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.501 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.502 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.503 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:06:47.504 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.505 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.506 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.507 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.507 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:06:47.507 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.171:6812/7327273 conn(0x7f6fdc0391c0 legacy=0x7f6fdc03b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 10 monclient: shutdown 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 >> v1:172.21.15.148:6790/0 conn(0x7f6ff813d060 legacy=0x7f6ff81d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f6ffe975640 1 -- 172.21.15.148:0/3640919466 reap_dead start 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f6ffe975640 1 -- 172.21.15.148:0/3640919466 reap_dead start 2024-09-16T04:06:47.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 shutdown_connections 2024-09-16T04:06:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 >> 172.21.15.148:0/3640919466 conn(0x7f6ff80a97b0 msgr2=0x7f6ff80a8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 shutdown_connections 2024-09-16T04:06:47.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.473+0000 7f70003d8640 1 -- 172.21.15.148:0/3640919466 wait complete. 2024-09-16T04:06:47.509 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:47.509 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:06:47.509 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.608+0000 7fa81654b640 10 monclient: get_monmap_and_config 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.608+0000 7fa81654b640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.608+0000 7fa81654b640 10 monclient: monmap: 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:47.609250+0000 2024-09-16T04:06:47.610 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:47.609250+0000 2024-09-16T04:06:47.611 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.611 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.611 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.611 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:47.611 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:47.611 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 1 Processor -- start 2024-09-16T04:06:47.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 1 -- start start 2024-09-16T04:06:47.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient: init 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient(hunting): picked mon.noname-b con 0x7fa81006de70 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient(hunting): picked mon.noname-a con 0x7fa8100a3e20 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient(hunting): picked mon.noname-c con 0x7fa81013d1b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81006e8f0 con 0x7fa8100a3e20 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8100a0840 con 0x7fa81013d1b0 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8100a4d70 con 0x7fa81006de70 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa81654b640 10 monclient(hunting): authenticate will time out at 828.234624s 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa80f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa8100a3e20 0x7fa81013ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45844/0 (socket says 172.21.15.148:45844) 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa80ffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa81006de70 0x7fa8100a2610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35932/0 (socket says 172.21.15.148:35932) 2024-09-16T04:06:47.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.609+0000 7fa80f7fe640 1 -- 172.21.15.148:0/2606324824 learned_addr learned my addr 172.21.15.148:0/2606324824 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322542425 0 0) 0x7fa81006e8f0 con 0x7fa8100a3e20 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): my global_id is 4428 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7f8003520 con 0x7fa8100a3e20 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611512023 0 0) 0x7fa8100a4d70 con 0x7fa81006de70 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): my global_id is 4418 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa81006e8f0 con 0x7fa81006de70 2024-09-16T04:06:47.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881003299 0 0) 0x7fa8100a0840 con 0x7fa81013d1b0 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient(hunting): my global_id is 4402 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8100a4d70 con 0x7fa81013d1b0 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 171848661 0 0) 0x7fa7f8003520 con 0x7fa8100a3e20 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 >> v1:172.21.15.148:6790/0 conn(0x7fa81006de70 legacy=0x7fa8100a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 >> v1:172.21.15.171:6789/0 conn(0x7fa81013d1b0 legacy=0x7fa81013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 monclient: found mon.noname-a 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8100a2d20 con 0x7fa8100a3e20 2024-09-16T04:06:47.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa804001610 con 0x7fa8100a3e20 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 5 monclient: authenticate success, global_id 4428 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: dump: 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.625 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.626 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.626 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.626 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.626 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa804001930 con 0x7fa8100a3e20 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 1 -- 172.21.15.148:0/2606324824 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa804001d40 con 0x7fa8100a3e20 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 4 set_mon_vals no callback set 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa80effd640 10 monclient: dump: 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.627 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 10 monclient: shutdown 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 1 -- 172.21.15.148:0/2606324824 >> v1:172.21.15.148:6789/0 conn(0x7fa8100a3e20 legacy=0x7fa81013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 1 -- 172.21.15.148:0/2606324824 shutdown_connections 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 1 -- 172.21.15.148:0/2606324824 >> 172.21.15.148:0/2606324824 conn(0x7fa8100a5a40 msgr2=0x7fa8100a5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.610+0000 7fa81654b640 1 -- 172.21.15.148:0/2606324824 shutdown_connections 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.611+0000 7fa81654b640 1 -- 172.21.15.148:0/2606324824 wait complete. 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.611+0000 7fa81654b640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.611+0000 7fa81654b640 10 monclient: monmap: 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:47.629 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.611+0000 7fa81654b640 1 Processor -- start 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 1 -- start start 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient: init 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient(hunting): picked mon.noname-b con 0x7fa81006de70 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8100a3e20 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient(hunting): picked mon.noname-a con 0x7fa81013d1b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8100a80c0 con 0x7fa81013d1b0 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8100a0130 con 0x7fa81006de70 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81013af30 con 0x7fa8100a3e20 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa81654b640 10 monclient(hunting): authenticate will time out at 828.237461s 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80f7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa8100a3e20 0x7fa8101cfc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35942/0 (socket says 172.21.15.148:35942) 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80f7fe640 1 -- 172.21.15.148:0/3850653792 learned_addr learned my addr 172.21.15.148:0/3850653792 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3681670136 0 0) 0x7fa8100a80c0 con 0x7fa81013d1b0 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): my global_id is 4431 2024-09-16T04:06:47.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7fc003520 con 0x7fa81013d1b0 2024-09-16T04:06:47.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1437818669 0 0) 0x7fa81013af30 con 0x7fa8100a3e20 2024-09-16T04:06:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): my global_id is 4421 2024-09-16T04:06:47.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8100a80c0 con 0x7fa8100a3e20 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671730907 0 0) 0x7fa8100a0130 con 0x7fa81006de70 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): my global_id is 4405 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa81013af30 con 0x7fa81006de70 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3431320673 0 0) 0x7fa7fc003520 con 0x7fa81013d1b0 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8100a0130 con 0x7fa81013d1b0 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa808001cd0 con 0x7fa81013d1b0 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.612+0000 7fa80cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): dump: 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3987184121 0 0) 0x7fa8100a80c0 con 0x7fa8100a3e20 2024-09-16T04:06:47.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7fc003520 con 0x7fa8100a3e20 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa8040047e0 con 0x7fa8100a3e20 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): dump: 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.636 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1583397296 0 0) 0x7fa81013af30 con 0x7fa81006de70 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8100a80c0 con 0x7fa81006de70 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa800001870 con 0x7fa81006de70 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient(hunting): dump: 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.637 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2762143792 0 0) 0x7fa8100a0130 con 0x7fa81013d1b0 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.148:6790/0 conn(0x7fa8100a3e20 legacy=0x7fa8101cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.171:6789/0 conn(0x7fa81006de70 legacy=0x7fa81013c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 monclient: found mon.a 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81013b7d0 con 0x7fa81013d1b0 2024-09-16T04:06:47.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 5 monclient: authenticate success, global_id 4431 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 10 monclient: _renew_subs 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa81013c8e0 con 0x7fa81013d1b0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 10 monclient: _renew_subs 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa810137450 con 0x7fa81013d1b0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa808003970 con 0x7fa81013d1b0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa808003cc0 con 0x7fa81013d1b0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 10 monclient: dump: 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80e7fc640 4 set_mon_vals no callback set 2024-09-16T04:06:47.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.613+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa80800ef50 con 0x7fa81013d1b0 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7fa808045b40 con 0x7fa81013d1b0 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa81654b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7f00065c0 con 0x7fa7f0002860 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80ffff640 10 monclient: get_auth_request con 0x7fa7fc039210 auth_method 0 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3049316943 0 0) 0x7fa7f00065c0 con 0x7fa7f0002860 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient(hunting): my global_id is 4424 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8100a0130 con 0x7fa7f0002860 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 215716191 0 0) 0x7fa8100a0130 con 0x7fa7f0002860 2024-09-16T04:06:47.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7f00065c0 con 0x7fa7f0002860 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa804004980 con 0x7fa7f0002860 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 10 monclient: dump: 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3529937074 0 0) 0x7fa7f00065c0 con 0x7fa7f0002860 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.614+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7f000a150 con 0x7fa7f0002860 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.615+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa804004cb0 con 0x7fa7f0002860 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.615+0000 7fa80cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.615+0000 7fa80cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.615+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.148:6790/0 conn(0x7fa7f0002860 legacy=0x7fa7f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.615+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.148:6790/0 conn(0x7fa7f0002860 legacy=0x7fa7f0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa81654b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa7f0009e40 con 0x7fa7f0009100 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341715417 0 0) 0x7fa7f0009e40 con 0x7fa7f0009100 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient(hunting): my global_id is 4427 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7f00065c0 con 0x7fa7f0009100 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2452290439 0 0) 0x7fa7f00065c0 con 0x7fa7f0009100 2024-09-16T04:06:47.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7f0009e40 con 0x7fa7f0009100 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa804005380 con 0x7fa7f0009100 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient: dump: 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.647 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 220759191 0 0) 0x7fa7f0009e40 con 0x7fa7f0009100 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7f000c3d0 con 0x7fa7f0009100 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4143 (unknown 0 0 4158535108) 0x7fa804006680 con 0x7fa7f0009100 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.148:6790/0 conn(0x7fa7f0009100 legacy=0x7fa7f0009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.621+0000 7fa80cff9640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.148:6790/0 conn(0x7fa7f0009100 legacy=0x7fa7f0009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:06:47.648 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 8, 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 31700, 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 26, 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:06:47.649 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:47.650 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.651 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:47.652 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.653 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.654 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:06:47.655 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.656 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.657 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.658 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.659 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:06:47.660 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:06:47.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.171:6812/7327273 conn(0x7fa7fc039210 legacy=0x7fa7fc03b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 10 monclient: shutdown 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 >> v1:172.21.15.148:6789/0 conn(0x7fa81013d1b0 legacy=0x7fa8101d3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa814ae8640 1 -- 172.21.15.148:0/3850653792 reap_dead start 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa814ae8640 1 -- 172.21.15.148:0/3850653792 reap_dead start 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 shutdown_connections 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 >> 172.21.15.148:0/3850653792 conn(0x7fa8100a5a40 msgr2=0x7fa8100a7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 shutdown_connections 2024-09-16T04:06:47.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.622+0000 7fa81654b640 1 -- 172.21.15.148:0/3850653792 wait complete. 2024-09-16T04:06:47.662 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:47.662 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:06:47.663 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:06:47.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.759+0000 7fc46c44c640 10 monclient: get_monmap_and_config 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.759+0000 7fc46c44c640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.759+0000 7fc46c44c640 10 monclient: monmap: 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:47.760728+0000 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:47.760728+0000 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:47.762 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.760+0000 7fc46c44c640 1 Processor -- start 2024-09-16T04:06:47.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.760+0000 7fc46c44c640 1 -- start start 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.760+0000 7fc46c44c640 10 monclient: init 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient(hunting): picked mon.noname-b con 0x7fc4640a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient(hunting): picked mon.noname-c con 0x7fc46409b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient(hunting): picked mon.noname-a con 0x7fc46409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46409e040 con 0x7fc46409b5c0 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46409d660 con 0x7fc46409b170 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46409c0d0 con 0x7fc4640a0e90 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient(hunting): authenticate will time out at 828.386097s 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46a9e9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc46409b5c0 0x7fc46413bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45862/0 (socket says 172.21.15.148:45862) 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46a1e8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc4640a0e90 0x7fc46409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35984/0 (socket says 172.21.15.148:35984) 2024-09-16T04:06:47.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46a9e9640 1 -- 172.21.15.148:0/3336800828 learned_addr learned my addr 172.21.15.148:0/3336800828 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955097965 0 0) 0x7fc45c001490 con 0x7fc46409b5c0 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): my global_id is 4434 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc450003520 con 0x7fc46409b5c0 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3454449303 0 0) 0x7fc46409c0d0 con 0x7fc4640a0e90 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): my global_id is 4430 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc45c001490 con 0x7fc4640a0e90 2024-09-16T04:06:47.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2787950148 0 0) 0x7fc454001110 con 0x7fc46409b170 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient(hunting): my global_id is 4408 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc46409c0d0 con 0x7fc46409b170 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611899444 0 0) 0x7fc45c001490 con 0x7fc4640a0e90 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 >> v1:172.21.15.171:6789/0 conn(0x7fc46409b170 legacy=0x7fc4641383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 >> v1:172.21.15.148:6789/0 conn(0x7fc46409b5c0 legacy=0x7fc46413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 monclient: found mon.noname-b 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46409d380 con 0x7fc4640a0e90 2024-09-16T04:06:47.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc4600015f0 con 0x7fc4640a0e90 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 5 monclient: authenticate success, global_id 4430 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: dump: 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.774 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc460001860 con 0x7fc4640a0e90 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 1 -- 172.21.15.148:0/3336800828 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc460001c70 con 0x7fc4640a0e90 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc46c44c640 4 set_mon_vals no callback set 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.761+0000 7fc4691e6640 10 monclient: dump: 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 10 monclient: shutdown 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 1 -- 172.21.15.148:0/3336800828 >> v1:172.21.15.148:6790/0 conn(0x7fc4640a0e90 legacy=0x7fc46409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 1 -- 172.21.15.148:0/3336800828 shutdown_connections 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 1 -- 172.21.15.148:0/3336800828 >> 172.21.15.148:0/3336800828 conn(0x7fc4640a5640 msgr2=0x7fc4640a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 1 -- 172.21.15.148:0/3336800828 shutdown_connections 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.762+0000 7fc46c44c640 1 -- 172.21.15.148:0/3336800828 wait complete. 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient: monmap: 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.777 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 1 Processor -- start 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 1 -- start start 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient: init 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient(hunting): picked mon.noname-c con 0x7fc46409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient(hunting): picked mon.noname-a con 0x7fc46409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient(hunting): picked mon.noname-b con 0x7fc4640a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4640a41c0 con 0x7fc46409b5c0 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc464125fa0 con 0x7fc4640a0e90 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc464140400 con 0x7fc46409b170 2024-09-16T04:06:47.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46c44c640 10 monclient(hunting): authenticate will time out at 828.388924s 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46a1e8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc46409b170 0x7fc46413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35986/0 (socket says 172.21.15.148:35986) 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc4699e7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc46409b5c0 0x7fc46413fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:45874/0 (socket says 172.21.15.148:45874) 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.763+0000 7fc46a1e8640 1 -- 172.21.15.148:0/3595595834 learned_addr learned my addr 172.21.15.148:0/3595595834 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940288386 0 0) 0x7fc464140400 con 0x7fc46409b170 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): my global_id is 4433 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc448003520 con 0x7fc46409b170 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325392562 0 0) 0x7fc46409c0d0 con 0x7fc46409b5c0 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): my global_id is 4437 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc464140400 con 0x7fc46409b5c0 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946143592 0 0) 0x7fc464125fa0 con 0x7fc4640a0e90 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): my global_id is 4411 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc46409c0d0 con 0x7fc4640a0e90 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 680888580 0 0) 0x7fc448003520 con 0x7fc46409b170 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc464125fa0 con 0x7fc46409b170 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc460001c70 con 0x7fc46409b170 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): dump: 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:47.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4105875020 0 0) 0x7fc464140400 con 0x7fc46409b5c0 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc448003520 con 0x7fc46409b5c0 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc454001a80 con 0x7fc46409b5c0 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): dump: 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.782 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1531804637 0 0) 0x7fc46409c0d0 con 0x7fc4640a0e90 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc464140400 con 0x7fc4640a0e90 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc45c001630 con 0x7fc4640a0e90 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient(hunting): dump: 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.783 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3902244376 0 0) 0x7fc464125fa0 con 0x7fc46409b170 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.171:6789/0 conn(0x7fc4640a0e90 legacy=0x7fc4641d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.148:6789/0 conn(0x7fc46409b5c0 legacy=0x7fc46413fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 monclient: found mon.c 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc464126560 con 0x7fc46409b170 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 5 monclient: authenticate success, global_id 4433 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 10 monclient: _renew_subs 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4640a4580 con 0x7fc46409b170 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 10 monclient: _renew_subs 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc464126280 con 0x7fc46409b170 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc460001860 con 0x7fc46409b170 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc460004d10 con 0x7fc46409b170 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc4689e5640 4 set_mon_vals no callback set 2024-09-16T04:06:47.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.764+0000 7fc45affd640 10 monclient: dump: 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc46c44c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc460004f30 con 0x7fc46409b170 2024-09-16T04:06:47.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc44c0065c0 con 0x7fc44c002860 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7fc460046c50 con 0x7fc46409b170 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc46a9e9640 10 monclient: get_auth_request con 0x7fc448039160 auth_method 0 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656919766 0 0) 0x7fc44c0065c0 con 0x7fc44c002860 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 10 monclient(hunting): my global_id is 4414 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.765+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc464125fa0 con 0x7fc44c002860 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3434826206 0 0) 0x7fc464125fa0 con 0x7fc44c002860 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc44c0065c0 con 0x7fc44c002860 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc4540039a0 con 0x7fc44c002860 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 10 monclient: dump: 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3164993472 0 0) 0x7fc44c0065c0 con 0x7fc44c002860 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc44c00a150 con 0x7fc44c002860 2024-09-16T04:06:47.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc4540057c0 con 0x7fc44c002860 2024-09-16T04:06:47.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:47.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.171:6789/0 conn(0x7fc44c002860 legacy=0x7fc44c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.766+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.171:6789/0 conn(0x7fc44c002860 legacy=0x7fc44c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:47.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.772+0000 7fc46c44c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:47.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.772+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc44c009e20 con 0x7fc44c0090e0 2024-09-16T04:06:47.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130892416 0 0) 0x7fc44c009e20 con 0x7fc44c0090e0 2024-09-16T04:06:47.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient(hunting): my global_id is 4417 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc44c0065c0 con 0x7fc44c0090e0 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1545214741 0 0) 0x7fc44c0065c0 con 0x7fc44c0090e0 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc44c009e20 con 0x7fc44c0090e0 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc454003fe0 con 0x7fc44c0090e0 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient: dump: 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.791 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2684786091 0 0) 0x7fc44c009e20 con 0x7fc44c0090e0 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc44c00c390 con 0x7fc44c0090e0 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4310 (unknown 0 0 2904822650) 0x7fc454003e40 con 0x7fc44c0090e0 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.171:6789/0 conn(0x7fc44c0090e0 legacy=0x7fc44c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.773+0000 7fc45affd640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.171:6789/0 conn(0x7fc44c0090e0 legacy=0x7fc44c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 8, 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 31811, 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:06:47.793 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 26, 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.794 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.795 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:06:47.796 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:06:47.797 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:06:47.798 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.799 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.800 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:06:47.801 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:06:47.802 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.803 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:06:47.804 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:06:47.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:06:47.805 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:06:47.805 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:06:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.171:6812/7327273 conn(0x7fc448039160 legacy=0x7fc44803b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 10 monclient: shutdown 2024-09-16T04:06:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 >> v1:172.21.15.148:6790/0 conn(0x7fc46409b170 legacy=0x7fc46413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46a9e9640 1 -- 172.21.15.148:0/3595595834 reap_dead start 2024-09-16T04:06:47.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46a9e9640 1 -- 172.21.15.148:0/3595595834 reap_dead start 2024-09-16T04:06:47.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 shutdown_connections 2024-09-16T04:06:47.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 >> 172.21.15.148:0/3595595834 conn(0x7fc4640a5640 msgr2=0x7fc4640710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 shutdown_connections 2024-09-16T04:06:47.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.774+0000 7fc46c44c640 1 -- 172.21.15.148:0/3595595834 wait complete. 2024-09-16T04:06:47.807 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:47.807 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:06:47.807 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:06:47.808 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:06:47.808 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:06:47.808 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:06:47.808 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:06:47.818 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:06:47.818 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:06:47.819 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.953+0000 7f1946f54640 10 monclient: get_monmap_and_config 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.953+0000 7f1946f54640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.953+0000 7f1946f54640 10 monclient: monmap: 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:47.954204+0000 2024-09-16T04:06:47.955 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:47.954204+0000 2024-09-16T04:06:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:47.956 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:47.956 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.953+0000 7f1946f54640 1 Processor -- start 2024-09-16T04:06:47.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 1 -- start start 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient: init 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient(hunting): picked mon.noname-c con 0x7f19400a3a90 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient(hunting): picked mon.noname-b con 0x7f19400a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient(hunting): picked mon.noname-a con 0x7f194013d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19400a0590 con 0x7f194013d3c0 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19400a1270 con 0x7f19400a3a90 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19400a4fd0 con 0x7f19400a4580 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f1946f54640 10 monclient(hunting): authenticate will time out at 828.579557s 2024-09-16T04:06:47.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f19454f1640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f194013d3c0 legacy=0x7f194013f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193ffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f19400a4580 0x7f194013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35996/0 (socket says 172.21.15.148:35996) 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193ffff640 1 -- 172.21.15.148:0/2939486240 learned_addr learned my addr 172.21.15.148:0/2939486240 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350167674 0 0) 0x7f19400a4fd0 con 0x7f19400a4580 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193f7fe640 10 monclient(hunting): my global_id is 4436 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.954+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1930003520 con 0x7f19400a4580 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3772704159 0 0) 0x7f1930003520 con 0x7f19400a4580 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 >> v1:172.21.15.171:6789/0 conn(0x7f19400a3a90 legacy=0x7f19400a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 >> v1:172.21.15.148:6789/0 conn(0x7f194013d3c0 legacy=0x7f194013f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 monclient: found mon.noname-b 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194006e170 con 0x7f19400a4580 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f192c001610 con 0x7f19400a4580 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 5 monclient: authenticate success, global_id 4436 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: dump: 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:47.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f192c001970 con 0x7f19400a4580 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 1 -- 172.21.15.148:0/2939486240 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f192c0048a0 con 0x7f19400a4580 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 4 set_mon_vals no callback set 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f193f7fe640 10 monclient: dump: 2024-09-16T04:06:47.969 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.969 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.969 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.969 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 10 monclient: shutdown 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:47.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 1 -- 172.21.15.148:0/2939486240 >> v1:172.21.15.148:6790/0 conn(0x7f19400a4580 legacy=0x7f194013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 1 -- 172.21.15.148:0/2939486240 shutdown_connections 2024-09-16T04:06:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 1 -- 172.21.15.148:0/2939486240 >> 172.21.15.148:0/2939486240 conn(0x7f19400a9a40 msgr2=0x7f19400a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 1 -- 172.21.15.148:0/2939486240 shutdown_connections 2024-09-16T04:06:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.955+0000 7f1946f54640 1 -- 172.21.15.148:0/2939486240 wait complete. 2024-09-16T04:06:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.956+0000 7f1946f54640 10 monclient: build_initial_monmap 2024-09-16T04:06:47.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.956+0000 7f1946f54640 10 monclient: monmap: 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:47.972 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.956+0000 7f1946f54640 1 Processor -- start 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.956+0000 7f1946f54640 1 -- start start 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.956+0000 7f1946f54640 10 monclient: init 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient(hunting): picked mon.noname-c con 0x7f19400a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient(hunting): picked mon.noname-b con 0x7f19400a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient(hunting): picked mon.noname-a con 0x7f194013d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:47.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1940139030 con 0x7f194013d3c0 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19400a80c0 con 0x7f19400a4580 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1940138370 con 0x7f19400a3a90 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient(hunting): authenticate will time out at 828.582245s 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f19454f1640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f194013d3c0 legacy=0x7f194009dac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1944cf0640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f19400a3a90 0x7f194009cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:36008/0 (socket says 172.21.15.148:36008) 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1944cf0640 1 -- 172.21.15.148:0/220498867 learned_addr learned my addr 172.21.15.148:0/220498867 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445683745 0 0) 0x7f1940138370 con 0x7f19400a3a90 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): my global_id is 4439 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1924003520 con 0x7f19400a3a90 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981562743 0 0) 0x7f19400a80c0 con 0x7f19400a4580 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): my global_id is 4423 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1940138370 con 0x7f19400a4580 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 807269067 0 0) 0x7f1924003520 con 0x7f19400a3a90 2024-09-16T04:06:47.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19400a80c0 con 0x7f19400a3a90 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1934001930 con 0x7f19400a3a90 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): dump: 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.976 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 444492092 0 0) 0x7f1940138370 con 0x7f19400a4580 2024-09-16T04:06:47.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1924003520 con 0x7f19400a4580 2024-09-16T04:06:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f192c001430 con 0x7f19400a4580 2024-09-16T04:06:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:47.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient(hunting): dump: 2024-09-16T04:06:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.979 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.980 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.980 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.980 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.980 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.980 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1766009160 0 0) 0x7f19400a80c0 con 0x7f19400a3a90 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 >> v1:172.21.15.171:6789/0 conn(0x7f19400a4580 legacy=0x7f194009d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 >> v1:172.21.15.148:6789/0 conn(0x7f194013d3c0 legacy=0x7f194009dac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 monclient: found mon.c 2024-09-16T04:06:47.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1940138650 con 0x7f19400a3a90 2024-09-16T04:06:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f193d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:47.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 5 monclient: authenticate success, global_id 4439 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient: _renew_subs 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f194009f2a0 con 0x7f19400a3a90 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient: _renew_subs 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.957+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f194009ea20 con 0x7f19400a3a90 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1934001d00 con 0x7f19400a3a90 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1934003a60 con 0x7f19400a3a90 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 10 monclient: dump: 2024-09-16T04:06:47.983 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193effd640 4 set_mon_vals no callback set 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1934003c00 con 0x7f19400a3a90 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.958+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f1934046d50 con 0x7f19400a3a90 2024-09-16T04:06:47.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.959+0000 7f1946f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.959+0000 7f1946f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.959+0000 7f1946f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.959+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1940138a30 con 0x7f19400a3a90 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.959+0000 7f193ffff640 10 monclient: get_auth_request con 0x7f1924038ef0 auth_method 0 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.963+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1934015d10 con 0x7f19400a3a90 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.963+0000 7f193d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:47.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:47.963+0000 7f193d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f1946f54640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:06:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f1946f54640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f1946f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1940062790 con 0x7f19400a3a90 2024-09-16T04:06:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f193d7fa640 1 -- 172.21.15.148:0/220498867 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 1863827454) 0x7f19340199c0 con 0x7f19400a3a90 2024-09-16T04:06:48.197 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f193d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:48.198 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.195+0000 7f193d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:48.198 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 >> v1:172.21.15.171:6812/7327273 conn(0x7f1924038ef0 legacy=0x7f192403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 10 monclient: shutdown 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 >> v1:172.21.15.148:6790/0 conn(0x7f19400a3a90 legacy=0x7f194009cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 shutdown_connections 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 >> 172.21.15.148:0/220498867 conn(0x7f19400a9a40 msgr2=0x7f19400a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 shutdown_connections 2024-09-16T04:06:48.200 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:48.197+0000 7f1946f54640 1 -- 172.21.15.148:0/220498867 wait complete. 2024-09-16T04:06:51.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:51.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.318+0000 7f1a02df8640 10 monclient: get_monmap_and_config 2024-09-16T04:06:51.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.318+0000 7f1a02df8640 10 monclient: build_initial_monmap 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.318+0000 7f1a02df8640 10 monclient: monmap: 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:51.319702+0000 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:51.319702+0000 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:51.321 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.319+0000 7f1a02df8640 1 Processor -- start 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.319+0000 7f1a02df8640 1 -- start start 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.319+0000 7f1a02df8640 10 monclient: init 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient(hunting): picked mon.noname-c con 0x7f19fc0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient(hunting): picked mon.noname-a con 0x7f19fc0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient(hunting): picked mon.noname-b con 0x7f19fc13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc0a0e00 con 0x7f19fc0a1b10 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc0a10e0 con 0x7f19fc0a2410 2024-09-16T04:06:51.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fbfff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f19fc0a1b10 legacy=0x7f19fc13b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc0a4970 con 0x7f19fc13d060 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient(hunting): authenticate will time out at 831.945280s 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a01395640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f19fc13d060 0x7f19fc13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:36024/0 (socket says 172.21.15.148:36024) 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a01395640 1 -- 172.21.15.148:0/767741300 learned_addr learned my addr 172.21.15.148:0/767741300 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621849070 0 0) 0x7f19fc0a4970 con 0x7f19fc13d060 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient(hunting): my global_id is 4442 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19e4003520 con 0x7f19fc13d060 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1104174157 0 0) 0x7f19e4003520 con 0x7f19fc13d060 2024-09-16T04:06:51.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 >> v1:172.21.15.171:6789/0 conn(0x7f19fc0a2410 legacy=0x7f19fc0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 >> v1:172.21.15.148:6789/0 conn(0x7f19fc0a1b10 legacy=0x7f19fc13b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 monclient: found mon.noname-b 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc06dfd0 con 0x7f19fc13d060 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19f4001990 con 0x7f19fc13d060 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:51.331 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: dump: 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 5 monclient: authenticate success, global_id 4442 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:51.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f19f4001cf0 con 0x7f19fc13d060 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 1 -- 172.21.15.148:0/767741300 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19f40020c0 con 0x7f19fc13d060 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f1a02df8640 4 set_mon_vals no callback set 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.320+0000 7f19fb7fe640 10 monclient: dump: 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.333 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 10 monclient: shutdown 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 1 -- 172.21.15.148:0/767741300 >> v1:172.21.15.148:6790/0 conn(0x7f19fc13d060 legacy=0x7f19fc13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 1 -- 172.21.15.148:0/767741300 shutdown_connections 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 1 -- 172.21.15.148:0/767741300 >> 172.21.15.148:0/767741300 conn(0x7f19fc0a97b0 msgr2=0x7f19fc0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 1 -- 172.21.15.148:0/767741300 shutdown_connections 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.321+0000 7f1a02df8640 1 -- 172.21.15.148:0/767741300 wait complete. 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.322+0000 7f1a02df8640 10 monclient: build_initial_monmap 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.322+0000 7f1a02df8640 10 monclient: monmap: 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:51.334 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.322+0000 7f1a02df8640 1 Processor -- start 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.322+0000 7f1a02df8640 1 -- start start 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.322+0000 7f1a02df8640 10 monclient: init 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:51.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient(hunting): picked mon.noname-a con 0x7f19fc0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient(hunting): picked mon.noname-b con 0x7f19fc0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient(hunting): picked mon.noname-c con 0x7f19fc13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc0a0130 con 0x7f19fc0a1b10 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc13cb70 con 0x7f19fc0a2410 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a00b94640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f19fc0a1b10 legacy=0x7f19fc1cbd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19fc09e740 con 0x7f19fc13d060 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient(hunting): authenticate will time out at 831.948243s 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a01395640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f19fc13d060 0x7f19fc1d4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:36030/0 (socket says 172.21.15.148:36030) 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a01395640 1 -- 172.21.15.148:0/1639530847 learned_addr learned my addr 172.21.15.148:0/1639530847 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141267588 0 0) 0x7f19fc09e740 con 0x7f19fc13d060 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:51.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): my global_id is 4445 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19e8003550 con 0x7f19fc13d060 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425436693 0 0) 0x7f19fc13cb70 con 0x7f19fc0a2410 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): my global_id is 4429 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19fc09e740 con 0x7f19fc0a2410 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 640438135 0 0) 0x7f19e8003550 con 0x7f19fc13d060 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19fc13cb70 con 0x7f19fc13d060 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19f4001950 con 0x7f19fc13d060 2024-09-16T04:06:51.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): dump: 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:51.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2411601305 0 0) 0x7f19fc09e740 con 0x7f19fc0a2410 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19e8003550 con 0x7f19fc0a2410 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19f00040c0 con 0x7f19fc0a2410 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient(hunting): dump: 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:51.339 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 106298236 0 0) 0x7f19fc13cb70 con 0x7f19fc13d060 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 >> v1:172.21.15.171:6789/0 conn(0x7f19fc0a2410 legacy=0x7f19fc1d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 >> v1:172.21.15.148:6789/0 conn(0x7f19fc0a1b10 legacy=0x7f19fc1cbd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 monclient: found mon.c 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc137a80 con 0x7f19fc13d060 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:51.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f19f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 5 monclient: authenticate success, global_id 4445 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient: _renew_subs 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.323+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f19fc137d60 con 0x7f19fc13d060 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f1a02df8640 10 monclient: _renew_subs 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f1a02df8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f19fc09dc40 con 0x7f19fc13d060 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f19f4001cf0 con 0x7f19fc13d060 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19f4005040 con 0x7f19fc13d060 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19faffd640 4 set_mon_vals no callback set 2024-09-16T04:06:51.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 10 monclient: dump: 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.324+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f19f4010470 con 0x7f19fc13d060 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.325+0000 7f1a02df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:51.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.325+0000 7f1a02df8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.325+0000 7f1a02df8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.325+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19dc005080 con 0x7f19fc13d060 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.325+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f19f4047040 con 0x7f19fc13d060 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.325+0000 7f1a00b94640 10 monclient: get_auth_request con 0x7f19e8038f40 auth_method 0 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.329+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f19f4019bb0 con 0x7f19fc13d060 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.329+0000 7f19f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:51.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.329+0000 7f19f97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:51.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f1a02df8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f1a02df8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f1a02df8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f19dc005370 con 0x7f19fc13d060 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f19f97fa640 1 -- 172.21.15.148:0/1639530847 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 1822024401) 0x7f19f40433c0 con 0x7f19fc13d060 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f19f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.557+0000 7f19f97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:51.560 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.559+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 >> v1:172.21.15.171:6812/7327273 conn(0x7f19e8038f40 legacy=0x7f19e803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.559+0000 7f1a02df8640 10 monclient: shutdown 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.559+0000 7f1a02df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.559+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 >> v1:172.21.15.148:6790/0 conn(0x7f19fc13d060 legacy=0x7f19fc1d4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.559+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 shutdown_connections 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.559+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 >> 172.21.15.148:0/1639530847 conn(0x7f19fc0a97b0 msgr2=0x7f19fc135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.560+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 shutdown_connections 2024-09-16T04:06:51.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:51.560+0000 7f1a02df8640 1 -- 172.21.15.148:0/1639530847 wait complete. 2024-09-16T04:06:54.573 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:54.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.681+0000 7f92f980e640 10 monclient: get_monmap_and_config 2024-09-16T04:06:54.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.681+0000 7f92f980e640 10 monclient: build_initial_monmap 2024-09-16T04:06:54.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.681+0000 7f92f980e640 10 monclient: monmap: 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:06:54.682330+0000 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:06:54.682330+0000 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:06:54.684 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.681+0000 7f92f980e640 1 Processor -- start 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 1 -- start start 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient: init 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient(hunting): picked mon.noname-c con 0x7f92f40733a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:54.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient(hunting): picked mon.noname-b con 0x7f92f4073780 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient(hunting): picked mon.noname-a con 0x7f92f413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f40a3410 con 0x7f92f413d010 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f40a3190 con 0x7f92f40733a0 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f409e4b0 con 0x7f92f4073780 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f980e640 10 monclient(hunting): authenticate will time out at 835.307561s 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f413f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f27fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f92f4073780 0x7f92f413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51742/0 (socket says 172.21.15.148:51742) 2024-09-16T04:06:54.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.682+0000 7f92f27fc640 1 -- 172.21.15.148:0/1460839582 learned_addr learned my addr 172.21.15.148:0/1460839582 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:06:54.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:54.883+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f413f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:55.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:55.284+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f413f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:56.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:56.085+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f413f400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): tick 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6790/0 conn(0x7f92f4073780 legacy=0x7f92f413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.171:6789/0 conn(0x7f92f40733a0 legacy=0x7f92f409ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f413f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f92d40036c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f92d4007350 addr v1:172.21.15.148:6790/0 2024-09-16T04:06:57.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f92f413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:06:57.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f406e700 con 0x7f92f413d010 2024-09-16T04:06:57.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f40a3410 con 0x7f92d40036c0 2024-09-16T04:06:57.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d40108a0 con 0x7f92d4007350 2024-09-16T04:06:57.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f0ff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:57.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.682+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d400d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:57.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:57.883+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d400d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:58.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:58.284+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d400d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:06:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:06:59.085+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d400d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:00.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): tick 2024-09-16T04:07:00.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:07:00.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:07:00.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:07:00.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6790/0 conn(0x7f92d4007350 legacy=0x7f92d40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:00.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.171:6789/0 conn(0x7f92d40036c0 legacy=0x7f92d4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d400d0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f92d400f950 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f92d4010a70 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f92f413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d4011b50 con 0x7f92f413d010 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f406e700 con 0x7f92d400f950 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d4011600 con 0x7f92d4010a70 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f0ff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:00.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.683+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d40097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:00.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:00.884+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d40097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:01.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:01.285+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d40097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.086+0000 7f92f37fe640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d40097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728322995 0 0) 0x7f92d4011600 con 0x7f92d4010a70 2024-09-16T04:07:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient(hunting): my global_id is 4448 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92e0003520 con 0x7f92d4010a70 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2583964258 0 0) 0x7f92e0003520 con 0x7f92d4010a70 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.171:6789/0 conn(0x7f92d400f950 legacy=0x7f92d400fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92d40097e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 monclient: found mon.noname-b 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d40117d0 con 0x7f92d4010a70 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:02.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92dc001510 con 0x7f92d4010a70 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: dump: 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.397 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f980e640 5 monclient: authenticate success, global_id 4448 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f980e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92dc0017a0 con 0x7f92d4010a70 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 1 -- 172.21.15.148:0/1460839582 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92dc001bb0 con 0x7f92d4010a70 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f980e640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f980e640 4 set_mon_vals no callback set 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.391+0000 7f92f1ffb640 10 monclient: dump: 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.398 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 10 monclient: shutdown 2024-09-16T04:07:02.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 1 -- 172.21.15.148:0/1460839582 >> v1:172.21.15.148:6790/0 conn(0x7f92d4010a70 legacy=0x7f92d4010ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 1 -- 172.21.15.148:0/1460839582 shutdown_connections 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 1 -- 172.21.15.148:0/1460839582 >> 172.21.15.148:0/1460839582 conn(0x7f92f40a78b0 msgr2=0x7f92f40a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 1 -- 172.21.15.148:0/1460839582 shutdown_connections 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.392+0000 7f92f980e640 1 -- 172.21.15.148:0/1460839582 wait complete. 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.393+0000 7f92f980e640 10 monclient: build_initial_monmap 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.393+0000 7f92f980e640 10 monclient: monmap: 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:07:02.400 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:02.401 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.401 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:02.401 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:02.401 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:02.401 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 1 Processor -- start 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 1 -- start start 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient: init 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient(hunting): picked mon.noname-b con 0x7f92d400f950 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient(hunting): picked mon.noname-c con 0x7f92d4010a70 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient(hunting): picked mon.noname-a con 0x7f92f413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f40a5290 con 0x7f92f413d010 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f4122220 con 0x7f92d400f950 2024-09-16T04:07:02.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f4122500 con 0x7f92d4010a70 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.394+0000 7f92f980e640 10 monclient(hunting): authenticate will time out at 843.019979s 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f41d3df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f27fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f92d4010a70 0x7f92f41d06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35394/0 (socket says 172.21.15.148:35394) 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f27fc640 1 -- 172.21.15.148:0/1731271974 learned_addr learned my addr 172.21.15.148:0/1731271974 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996481815 0 0) 0x7f92f4122500 con 0x7f92d4010a70 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): my global_id is 4451 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92d4011af0 con 0x7f92d4010a70 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1430565299 0 0) 0x7f92d4011af0 con 0x7f92d4010a70 2024-09-16T04:07:02.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92f4122500 con 0x7f92d4010a70 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92dc001af0 con 0x7f92d4010a70 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): dump: 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4092935337 0 0) 0x7f92f4122500 con 0x7f92d4010a70 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 >> v1:172.21.15.171:6789/0 conn(0x7f92d400f950 legacy=0x7f92f4136a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 >> v1:172.21.15.148:6789/0 conn(0x7f92f413d010 legacy=0x7f92f41d3df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 monclient: found mon.c 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f4121770 con 0x7f92d4010a70 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient: _finish_auth 0 2024-09-16T04:07:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f980e640 5 monclient: authenticate success, global_id 4451 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f980e640 10 monclient: _renew_subs 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f980e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.395+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92f409cca0 con 0x7f92d4010a70 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92f980e640 10 monclient: _renew_subs 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92f980e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f92f4137110 con 0x7f92d4010a70 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92dc004a90 con 0x7f92d4010a70 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92ebfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92dc004de0 con 0x7f92d4010a70 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92f17fa640 4 set_mon_vals no callback set 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.396+0000 7f92ebfff640 10 monclient: dump: 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f92dc010210 con 0x7f92d4010a70 2024-09-16T04:07:02.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f92dc046b90 con 0x7f92d4010a70 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92f980e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92f980e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92f980e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d8005080 con 0x7f92d4010a70 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.397+0000 7f92f2ffd640 10 monclient: get_auth_request con 0x7f92d403a880 auth_method 0 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.402+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f92dc015bf0 con 0x7f92d4010a70 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.402+0000 7f92ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.402+0000 7f92ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92f980e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92f980e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92f980e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f92d8005370 con 0x7f92d4010a70 2024-09-16T04:07:02.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92ebfff640 1 -- 172.21.15.148:0/1731271974 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2560167945) 0x7f92dc0154f0 con 0x7f92d4010a70 2024-09-16T04:07:02.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:02.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.636+0000 7f92ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:02.639 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 >> v1:172.21.15.171:6812/7327273 conn(0x7f92d403a880 legacy=0x7f92d403cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 10 monclient: shutdown 2024-09-16T04:07:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 >> v1:172.21.15.148:6790/0 conn(0x7f92d4010a70 legacy=0x7f92f41d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 shutdown_connections 2024-09-16T04:07:02.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 >> 172.21.15.148:0/1731271974 conn(0x7f92f40a78b0 msgr2=0x7f92f40a0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:02.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 shutdown_connections 2024-09-16T04:07:02.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.638+0000 7f92f980e640 1 -- 172.21.15.148:0/1731271974 wait complete. 2024-09-16T04:07:02.651 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:07:02.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:07:02.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:07:02.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.747+0000 7f1bfc05e640 10 monclient: get_monmap_and_config 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.747+0000 7f1bfc05e640 10 monclient: build_initial_monmap 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.747+0000 7f1bfc05e640 10 monclient: monmap: 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:07:02.748410+0000 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:07:02.748410+0000 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:02.750 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 1 Processor -- start 2024-09-16T04:07:02.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 1 -- start start 2024-09-16T04:07:02.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient: init 2024-09-16T04:07:02.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf40a1010 con 0x7f1bf409cc30 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf409f030 con 0x7f1bf409bb70 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf409dc60 con 0x7f1bf409c660 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfa5fb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f1bf409cc30 legacy=0x7f1bf413c090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bfc05e640 10 monclient(hunting): authenticate will time out at 843.373932s 2024-09-16T04:07:02.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.748+0000 7f1bf95f9640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f1bf409c660 0x7f1bf4138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35410/0 (socket says 172.21.15.148:35410) 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf95f9640 1 -- 172.21.15.148:0/3816589999 learned_addr learned my addr 172.21.15.148:0/3816589999 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045517774 0 0) 0x7f1bf409dc60 con 0x7f1bf409c660 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): my global_id is 4454 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be0003520 con 0x7f1bf409c660 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320592900 0 0) 0x7f1bf409f030 con 0x7f1bf409bb70 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient(hunting): my global_id is 4432 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf409dc60 con 0x7f1bf409bb70 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1346796245 0 0) 0x7f1be0003520 con 0x7f1bf409c660 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 >> v1:172.21.15.171:6789/0 conn(0x7f1bf409bb70 legacy=0x7f1bf409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 >> v1:172.21.15.148:6789/0 conn(0x7f1bf409cc30 legacy=0x7f1bf413c090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: _finish_hunting 0 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 monclient: found mon.noname-b 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf409df40 con 0x7f1bf409c660 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: _finish_auth 0 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: _check_auth_tickets 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bfc05e640 5 monclient: authenticate success, global_id 4454 2024-09-16T04:07:02.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1be8001610 con 0x7f1bf409c660 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bfc05e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: dump: 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bfc05e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1be80018c0 con 0x7f1bf409c660 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 1 -- 172.21.15.148:0/3816589999 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1be8001c90 con 0x7f1bf409c660 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bfc05e640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bfc05e640 4 set_mon_vals no callback set 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.749+0000 7f1bf8df8640 10 monclient: dump: 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.763 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 10 monclient: shutdown 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3816589999 >> v1:172.21.15.148:6790/0 conn(0x7f1bf409c660 legacy=0x7f1bf4138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3816589999 shutdown_connections 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3816589999 >> 172.21.15.148:0/3816589999 conn(0x7f1bf40a5640 msgr2=0x7f1bf40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3816589999 shutdown_connections 2024-09-16T04:07:02.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3816589999 wait complete. 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 10 monclient: build_initial_monmap 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 10 monclient: monmap: 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.750+0000 7f1bfc05e640 1 Processor -- start 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 1 -- start start 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient: init 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf406e400 con 0x7f1bf409c660 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf411ca10 con 0x7f1bf409bb70 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf413c0f0 con 0x7f1bf409cc30 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfc05e640 10 monclient(hunting): authenticate will time out at 843.376467s 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bf95f9640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f1bf409c660 legacy=0x7f1bf413b9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfa5fb640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f1bf409cc30 0x7f1bf41d1820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35412/0 (socket says 172.21.15.148:35412) 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1bfa5fb640 1 -- 172.21.15.148:0/3894402924 learned_addr learned my addr 172.21.15.148:0/3894402924 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1489566963 0 0) 0x7f1bf413c0f0 con 0x7f1bf409cc30 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): my global_id is 4457 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bd8003520 con 0x7f1bf409cc30 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3836052645 0 0) 0x7f1bd8003520 con 0x7f1bf409cc30 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bf413c0f0 con 0x7f1bf409cc30 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1bec0043b0 con 0x7f1bf409cc30 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:02.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): dump: 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185659608 0 0) 0x7f1bf411ca10 con 0x7f1bf409bb70 2024-09-16T04:07:02.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 10 monclient(hunting): my global_id is 4435 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.751+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bd8003520 con 0x7f1bf409bb70 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 551782138 0 0) 0x7f1bf413c0f0 con 0x7f1bf409cc30 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.171:6789/0 conn(0x7f1bf409bb70 legacy=0x7f1bf413b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.148:6789/0 conn(0x7f1bf409c660 legacy=0x7f1bf413b9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 monclient: found mon.c 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40740a0 con 0x7f1bf409cc30 2024-09-16T04:07:02.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 5 monclient: authenticate success, global_id 4457 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 10 monclient: _renew_subs 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf411c730 con 0x7f1bf409cc30 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 10 monclient: _renew_subs 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1bf409aee0 con 0x7f1bf409cc30 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bec0046a0 con 0x7f1bf409cc30 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:02.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1bec004a70 con 0x7f1bf409cc30 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be7fff640 4 set_mon_vals no callback set 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 10 monclient: dump: 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.771 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1bec004c90 con 0x7f1bf409cc30 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.752+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1bdc0065c0 con 0x7f1bdc002860 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f1bec0470e0 con 0x7f1bf409cc30 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373264225 0 0) 0x7f1bdc0065c0 con 0x7f1bdc002860 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient(hunting): my global_id is 4460 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf413c0f0 con 0x7f1bdc002860 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1bf95f9640 10 monclient: get_auth_request con 0x7f1bd8038f70 auth_method 0 2024-09-16T04:07:02.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1843007592 0 0) 0x7f1bf413c0f0 con 0x7f1bdc002860 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bdc0065c0 con 0x7f1bdc002860 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1bf00016a0 con 0x7f1bdc002860 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient: dump: 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.773 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.774 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 108408147 0 0) 0x7f1bdc0065c0 con 0x7f1bdc002860 2024-09-16T04:07:02.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1bdc00a150 con 0x7f1bdc002860 2024-09-16T04:07:02.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1bf0001ad0 con 0x7f1bdc002860 2024-09-16T04:07:02.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:02.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.148:6790/0 conn(0x7f1bdc002860 legacy=0x7f1bdc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.753+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.148:6790/0 conn(0x7f1bdc002860 legacy=0x7f1bdc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:02.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.759+0000 7f1bfc05e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:02.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.759+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1bdc009e40 con 0x7f1bdc009100 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275341456 0 0) 0x7f1bdc009e40 con 0x7f1bdc009100 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient(hunting): my global_id is 4463 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bdc0065c0 con 0x7f1bdc009100 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1195115465 0 0) 0x7f1bdc0065c0 con 0x7f1bdc009100 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bdc009e40 con 0x7f1bdc009100 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1bf0003790 con 0x7f1bdc009100 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient: dump: 2024-09-16T04:07:02.777 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 455053172 0 0) 0x7f1bdc009e40 con 0x7f1bdc009100 2024-09-16T04:07:02.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1bdc00c3d0 con 0x7f1bdc009100 2024-09-16T04:07:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4132 (unknown 0 0 2412882213) 0x7f1bf0005a50 con 0x7f1bdc009100 2024-09-16T04:07:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.148:6790/0 conn(0x7f1bdc009100 legacy=0x7f1bdc009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.760+0000 7f1be67fc640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.148:6790/0 conn(0x7f1bdc009100 legacy=0x7f1bdc009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:02.779 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 12, 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 46838, 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:07:02.780 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:07:02.781 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:07:02.782 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:07:02.783 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:07:02.784 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:07:02.785 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:07:02.786 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.788 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.789 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:07:02.790 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:07:02.791 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.171:6812/7327273 conn(0x7f1bd8038f70 legacy=0x7f1bd803b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 10 monclient: shutdown 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 >> v1:172.21.15.148:6790/0 conn(0x7f1bf409cc30 legacy=0x7f1bf41d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfa5fb640 1 -- 172.21.15.148:0/3894402924 reap_dead start 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfa5fb640 1 -- 172.21.15.148:0/3894402924 reap_dead start 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 shutdown_connections 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 >> 172.21.15.148:0/3894402924 conn(0x7f1bf40a5640 msgr2=0x7f1bf4071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 shutdown_connections 2024-09-16T04:07:02.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.761+0000 7f1bfc05e640 1 -- 172.21.15.148:0/3894402924 wait complete. 2024-09-16T04:07:02.794 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:07:02.794 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:07:02.794 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:07:02.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.890+0000 7f17ed53a640 10 monclient: get_monmap_and_config 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.890+0000 7f17ed53a640 10 monclient: build_initial_monmap 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.890+0000 7f17ed53a640 10 monclient: monmap: 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:07:02.891613+0000 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:07:02.891613+0000 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:02.893 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 1 Processor -- start 2024-09-16T04:07:02.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 1 -- start start 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient: init 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient(hunting): picked mon.noname-b con 0x7f17e80a2680 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient(hunting): picked mon.noname-a con 0x7f17e80a4920 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient(hunting): picked mon.noname-c con 0x7f17e8136c50 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e80a08b0 con 0x7f17e80a4920 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e80a0b90 con 0x7f17e8136c50 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e80a4da0 con 0x7f17e80a2680 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17ed53a640 10 monclient(hunting): authenticate will time out at 843.516922s 2024-09-16T04:07:02.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.891+0000 7f17dffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f17e80a4920 legacy=0x7f17e813ba90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f17e80a2680 0x7f17e80a2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35446/0 (socket says 172.21.15.148:35446) 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e6ffd640 1 -- 172.21.15.148:0/4084265407 learned_addr learned my addr 172.21.15.148:0/4084265407 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129111916 0 0) 0x7f17e80a4da0 con 0x7f17e80a2680 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient(hunting): my global_id is 4466 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d4003520 con 0x7f17e80a2680 2024-09-16T04:07:02.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933073718 0 0) 0x7f17d4003520 con 0x7f17e80a2680 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 >> v1:172.21.15.171:6789/0 conn(0x7f17e8136c50 legacy=0x7f17e813f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 >> v1:172.21.15.148:6789/0 conn(0x7f17e80a4920 legacy=0x7f17e813ba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 monclient: found mon.noname-b 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e806dfd0 con 0x7f17e80a2680 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17d8001610 con 0x7f17e80a2680 2024-09-16T04:07:02.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17ed53a640 5 monclient: authenticate success, global_id 4466 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17ed53a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: dump: 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.905 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17ed53a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17d8001970 con 0x7f17e80a2680 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17ed53a640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17ed53a640 4 set_mon_vals no callback set 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 1 -- 172.21.15.148:0/4084265407 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17d80048a0 con 0x7f17e80a2680 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.892+0000 7f17e67fc640 10 monclient: dump: 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.906 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 10 monclient: shutdown 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 1 -- 172.21.15.148:0/4084265407 >> v1:172.21.15.148:6790/0 conn(0x7f17e80a2680 legacy=0x7f17e80a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 1 -- 172.21.15.148:0/4084265407 shutdown_connections 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 1 -- 172.21.15.148:0/4084265407 >> 172.21.15.148:0/4084265407 conn(0x7f17e80a97b0 msgr2=0x7f17e80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 1 -- 172.21.15.148:0/4084265407 shutdown_connections 2024-09-16T04:07:02.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 1 -- 172.21.15.148:0/4084265407 wait complete. 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 10 monclient: build_initial_monmap 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 10 monclient: monmap: 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.893+0000 7f17ed53a640 1 Processor -- start 2024-09-16T04:07:02.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 1 -- start start 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient: init 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient(hunting): picked mon.noname-c con 0x7f17e80a2680 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient(hunting): picked mon.noname-b con 0x7f17e80a4920 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient(hunting): picked mon.noname-a con 0x7f17e8136c50 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e8131e20 con 0x7f17e8136c50 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e81319a0 con 0x7f17e80a4920 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17e80a0130 con 0x7f17e80a2680 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17ed53a640 10 monclient(hunting): authenticate will time out at 843.519453s 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17e77fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f17e8136c50 legacy=0x7f17e81d3d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:02.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17e6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f17e80a2680 0x7f17e8130130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35450/0 (socket says 172.21.15.148:35450) 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17e6ffd640 1 -- 172.21.15.148:0/3262988783 learned_addr learned my addr 172.21.15.148:0/3262988783 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969269396 0 0) 0x7f17e80a0130 con 0x7f17e80a2680 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): my global_id is 4469 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c8003550 con 0x7f17e80a2680 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3543004856 0 0) 0x7f17c8003550 con 0x7f17e80a2680 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e80a0130 con 0x7f17e80a2680 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17d8001430 con 0x7f17e80a2680 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:02.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): dump: 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 537367826 0 0) 0x7f17e81319a0 con 0x7f17e80a4920 2024-09-16T04:07:02.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 10 monclient(hunting): my global_id is 4441 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c8003550 con 0x7f17e80a4920 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.894+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1282450764 0 0) 0x7f17e80a0130 con 0x7f17e80a2680 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.171:6789/0 conn(0x7f17e80a4920 legacy=0x7f17e8130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.148:6789/0 conn(0x7f17e8136c50 legacy=0x7f17e81d3d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 monclient: found mon.c 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e80a74b0 con 0x7f17e80a2680 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:02.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 5 monclient: authenticate success, global_id 4469 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 10 monclient: _renew_subs 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17e8133c70 con 0x7f17e80a2680 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 10 monclient: _renew_subs 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f17e8132700 con 0x7f17e80a2680 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17d8004e60 con 0x7f17e80a2680 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17d80051b0 con 0x7f17e80a2680 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 10 monclient: dump: 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.914 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17e5ffb640 4 set_mon_vals no callback set 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f17d80105e0 con 0x7f17e80a2680 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.895+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17e80008d0 con 0x7f17e8130f50 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f17d8046e00 con 0x7f17e80a2680 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804282490 0 0) 0x7f17e80008d0 con 0x7f17e8130f50 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17e77fe640 10 monclient: get_auth_request con 0x7f17c8038fc0 auth_method 0 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient(hunting): my global_id is 4444 2024-09-16T04:07:02.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e80a0130 con 0x7f17e8130f50 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2653359017 0 0) 0x7f17e80a0130 con 0x7f17e8130f50 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e80008d0 con 0x7f17e8130f50 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17d0004070 con 0x7f17e8130f50 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 10 monclient: dump: 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.916 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2801621512 0 0) 0x7f17e80008d0 con 0x7f17e8130f50 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.896+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17e8140af0 con 0x7f17e8130f50 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.897+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f17d0006400 con 0x7f17e8130f50 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.897+0000 7f17df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.897+0000 7f17df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.897+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.171:6789/0 conn(0x7f17e8130f50 legacy=0x7f17e809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.897+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.171:6789/0 conn(0x7f17e8130f50 legacy=0x7f17e809ba70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17ed53a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17e802da60 con 0x7f17e802d650 2024-09-16T04:07:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047399742 0 0) 0x7f17e802da60 con 0x7f17e802d650 2024-09-16T04:07:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient(hunting): my global_id is 4447 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e80008d0 con 0x7f17e802d650 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 682091363 0 0) 0x7f17e80008d0 con 0x7f17e802d650 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e802da60 con 0x7f17e802d650 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17d00046b0 con 0x7f17e802d650 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:02.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.903+0000 7f17df7fe640 10 monclient: dump: 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 50432943 0 0) 0x7f17e802da60 con 0x7f17e802d650 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17e809c890 con 0x7f17e802d650 2024-09-16T04:07:02.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4301 (unknown 0 0 1964174856) 0x7f17d0005a50 con 0x7f17e802d650 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.171:6789/0 conn(0x7f17e802d650 legacy=0x7f17e809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17df7fe640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.171:6789/0 conn(0x7f17e802d650 legacy=0x7f17e809c180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 12, 2024-09-16T04:07:02.922 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 46942, 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:07:02.923 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:07:02.924 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:07:02.925 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:07:02.926 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.927 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:07:02.928 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:07:02.929 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:07:02.930 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.931 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.932 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.933 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:07:02.934 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:07:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.171:6812/7327273 conn(0x7f17c8038fc0 legacy=0x7f17c803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 10 monclient: shutdown 2024-09-16T04:07:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 >> v1:172.21.15.148:6790/0 conn(0x7f17e80a2680 legacy=0x7f17e8130130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17e77fe640 1 -- 172.21.15.148:0/3262988783 reap_dead start 2024-09-16T04:07:02.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17e77fe640 1 -- 172.21.15.148:0/3262988783 reap_dead start 2024-09-16T04:07:02.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 shutdown_connections 2024-09-16T04:07:02.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 >> 172.21.15.148:0/3262988783 conn(0x7f17e80a97b0 msgr2=0x7f17e812e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:02.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 shutdown_connections 2024-09-16T04:07:02.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:07:02.904+0000 7f17ed53a640 1 -- 172.21.15.148:0/3262988783 wait complete. 2024-09-16T04:07:02.937 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:07:02.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:07:44.220 INFO:tasks.workunit.client.0.smithi171.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T04:07:44.220 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr: git switch -c 2024-09-16T04:07:44.221 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr:Or undo this operation with: 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr: git switch - 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:44.222 INFO:tasks.workunit.client.0.smithi171.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T04:07:44.224 DEBUG:teuthology.orchestra.run.smithi171:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T04:07:44.324 INFO:tasks.workunit.client.0.smithi171.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T04:07:44.326 INFO:tasks.workunit.client.0.smithi171.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T04:07:44.326 INFO:tasks.workunit.client.0.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T04:07:44.998 INFO:tasks.workunit.client.0.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T04:07:45.076 INFO:tasks.workunit.client.0.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T04:07:45.118 INFO:tasks.workunit.client.0.smithi171.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T04:07:45.120 INFO:tasks.workunit.client.0.smithi171.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T04:07:45.120 INFO:tasks.workunit.client.0.smithi171.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T04:07:45.160 INFO:tasks.workunit.client.0.smithi171.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T04:07:45.163 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-09-16T04:07:45.163 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T04:07:45.182 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-16T04:07:45.184 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-16T04:07:45.184 DEBUG:teuthology.orchestra.run.smithi171:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-16T04:07:45.245 INFO:tasks.workunit.client.0.smithi171.stderr:+ TEST_POOL=testpool1234 2024-09-16T04:07:45.246 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-16T04:07:45.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.321+0000 7f46c07f8640 10 monclient: get_monmap_and_config 2024-09-16T04:07:45.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.321+0000 7f46c07f8640 10 monclient: build_initial_monmap 2024-09-16T04:07:45.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.321+0000 7f46c07f8640 10 monclient: monmap: 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:45.323549+0000 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:45.323549+0000 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:45.325 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:45.326 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:45.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.322+0000 7f46c07f8640 1 Processor -- start 2024-09-16T04:07:45.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.322+0000 7f46c07f8640 1 -- start start 2024-09-16T04:07:45.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.322+0000 7f46c07f8640 10 monclient: init 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient(hunting): picked mon.noname-a con 0x7f46b809ff90 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient(hunting): picked mon.noname-b con 0x7f46b80a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient(hunting): picked mon.noname-c con 0x7f46b80a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b80a0f20 con 0x7f46b809ff90 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b80a1200 con 0x7f46b80a0800 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b809bca0 con 0x7f46b80a0370 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient(hunting): authenticate will time out at 840.829171s 2024-09-16T04:07:45.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46be594640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f46b809ff90 legacy=0x7f46b809f7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bed95640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f46b80a0800 0x7f46b813d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:41514/0 (socket says 172.21.15.171:41514) 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bed95640 1 -- 172.21.15.171:0/2992222087 learned_addr learned my addr 172.21.15.171:0/2992222087 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148684900 0 0) 0x7f46b80a1200 con 0x7f46b80a0800 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): _negotiate 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): my global_id is 4450 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a4003520 con 0x7f46b80a0800 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256442830 0 0) 0x7f46b809bca0 con 0x7f46b80a0370 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): _negotiate 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:45.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient(hunting): my global_id is 4472 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b80a1200 con 0x7f46b80a0370 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3912446184 0 0) 0x7f46a4003520 con 0x7f46b80a0800 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 >> v1:172.21.15.148:6790/0 conn(0x7f46b80a0370 legacy=0x7f46b8139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 >> v1:172.21.15.148:6789/0 conn(0x7f46b809ff90 legacy=0x7f46b809f7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: _finish_hunting 0 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 monclient: found mon.noname-c 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b809e830 con 0x7f46b80a0800 2024-09-16T04:07:45.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: _finish_auth 0 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: _check_auth_tickets 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 5 monclient: authenticate success, global_id 4450 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f46b0001910 con 0x7f46b80a0800 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: dump: 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:45.338 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46b0001b70 con 0x7f46b80a0800 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 1 -- 172.21.15.171:0/2992222087 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f46b0001f40 con 0x7f46b80a0800 2024-09-16T04:07:45.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46c07f8640 4 set_mon_vals no callback set 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.323+0000 7f46bd592640 10 monclient: dump: 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:45.340 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 10 monclient: shutdown 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 1 -- 172.21.15.171:0/2992222087 >> v1:172.21.15.171:6789/0 conn(0x7f46b80a0800 legacy=0x7f46b813d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 1 -- 172.21.15.171:0/2992222087 shutdown_connections 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 1 -- 172.21.15.171:0/2992222087 >> 172.21.15.171:0/2992222087 conn(0x7f46b80a5450 msgr2=0x7f46b80a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 1 -- 172.21.15.171:0/2992222087 shutdown_connections 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 1 -- 172.21.15.171:0/2992222087 wait complete. 2024-09-16T04:07:45.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 10 monclient: build_initial_monmap 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.324+0000 7f46c07f8640 10 monclient: monmap: 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 1 Processor -- start 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 1 -- start start 2024-09-16T04:07:45.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient: init 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient(hunting): picked mon.noname-b con 0x7f46b809ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient(hunting): picked mon.noname-c con 0x7f46b80a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80a0800 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b8124190 con 0x7f46b80a0800 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b8125ba0 con 0x7f46b809ff90 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b80716e0 con 0x7f46b80a0370 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:45.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46c07f8640 10 monclient(hunting): authenticate will time out at 840.831728s 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46be594640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f46b809ff90 0x7f46b8121e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:41518/0 (socket says 172.21.15.171:41518) 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46be594640 1 -- 172.21.15.171:0/3994134429 learned_addr learned my addr 172.21.15.171:0/3994134429 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46bed95640 1 -- 172.21.15.171:0/3994134429 >> v1:172.21.15.148:6789/0 conn(0x7f46b80a0800 legacy=0x7f46b81d1940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765572349 0 0) 0x7f46b8125ba0 con 0x7f46b809ff90 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46aaffd640 10 monclient(hunting): my global_id is 4453 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.325+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f469c003520 con 0x7f46b809ff90 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2860996698 0 0) 0x7f469c003520 con 0x7f46b809ff90 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46b8125ba0 con 0x7f46b809ff90 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f46b40041d0 con 0x7f46b809ff90 2024-09-16T04:07:45.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): dump: 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:45.345 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004695661 0 0) 0x7f46b80716e0 con 0x7f46b80a0370 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient(hunting): my global_id is 4475 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f469c003520 con 0x7f46b80a0370 2024-09-16T04:07:45.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 313055150 0 0) 0x7f46b8125ba0 con 0x7f46b809ff90 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 >> v1:172.21.15.148:6790/0 conn(0x7f46b80a0370 legacy=0x7f46b8122590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 >> v1:172.21.15.148:6789/0 conn(0x7f46b80a0800 legacy=0x7f46b81d1940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 monclient: found mon.b 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b806eb60 con 0x7f46b809ff90 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 5 monclient: authenticate success, global_id 4453 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 10 monclient: _renew_subs 2024-09-16T04:07:45.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46b809c0b0 con 0x7f46b809ff90 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 10 monclient: _renew_subs 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46b81258c0 con 0x7f46b809ff90 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f46b4004530 con 0x7f46b809ff90 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:45.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f46b4004900 con 0x7f46b809ff90 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46bcd91640 4 set_mon_vals no callback set 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.326+0000 7f46aaffd640 10 monclient: dump: 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:45.349 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46c07f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46c07f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46c07f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f46b4004b20 con 0x7f46b809ff90 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a0005080 con 0x7f46b809ff90 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (unknown 3345967258 0 0) 0x7f46b4046f50 con 0x7f46b809ff90 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.327+0000 7f46bdd93640 10 monclient: get_auth_request con 0x7f469c038f90 auth_method 0 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.331+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f46b4015e90 con 0x7f46b809ff90 2024-09-16T04:07:45.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.331+0000 7f46aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:45.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.331+0000 7f46aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:45.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.497+0000 7f46c07f8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T04:07:45.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.497+0000 7f46c07f8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T04:07:45.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.497+0000 7f46c07f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:45.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:45.497+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) -- 0x7f46a0005370 con 0x7f46b809ff90 2024-09-16T04:07:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.038+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v20) ==== 152+0+0 (unknown 3597812028 0 0) 0x7f46b4015790 con 0x7f46b809ff90 2024-09-16T04:07:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.038+0000 7f46aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T04:07:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.038+0000 7f46aaffd640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-16T04:07:46.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46c07f8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T04:07:46.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46c07f8640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T04:07:46.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46c07f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) -- 0x7f46a0001140 con 0x7f46b809ff90 2024-09-16T04:07:46.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46aaffd640 1 -- 172.21.15.171:0/3994134429 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v20) ==== 159+0+0 (unknown 264827452 0 0) 0x7f46b401e340 con 0x7f46b809ff90 2024-09-16T04:07:46.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46aaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-16T04:07:46.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.143+0000 7f46aaffd640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-16T04:07:46.147 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'testpool1234' already exists 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 >> v1:172.21.15.171:6812/7327273 conn(0x7f469c038f90 legacy=0x7f469c03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 10 monclient: shutdown 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 >> v1:172.21.15.171:6789/0 conn(0x7f46b809ff90 legacy=0x7f46b8121e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 shutdown_connections 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 >> 172.21.15.171:0/3994134429 conn(0x7f46b80a5450 msgr2=0x7f46b809f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 shutdown_connections 2024-09-16T04:07:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.145+0000 7f46c07f8640 1 -- 172.21.15.171:0/3994134429 wait complete. 2024-09-16T04:07:46.157 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-16T04:07:46.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.236+0000 7f0675253640 10 monclient: get_monmap_and_config 2024-09-16T04:07:46.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.236+0000 7f0675253640 10 monclient: build_initial_monmap 2024-09-16T04:07:46.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.236+0000 7f0675253640 10 monclient: monmap: 2024-09-16T04:07:46.239 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:46.239 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:46.240 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:46.238328+0000 2024-09-16T04:07:46.240 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:46.238328+0000 2024-09-16T04:07:46.240 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:46.240 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.240 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:46.241 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:46.241 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:46.241 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 1 Processor -- start 2024-09-16T04:07:46.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 1 -- start start 2024-09-16T04:07:46.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient: init 2024-09-16T04:07:46.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:46.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:46.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient(hunting): picked mon.noname-b con 0x7f0670133950 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient(hunting): picked mon.noname-a con 0x7f0670137920 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient(hunting): picked mon.noname-c con 0x7f067013b430 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0670076200 con 0x7f0670137920 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06700a0260 con 0x7f067013b430 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0670075880 con 0x7f0670133950 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f0675253640 10 monclient(hunting): authenticate will time out at 841.743750s 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f066f59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f067013b430 0x7f067013d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:41520/0 (socket says 172.21.15.171:41520) 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f066f59e640 1 -- 172.21.15.171:0/1433475221 learned_addr learned my addr 172.21.15.171:0/1433475221 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:46.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.237+0000 7f066659c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f0670137920 legacy=0x7f0670139d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677856923 0 0) 0x7f06700a0260 con 0x7f067013b430 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): _negotiate 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): my global_id is 4456 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f065c003520 con 0x7f067013b430 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 410401210 0 0) 0x7f0670075880 con 0x7f0670133950 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): _negotiate 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:46.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient(hunting): my global_id is 4478 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06700a0260 con 0x7f0670133950 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1920075000 0 0) 0x7f065c003520 con 0x7f067013b430 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 >> v1:172.21.15.148:6790/0 conn(0x7f0670133950 legacy=0x7f0670133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 >> v1:172.21.15.148:6789/0 conn(0x7f0670137920 legacy=0x7f0670139d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: _finish_hunting 0 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 monclient: found mon.noname-c 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067009da90 con 0x7f067013b430 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: _finish_auth 0 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: _check_auth_tickets 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 5 monclient: authenticate success, global_id 4456 2024-09-16T04:07:46.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0668001990 con 0x7f067013b430 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: dump: 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:46.259 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0668001c40 con 0x7f067013b430 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 4 set_mon_vals no callback set 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 1 -- 172.21.15.171:0/1433475221 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0668002010 con 0x7f067013b430 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f066e59c640 10 monclient: dump: 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.260 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 10 monclient: shutdown 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 1 -- 172.21.15.171:0/1433475221 >> v1:172.21.15.171:6789/0 conn(0x7f067013b430 legacy=0x7f067013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 1 -- 172.21.15.171:0/1433475221 shutdown_connections 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.238+0000 7f0675253640 1 -- 172.21.15.171:0/1433475221 >> 172.21.15.171:0/1433475221 conn(0x7f06700a6460 msgr2=0x7f06700a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.239+0000 7f0675253640 1 -- 172.21.15.171:0/1433475221 shutdown_connections 2024-09-16T04:07:46.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.239+0000 7f0675253640 1 -- 172.21.15.171:0/1433475221 wait complete. 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.239+0000 7f0675253640 10 monclient: build_initial_monmap 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.239+0000 7f0675253640 10 monclient: monmap: 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.239+0000 7f0675253640 1 Processor -- start 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 1 -- start start 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient: init 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient(hunting): picked mon.noname-a con 0x7f0670133950 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient(hunting): picked mon.noname-b con 0x7f0670137920 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient(hunting): picked mon.noname-c con 0x7f067013b430 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067011d4d0 con 0x7f0670133950 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067013afc0 con 0x7f0670137920 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06700a2cf0 con 0x7f067013b430 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:46.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0675253640 10 monclient(hunting): authenticate will time out at 841.746662s 2024-09-16T04:07:46.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f066ed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f0670133950 legacy=0x7f0670070a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:46.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f066659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0670137920 0x7f06700711a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:41524/0 (socket says 172.21.15.171:41524) 2024-09-16T04:07:46.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f066659c640 1 -- 172.21.15.171:0/3053407284 learned_addr learned my addr 172.21.15.171:0/3053407284 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:46.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701418555 0 0) 0x7f067013afc0 con 0x7f0670137920 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0667fff640 10 monclient(hunting): _negotiate 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0667fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0667fff640 10 monclient(hunting): my global_id is 4459 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.240+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0650003520 con 0x7f0670137920 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103330794 0 0) 0x7f06700a2cf0 con 0x7f067013b430 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): _negotiate 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): my global_id is 4481 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f067013afc0 con 0x7f067013b430 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1426509983 0 0) 0x7f0650003520 con 0x7f0670137920 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06700a2cf0 con 0x7f0670137920 2024-09-16T04:07:46.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0658004180 con 0x7f0670137920 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): dump: 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4139990755 0 0) 0x7f067013afc0 con 0x7f067013b430 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0650003520 con 0x7f067013b430 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0668001df0 con 0x7f067013b430 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient(hunting): dump: 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.268 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3290954241 0 0) 0x7f06700a2cf0 con 0x7f0670137920 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 >> v1:172.21.15.148:6790/0 conn(0x7f067013b430 legacy=0x7f06701d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 >> v1:172.21.15.148:6789/0 conn(0x7f0670133950 legacy=0x7f0670070a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 monclient: found mon.b 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:46.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067009d0c0 con 0x7f0670137920 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: _finish_auth 0 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 5 monclient: authenticate success, global_id 4459 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 10 monclient: _renew_subs 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0670133350 con 0x7f0670137920 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 10 monclient: _renew_subs 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f06700718b0 con 0x7f0670137920 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06580044a0 con 0x7f0670137920 2024-09-16T04:07:46.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f06580047f0 con 0x7f0670137920 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f066dd9b640 4 set_mon_vals no callback set 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.241+0000 7f0667fff640 10 monclient: dump: 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:46.271 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0658004a10 con 0x7f0670137920 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f0675253640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f0675253640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f0675253640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0654005080 con 0x7f0670137920 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3857+0+0 (unknown 2720953780 0 0) 0x7f0658046e20 con 0x7f0670137920 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.242+0000 7f066ed9d640 10 monclient: get_auth_request con 0x7f0650038ef0 auth_method 0 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.247+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0658015c70 con 0x7f0670137920 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.247+0000 7f0667fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:46.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.247+0000 7f0667fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:46.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.413+0000 7f0675253640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T04:07:46.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.413+0000 7f0675253640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T04:07:46.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.413+0000 7f0675253640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:46.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:46.413+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) -- 0x7f0654005370 con 0x7f0670137920 2024-09-16T04:07:47.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.046+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v21) ==== 143+0+0 (unknown 2428324039 0 0) 0x7f0658015570 con 0x7f0670137920 2024-09-16T04:07:47.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.046+0000 7f0667fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T04:07:47.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.046+0000 7f0667fff640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-16T04:07:47.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.147+0000 7f0675253640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T04:07:47.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.147+0000 7f0675253640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T04:07:47.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.147+0000 7f0675253640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:47.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:47.147+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) -- 0x7f0654000bf0 con 0x7f0670137920 2024-09-16T04:07:48.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.051+0000 7f0667fff640 1 -- 172.21.15.171:0/3053407284 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v22) ==== 143+0+0 (unknown 3538591414 0 0) 0x7f065801e120 con 0x7f0670137920 2024-09-16T04:07:48.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.051+0000 7f0667fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-16T04:07:48.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.051+0000 7f0667fff640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-16T04:07:48.055 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 3 pg_num_min to 2 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 >> v1:172.21.15.171:6812/7327273 conn(0x7f0650038ef0 legacy=0x7f065003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 10 monclient: shutdown 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 >> v1:172.21.15.171:6789/0 conn(0x7f0670137920 legacy=0x7f06700711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 shutdown_connections 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 >> 172.21.15.171:0/3053407284 conn(0x7f06700a6460 msgr2=0x7f067013a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:48.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 shutdown_connections 2024-09-16T04:07:48.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.054+0000 7f0675253640 1 -- 172.21.15.171:0/3053407284 wait complete. 2024-09-16T04:07:48.070 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T04:07:48.071 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 2 2024-09-16T04:07:48.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.154+0000 7ff775265640 10 monclient: get_monmap_and_config 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.154+0000 7ff775265640 10 monclient: build_initial_monmap 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.154+0000 7ff775265640 10 monclient: monmap: 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:48.155863+0000 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:48.155863+0000 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:48.157 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.154+0000 7ff775265640 1 Processor -- start 2024-09-16T04:07:48.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.154+0000 7ff775265640 1 -- start start 2024-09-16T04:07:48.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.154+0000 7ff775265640 10 monclient: init 2024-09-16T04:07:48.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient(hunting): picked mon.noname-b con 0x7ff770136dd0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient(hunting): picked mon.noname-c con 0x7ff7701371b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient(hunting): picked mon.noname-a con 0x7ff7701375b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77009af40 con 0x7ff7701375b0 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77009b220 con 0x7ff7701371b0 2024-09-16T04:07:48.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77009aa20 con 0x7ff770136dd0 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient(hunting): authenticate will time out at 843.661325s 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76e7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff7701371b0 0x7ff77013bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40310/0 (socket says 172.21.15.171:40310) 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76e7fc640 1 -- 172.21.15.171:0/44508624 learned_addr learned my addr 172.21.15.171:0/44508624 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76f7fe640 1 -- 172.21.15.171:0/44508624 >> v1:172.21.15.148:6789/0 conn(0x7ff7701375b0 legacy=0x7ff77013f2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095074614 0 0) 0x7ff77009b220 con 0x7ff7701371b0 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): my global_id is 4462 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff758003520 con 0x7ff7701371b0 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840216707 0 0) 0x7ff77009aa20 con 0x7ff770136dd0 2024-09-16T04:07:48.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient(hunting): my global_id is 4484 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff77009b220 con 0x7ff770136dd0 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2806874111 0 0) 0x7ff758003520 con 0x7ff7701371b0 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 >> v1:172.21.15.148:6790/0 conn(0x7ff770136dd0 legacy=0x7ff77009bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 >> v1:172.21.15.148:6789/0 conn(0x7ff7701375b0 legacy=0x7ff77013f2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 monclient: found mon.noname-c 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7700a4d30 con 0x7ff7701371b0 2024-09-16T04:07:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff764001610 con 0x7ff7701371b0 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 5 monclient: authenticate success, global_id 4462 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: dump: 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff764001930 con 0x7ff7701371b0 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 1 -- 172.21.15.171:0/44508624 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff764001d00 con 0x7ff7701371b0 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff775265640 4 set_mon_vals no callback set 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.155+0000 7ff76dffb640 10 monclient: dump: 2024-09-16T04:07:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 10 monclient: shutdown 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 1 -- 172.21.15.171:0/44508624 >> v1:172.21.15.171:6789/0 conn(0x7ff7701371b0 legacy=0x7ff77013bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 1 -- 172.21.15.171:0/44508624 shutdown_connections 2024-09-16T04:07:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 1 -- 172.21.15.171:0/44508624 >> 172.21.15.171:0/44508624 conn(0x7ff7700a9860 msgr2=0x7ff7700a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 1 -- 172.21.15.171:0/44508624 shutdown_connections 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.156+0000 7ff775265640 1 -- 172.21.15.171:0/44508624 wait complete. 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.157+0000 7ff775265640 10 monclient: build_initial_monmap 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.157+0000 7ff775265640 10 monclient: monmap: 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.157+0000 7ff775265640 1 Processor -- start 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.157+0000 7ff775265640 1 -- start start 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.157+0000 7ff775265640 10 monclient: init 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.157+0000 7ff775265640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient(hunting): picked mon.noname-c con 0x7ff770136dd0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient(hunting): picked mon.noname-a con 0x7ff7701371b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient(hunting): picked mon.noname-b con 0x7ff7701375b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77013cae0 con 0x7ff7701371b0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff770139c50 con 0x7ff7701375b0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77013cdc0 con 0x7ff770136dd0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient(hunting): authenticate will time out at 843.664097s 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff76e7fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff7701371b0 legacy=0x7ff770139540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff76f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff7701375b0 0x7ff7701d3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40314/0 (socket says 172.21.15.171:40314) 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff76f7fe640 1 -- 172.21.15.171:0/2334895084 learned_addr learned my addr 172.21.15.171:0/2334895084 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936640021 0 0) 0x7ff770139c50 con 0x7ff7701375b0 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): my global_id is 4465 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff75c003520 con 0x7ff7701375b0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 272676999 0 0) 0x7ff77013cdc0 con 0x7ff770136dd0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): my global_id is 4487 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff770139c50 con 0x7ff770136dd0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2935815114 0 0) 0x7ff75c003520 con 0x7ff7701375b0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff77013cdc0 con 0x7ff7701375b0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff768004440 con 0x7ff7701375b0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): dump: 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2059419573 0 0) 0x7ff77013cdc0 con 0x7ff7701375b0 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 >> v1:172.21.15.148:6790/0 conn(0x7ff770136dd0 legacy=0x7ff770138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 >> v1:172.21.15.148:6789/0 conn(0x7ff7701371b0 legacy=0x7ff770139540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 monclient: found mon.b 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff770136190 con 0x7ff7701375b0 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 5 monclient: authenticate success, global_id 4465 2024-09-16T04:07:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient: _renew_subs 2024-09-16T04:07:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff770135c40 con 0x7ff7701375b0 2024-09-16T04:07:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient: _renew_subs 2024-09-16T04:07:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff77013c800 con 0x7ff7701375b0 2024-09-16T04:07:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff7680046f0 con 0x7ff7701375b0 2024-09-16T04:07:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff768004ac0 con 0x7ff7701375b0 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff76d7fa640 4 set_mon_vals no callback set 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.158+0000 7ff7577fe640 10 monclient: dump: 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.159+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff768004d20 con 0x7ff7701375b0 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.159+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(22..22 src has 1..22) ==== 3873+0+0 (unknown 656418306 0 0) 0x7ff7680470b0 con 0x7ff7701375b0 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.159+0000 7ff76effd640 10 monclient: get_auth_request con 0x7ff75c038b90 auth_method 0 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.159+0000 7ff775265640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.159+0000 7ff775265640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.160+0000 7ff775265640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.160+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff750005080 con 0x7ff7701375b0 2024-09-16T04:07:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.164+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff768015f70 con 0x7ff7701375b0 2024-09-16T04:07:48.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.164+0000 7ff7577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.164+0000 7ff7577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:48.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff775265640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:48.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff775265640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:48.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff775265640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) -- 0x7ff750005b80 con 0x7ff7701375b0 2024-09-16T04:07:48.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v22) ==== 105+0+14 (unknown 1808143916 0 3052767636) 0x7ff768015870 con 0x7ff7701375b0 2024-09-16T04:07:48.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff7577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:48.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.329+0000 7ff7577fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:48.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff775265640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:48.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff775265640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:48.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff775265640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) -- 0x7ff750005760 con 0x7ff7701375b0 2024-09-16T04:07:48.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff7577fe640 1 -- 172.21.15.171:0/2334895084 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v22) ==== 105+0+14 (unknown 1808143916 0 3052767636) 0x7ff768019bc0 con 0x7ff7701375b0 2024-09-16T04:07:48.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff7577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:48.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.426+0000 7ff7577fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 >> v1:172.21.15.171:6812/7327273 conn(0x7ff75c038b90 legacy=0x7ff75c03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 10 monclient: shutdown 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 >> v1:172.21.15.171:6789/0 conn(0x7ff7701375b0 legacy=0x7ff7701d3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 shutdown_connections 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 >> 172.21.15.171:0/2334895084 conn(0x7ff7700a9860 msgr2=0x7ff7700a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 shutdown_connections 2024-09-16T04:07:48.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.428+0000 7ff775265640 1 -- 172.21.15.171:0/2334895084 wait complete. 2024-09-16T04:07:48.440 INFO:tasks.workunit.client.0.smithi171.stdout:pg_num_min: 2 2024-09-16T04:07:48.440 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-16T04:07:48.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.517+0000 7f69c9c65640 10 monclient: get_monmap_and_config 2024-09-16T04:07:48.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.517+0000 7f69c9c65640 10 monclient: build_initial_monmap 2024-09-16T04:07:48.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.517+0000 7f69c9c65640 10 monclient: monmap: 2024-09-16T04:07:48.520 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:48.520 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:48.519400+0000 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:48.519400+0000 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:48.521 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 1 Processor -- start 2024-09-16T04:07:48.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 1 -- start start 2024-09-16T04:07:48.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient: init 2024-09-16T04:07:48.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:48.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:48.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient(hunting): picked mon.noname-a con 0x7f69c4133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient(hunting): picked mon.noname-c con 0x7f69c41341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient(hunting): picked mon.noname-b con 0x7f69c413a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c409da90 con 0x7f69c4133700 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c409dd70 con 0x7f69c41341f0 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c409e050 con 0x7f69c413a350 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c9c65640 10 monclient(hunting): authenticate will time out at 844.024924s 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f69c41341f0 0x7f69c4138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40324/0 (socket says 172.21.15.171:40324) 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c2ffd640 1 -- 172.21.15.171:0/2519277028 learned_addr learned my addr 172.21.15.171:0/2519277028 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.518+0000 7f69c37fe640 1 -- 172.21.15.171:0/2519277028 >> v1:172.21.15.148:6789/0 conn(0x7f69c4133700 legacy=0x7f69c4133ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 566410621 0 0) 0x7f69c409dd70 con 0x7f69c41341f0 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient(hunting): my global_id is 4468 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b0003520 con 0x7f69c41341f0 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 784311109 0 0) 0x7f69b0003520 con 0x7f69c41341f0 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 >> v1:172.21.15.148:6790/0 conn(0x7f69c413a350 legacy=0x7f69c413c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 >> v1:172.21.15.148:6789/0 conn(0x7f69c4133700 legacy=0x7f69c4133ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 monclient: found mon.noname-c 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c409d4f0 con 0x7f69c41341f0 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:48.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69ac001610 con 0x7f69c41341f0 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c9c65640 5 monclient: authenticate success, global_id 4468 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c9c65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: dump: 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.532 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c9c65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69ac001970 con 0x7f69c41341f0 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 1 -- 172.21.15.171:0/2519277028 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69ac0048a0 con 0x7f69c41341f0 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c9c65640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c9c65640 4 set_mon_vals no callback set 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.519+0000 7f69c27fc640 10 monclient: dump: 2024-09-16T04:07:48.533 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 10 monclient: shutdown 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 1 -- 172.21.15.171:0/2519277028 >> v1:172.21.15.171:6789/0 conn(0x7f69c41341f0 legacy=0x7f69c4138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 1 -- 172.21.15.171:0/2519277028 shutdown_connections 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 1 -- 172.21.15.171:0/2519277028 >> 172.21.15.171:0/2519277028 conn(0x7f69c40a5450 msgr2=0x7f69c40a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 1 -- 172.21.15.171:0/2519277028 shutdown_connections 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 1 -- 172.21.15.171:0/2519277028 wait complete. 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 10 monclient: build_initial_monmap 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 10 monclient: monmap: 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:48.535 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.520+0000 7f69c9c65640 1 Processor -- start 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 1 -- start start 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient: init 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient(hunting): picked mon.noname-a con 0x7f69c4133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient(hunting): picked mon.noname-c con 0x7f69c41341f0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient(hunting): picked mon.noname-b con 0x7f69c413a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c4120f50 con 0x7f69c4133700 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c411db50 con 0x7f69c413a350 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c4120b30 con 0x7f69c41341f0 2024-09-16T04:07:48.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c9c65640 10 monclient(hunting): authenticate will time out at 844.027667s 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f69c4133700 legacy=0x7f69c411cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f69c413a350 0x7f69c41d3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40326/0 (socket says 172.21.15.171:40326) 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69c3fff640 1 -- 172.21.15.171:0/324803664 learned_addr learned my addr 172.21.15.171:0/324803664 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109656778 0 0) 0x7f69c411db50 con 0x7f69c413a350 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69bbfff640 10 monclient(hunting): my global_id is 4471 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.521+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69a4003520 con 0x7f69c413a350 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317731192 0 0) 0x7f69c4120b30 con 0x7f69c41341f0 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): my global_id is 4493 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c411db50 con 0x7f69c41341f0 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1582446657 0 0) 0x7f69a4003520 con 0x7f69c413a350 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69c4120b30 con 0x7f69c413a350 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69bc001cc0 con 0x7f69c413a350 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): dump: 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.538 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3244219897 0 0) 0x7f69c411db50 con 0x7f69c41341f0 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69a4003520 con 0x7f69c41341f0 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69ac001430 con 0x7f69c41341f0 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient(hunting): dump: 2024-09-16T04:07:48.539 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3348313269 0 0) 0x7f69c4120b30 con 0x7f69c413a350 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 >> v1:172.21.15.148:6790/0 conn(0x7f69c41341f0 legacy=0x7f69c411d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 >> v1:172.21.15.148:6789/0 conn(0x7f69c4133700 legacy=0x7f69c411cd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 monclient: found mon.b 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c40a46b0 con 0x7f69c413a350 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 5 monclient: authenticate success, global_id 4471 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 10 monclient: _renew_subs 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69c4139ff0 con 0x7f69c413a350 2024-09-16T04:07:48.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 10 monclient: _renew_subs 2024-09-16T04:07:48.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f69c4120850 con 0x7f69c413a350 2024-09-16T04:07:48.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69bc002070 con 0x7f69c413a350 2024-09-16T04:07:48.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:48.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69bc003de0 con 0x7f69c413a350 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69bbfff640 10 monclient: dump: 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:48.543 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.522+0000 7f69c1ffb640 4 set_mon_vals no callback set 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f69bc003f80 con 0x7f69c413a350 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(22..22 src has 1..22) ==== 3873+0+0 (unknown 656418306 0 0) 0x7f69bc0471c0 con 0x7f69c413a350 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69c37fe640 10 monclient: get_auth_request con 0x7f69a4038f60 auth_method 0 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69c9c65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69c9c65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69c9c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.523+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a8005080 con 0x7f69c413a350 2024-09-16T04:07:48.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.528+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f69bc015f40 con 0x7f69c413a350 2024-09-16T04:07:48.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.528+0000 7f69bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.528+0000 7f69bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:48.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.692+0000 7f69c9c65640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T04:07:48.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.692+0000 7f69c9c65640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T04:07:48.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.692+0000 7f69c9c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:48.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:48.692+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) -- 0x7f69a8005370 con 0x7f69c413a350 2024-09-16T04:07:49.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.055+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v23) ==== 145+0+0 (unknown 2513166620 0 0) 0x7f69bc019bf0 con 0x7f69c413a350 2024-09-16T04:07:49.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.055+0000 7f69bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T04:07:49.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.055+0000 7f69bbfff640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-16T04:07:49.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.157+0000 7f69c9c65640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T04:07:49.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.157+0000 7f69c9c65640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T04:07:49.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.157+0000 7f69c9c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:49.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:49.157+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) -- 0x7f69a8000c20 con 0x7f69c413a350 2024-09-16T04:07:50.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.060+0000 7f69bbfff640 1 -- 172.21.15.171:0/324803664 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v24) ==== 145+0+0 (unknown 2945105049 0 0) 0x7f69bc01e4b0 con 0x7f69c413a350 2024-09-16T04:07:50.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.060+0000 7f69bbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-16T04:07:50.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.060+0000 7f69bbfff640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-16T04:07:50.063 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 3 pg_num_max to 33 2024-09-16T04:07:50.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 >> v1:172.21.15.171:6812/7327273 conn(0x7f69a4038f60 legacy=0x7f69a403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 10 monclient: shutdown 2024-09-16T04:07:50.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:50.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 >> v1:172.21.15.171:6789/0 conn(0x7f69c413a350 legacy=0x7f69c41d3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 shutdown_connections 2024-09-16T04:07:50.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 >> 172.21.15.171:0/324803664 conn(0x7f69c40a5450 msgr2=0x7f69c409b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:50.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 shutdown_connections 2024-09-16T04:07:50.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.063+0000 7f69c9c65640 1 -- 172.21.15.171:0/324803664 wait complete. 2024-09-16T04:07:50.078 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T04:07:50.078 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 33 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.160+0000 7fc4af01e640 10 monclient: get_monmap_and_config 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.160+0000 7fc4af01e640 10 monclient: build_initial_monmap 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.160+0000 7fc4af01e640 10 monclient: monmap: 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:50.161983+0000 2024-09-16T04:07:50.163 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:50.161983+0000 2024-09-16T04:07:50.164 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:50.164 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.164 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:50.164 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:50.164 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:50.164 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.160+0000 7fc4af01e640 1 Processor -- start 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.160+0000 7fc4af01e640 1 -- start start 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.160+0000 7fc4af01e640 10 monclient: init 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a8136dd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a81371b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a81375b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a809af40 con 0x7fc4a81375b0 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a809b220 con 0x7fc4a8136dd0 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a809aa20 con 0x7fc4a81371b0 2024-09-16T04:07:50.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4af01e640 10 monclient(hunting): authenticate will time out at 845.667465s 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4ad5bb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc4a81375b0 legacy=0x7fc4a813f2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4acdba640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc4a8136dd0 0x7fc4a809bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40336/0 (socket says 172.21.15.171:40336) 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4acdba640 1 -- 172.21.15.171:0/1583189587 learned_addr learned my addr 172.21.15.171:0/1583189587 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454634397 0 0) 0x7fc4a809b220 con 0x7fc4a8136dd0 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): my global_id is 4474 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc498003520 con 0x7fc4a8136dd0 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86770178 0 0) 0x7fc4a809aa20 con 0x7fc4a81371b0 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 10 monclient(hunting): my global_id is 4496 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4a809b220 con 0x7fc4a81371b0 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.161+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 779598388 0 0) 0x7fc498003520 con 0x7fc4a8136dd0 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 >> v1:172.21.15.148:6790/0 conn(0x7fc4a81371b0 legacy=0x7fc4a813bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 >> v1:172.21.15.148:6789/0 conn(0x7fc4a81375b0 legacy=0x7fc4a813f2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a80a4d30 con 0x7fc4a8136dd0 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:50.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc49c001610 con 0x7fc4a8136dd0 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 5 monclient: authenticate success, global_id 4474 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: dump: 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.175 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc49c001930 con 0x7fc4a8136dd0 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 1 -- 172.21.15.171:0/1583189587 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc49c001d00 con 0x7fc4a8136dd0 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 4 set_mon_vals no callback set 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4a77fe640 10 monclient: dump: 2024-09-16T04:07:50.176 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 10 monclient: shutdown 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 1 -- 172.21.15.171:0/1583189587 >> v1:172.21.15.171:6789/0 conn(0x7fc4a8136dd0 legacy=0x7fc4a809bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 1 -- 172.21.15.171:0/1583189587 shutdown_connections 2024-09-16T04:07:50.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 1 -- 172.21.15.171:0/1583189587 >> 172.21.15.171:0/1583189587 conn(0x7fc4a80a9860 msgr2=0x7fc4a80a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 1 -- 172.21.15.171:0/1583189587 shutdown_connections 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.162+0000 7fc4af01e640 1 -- 172.21.15.171:0/1583189587 wait complete. 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.163+0000 7fc4af01e640 10 monclient: build_initial_monmap 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.163+0000 7fc4af01e640 10 monclient: monmap: 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:50.178 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.163+0000 7fc4af01e640 1 Processor -- start 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.163+0000 7fc4af01e640 1 -- start start 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.163+0000 7fc4af01e640 10 monclient: init 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a8136dd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a81371b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a81375b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a80a0340 con 0x7fc4a8136dd0 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a813ba90 con 0x7fc4a81375b0 2024-09-16T04:07:50.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a813a8c0 con 0x7fc4a81371b0 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4af01e640 10 monclient(hunting): authenticate will time out at 845.670326s 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4acdba640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc4a8136dd0 legacy=0x7fc4a8138330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4ad5bb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc4a81375b0 0x7fc4a81d3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40342/0 (socket says 172.21.15.171:40342) 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4ad5bb640 1 -- 172.21.15.171:0/1485455969 learned_addr learned my addr 172.21.15.171:0/1485455969 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547392261 0 0) 0x7fc4a813ba90 con 0x7fc4a81375b0 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): my global_id is 4477 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc48c003520 con 0x7fc4a81375b0 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065868740 0 0) 0x7fc4a813a8c0 con 0x7fc4a81371b0 2024-09-16T04:07:50.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): my global_id is 4499 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4a813ba90 con 0x7fc4a81371b0 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2930091775 0 0) 0x7fc48c003520 con 0x7fc4a81375b0 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc4a813a8c0 con 0x7fc4a81375b0 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc4a0004440 con 0x7fc4a81375b0 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): dump: 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.181 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.182 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.182 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.182 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1467377318 0 0) 0x7fc4a813ba90 con 0x7fc4a81371b0 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc48c003520 con 0x7fc4a81371b0 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc4940018b0 con 0x7fc4a81371b0 2024-09-16T04:07:50.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.164+0000 7fc4a57fa640 10 monclient(hunting): dump: 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2634265264 0 0) 0x7fc4a813a8c0 con 0x7fc4a81375b0 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 >> v1:172.21.15.148:6790/0 conn(0x7fc4a81371b0 legacy=0x7fc4a8138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 >> v1:172.21.15.148:6789/0 conn(0x7fc4a8136dd0 legacy=0x7fc4a8138330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: _finish_hunting 0 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 monclient: found mon.b 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a8135dd0 con 0x7fc4a81375b0 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: _finish_auth 0 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 5 monclient: authenticate success, global_id 4477 2024-09-16T04:07:50.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 10 monclient: _renew_subs 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a813bd70 con 0x7fc4a81375b0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 10 monclient: _renew_subs 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc4a8139330 con 0x7fc4a81375b0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc4a00046f0 con 0x7fc4a81375b0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a6ffd640 4 set_mon_vals no callback set 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc4a0004ac0 con 0x7fc4a81375b0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 10 monclient: dump: 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.165+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc4a0004ce0 con 0x7fc4a81375b0 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.166+0000 7fc4af01e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.166+0000 7fc4af01e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.166+0000 7fc4af01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.166+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc488005080 con 0x7fc4a81375b0 2024-09-16T04:07:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.166+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 3889+0+0 (unknown 1330234895 0 0) 0x7fc4a0047160 con 0x7fc4a81375b0 2024-09-16T04:07:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.166+0000 7fc4acdba640 10 monclient: get_auth_request con 0x7fc48c038f10 auth_method 0 2024-09-16T04:07:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.170+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc4a0015fb0 con 0x7fc4a81375b0 2024-09-16T04:07:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.170+0000 7fc4a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.170+0000 7fc4a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:50.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.335+0000 7fc4af01e640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:50.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.335+0000 7fc4af01e640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:50.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.335+0000 7fc4af01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.335+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) -- 0x7fc488005b80 con 0x7fc4a81375b0 2024-09-16T04:07:50.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.336+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v24) ==== 105+0+15 (unknown 2538665401 0 2995009362) 0x7fc4a0019c60 con 0x7fc4a81375b0 2024-09-16T04:07:50.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.336+0000 7fc4a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:50.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.336+0000 7fc4a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4af01e640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4af01e640 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4af01e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) -- 0x7fc488001090 con 0x7fc4a81375b0 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4a57fa640 1 -- 172.21.15.171:0/1485455969 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v24) ==== 105+0+15 (unknown 2538665401 0 2995009362) 0x7fc4a0013c80 con 0x7fc4a81375b0 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4a57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:50.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.432+0000 7fc4a57fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:07:50.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 >> v1:172.21.15.171:6812/7327273 conn(0x7fc48c038f10 legacy=0x7fc48c03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 10 monclient: shutdown 2024-09-16T04:07:50.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:50.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 >> v1:172.21.15.171:6789/0 conn(0x7fc4a81375b0 legacy=0x7fc4a81d3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 shutdown_connections 2024-09-16T04:07:50.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 >> 172.21.15.171:0/1485455969 conn(0x7fc4a80a9860 msgr2=0x7fc4a813fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:50.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 shutdown_connections 2024-09-16T04:07:50.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.434+0000 7fc4af01e640 1 -- 172.21.15.171:0/1485455969 wait complete. 2024-09-16T04:07:50.446 INFO:tasks.workunit.client.0.smithi171.stdout:pg_num_max: 33 2024-09-16T04:07:50.446 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T04:07:50.447 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:50.447 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-16T04:07:50.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.526+0000 7f452d40e640 10 monclient: get_monmap_and_config 2024-09-16T04:07:50.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.526+0000 7f452d40e640 10 monclient: build_initial_monmap 2024-09-16T04:07:50.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.526+0000 7f452d40e640 10 monclient: monmap: 2024-09-16T04:07:50.529 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:50.529 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:50.528097+0000 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:50.528097+0000 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.526+0000 7f452d40e640 1 Processor -- start 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.526+0000 7f452d40e640 1 -- start start 2024-09-16T04:07:50.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.526+0000 7f452d40e640 10 monclient: init 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient(hunting): picked mon.noname-b con 0x7f45280a39e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient(hunting): picked mon.noname-a con 0x7f45280a44d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient(hunting): picked mon.noname-c con 0x7f45280a1c90 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280a4f90 con 0x7f45280a44d0 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280a0a30 con 0x7f45280a1c90 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280a0ff0 con 0x7f45280a39e0 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f452d40e640 10 monclient(hunting): authenticate will time out at 846.033434s 2024-09-16T04:07:50.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f45267fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f45280a44d0 legacy=0x7f452813bd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f45277fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f45280a1c90 0x7f452813f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40348/0 (socket says 172.21.15.171:40348) 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f45277fe640 1 -- 172.21.15.171:0/3738295779 learned_addr learned my addr 172.21.15.171:0/3738295779 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455791556 0 0) 0x7f45280a0a30 con 0x7f45280a1c90 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): my global_id is 4480 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4510003520 con 0x7f45280a1c90 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091682815 0 0) 0x7f45280a0ff0 con 0x7f45280a39e0 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 10 monclient(hunting): my global_id is 4502 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45280a0a30 con 0x7f45280a39e0 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.527+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2973927864 0 0) 0x7f4510003520 con 0x7f45280a1c90 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 >> v1:172.21.15.148:6790/0 conn(0x7f45280a39e0 legacy=0x7f45280a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 >> v1:172.21.15.148:6789/0 conn(0x7f45280a44d0 legacy=0x7f452813bd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 monclient: found mon.noname-c 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4528073820 con 0x7f45280a1c90 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:50.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 5 monclient: authenticate success, global_id 4480 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4518001990 con 0x7f45280a1c90 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: dump: 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.542 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4518001bf0 con 0x7f45280a1c90 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 1 -- 172.21.15.171:0/3738295779 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4518001fc0 con 0x7f45280a1c90 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 4 set_mon_vals no callback set 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f4525ffb640 10 monclient: dump: 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.543 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 10 monclient: shutdown 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 1 -- 172.21.15.171:0/3738295779 >> v1:172.21.15.171:6789/0 conn(0x7f45280a1c90 legacy=0x7f452813f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 1 -- 172.21.15.171:0/3738295779 shutdown_connections 2024-09-16T04:07:50.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 1 -- 172.21.15.171:0/3738295779 >> 172.21.15.171:0/3738295779 conn(0x7f45280a9860 msgr2=0x7f45280a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 1 -- 172.21.15.171:0/3738295779 shutdown_connections 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.528+0000 7f452d40e640 1 -- 172.21.15.171:0/3738295779 wait complete. 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.529+0000 7f452d40e640 10 monclient: build_initial_monmap 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.529+0000 7f452d40e640 10 monclient: monmap: 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:50.545 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.529+0000 7f452d40e640 1 Processor -- start 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.529+0000 7f452d40e640 1 -- start start 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.529+0000 7f452d40e640 10 monclient: init 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient(hunting): picked mon.noname-a con 0x7f45280a1c90 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient(hunting): picked mon.noname-c con 0x7f45280a39e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient(hunting): picked mon.noname-b con 0x7f45280a44d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:50.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4528132eb0 con 0x7f45280a1c90 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280a7e00 con 0x7f45280a44d0 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280a01d0 con 0x7f45280a39e0 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient(hunting): authenticate will time out at 846.036161s 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f4526ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f45280a1c90 legacy=0x7f4528132090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f45277fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f45280a44d0 0x7f45281d3c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40362/0 (socket says 172.21.15.171:40362) 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f45277fe640 1 -- 172.21.15.171:0/3283945832 learned_addr learned my addr 172.21.15.171:0/3283945832 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3780607618 0 0) 0x7f45280a7e00 con 0x7f45280a44d0 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): my global_id is 4483 2024-09-16T04:07:50.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4514003520 con 0x7f45280a44d0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3112826419 0 0) 0x7f45280a01d0 con 0x7f45280a39e0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): my global_id is 4505 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45280a7e00 con 0x7f45280a39e0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2278612705 0 0) 0x7f4514003520 con 0x7f45280a44d0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f45280a01d0 con 0x7f45280a44d0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4518001f40 con 0x7f45280a44d0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): dump: 2024-09-16T04:07:50.548 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3918801209 0 0) 0x7f45280a7e00 con 0x7f45280a39e0 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4514003520 con 0x7f45280a39e0 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f451c0040f0 con 0x7f45280a39e0 2024-09-16T04:07:50.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient(hunting): dump: 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2379856584 0 0) 0x7f45280a01d0 con 0x7f45280a44d0 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 >> v1:172.21.15.148:6790/0 conn(0x7f45280a39e0 legacy=0x7f45281327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 >> v1:172.21.15.148:6789/0 conn(0x7f45280a1c90 legacy=0x7f4528132090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 monclient: found mon.b 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45281375a0 con 0x7f45280a44d0 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 5 monclient: authenticate success, global_id 4483 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient: _renew_subs 2024-09-16T04:07:50.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4528135e10 con 0x7f45280a44d0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient: _renew_subs 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f452d40e640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f45281347a0 con 0x7f45280a44d0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4518001bf0 con 0x7f45280a44d0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4518004e10 con 0x7f45280a44d0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f45257fa640 4 set_mon_vals no callback set 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.530+0000 7f450f7fe640 10 monclient: dump: 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.552 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.531+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4518005030 con 0x7f45280a44d0 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.531+0000 7f452d40e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.531+0000 7f452d40e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.531+0000 7f452d40e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.531+0000 7f452d40e640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4508005080 con 0x7f45280a44d0 2024-09-16T04:07:50.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.532+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 3889+0+0 (unknown 1330234895 0 0) 0x7f45180471b0 con 0x7f45280a44d0 2024-09-16T04:07:50.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.532+0000 7f4526ffd640 10 monclient: get_auth_request con 0x7f4514038ec0 auth_method 0 2024-09-16T04:07:50.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.536+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4518015fe0 con 0x7f45280a44d0 2024-09-16T04:07:50.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.536+0000 7f450f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.536+0000 7f450f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:50.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f452d40e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T04:07:50.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f452d40e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T04:07:50.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f452d40e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f452d40e640 1 -- 172.21.15.171:0/3283945832 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) -- 0x7f4508005370 con 0x7f45280a44d0 2024-09-16T04:07:50.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f450f7fe640 1 -- 172.21.15.171:0/3283945832 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v24) ==== 150+0+0 (unknown 4206633043 0 0) 0x7f45180158e0 con 0x7f45280a44d0 2024-09-16T04:07:50.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f450f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-16T04:07:50.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.702+0000 7f450f7fe640 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-16T04:07:50.706 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-16T04:07:50.709 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:50.709 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T04:07:50.709 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:50.709 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.787+0000 7f3d0feac640 10 monclient: get_monmap_and_config 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.787+0000 7f3d0feac640 10 monclient: build_initial_monmap 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.787+0000 7f3d0feac640 10 monclient: monmap: 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:50.789026+0000 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:50.789026+0000 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:50.791 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.787+0000 7f3d0feac640 1 Processor -- start 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 1 -- start start 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient: init 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient(hunting): picked mon.noname-c con 0x7f3d08136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient(hunting): picked mon.noname-a con 0x7f3d08137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient(hunting): picked mon.noname-b con 0x7f3d081375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d0809b2c0 con 0x7f3d08137140 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d0809b5a0 con 0x7f3d08136d60 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d080a6640 con 0x7f3d081375a0 2024-09-16T04:07:50.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0feac640 10 monclient(hunting): authenticate will time out at 846.294504s 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0dc48640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3d08136d60 0x7f3d0809bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40370/0 (socket says 172.21.15.171:40370) 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0dc48640 1 -- 172.21.15.171:0/159188993 learned_addr learned my addr 172.21.15.171:0/159188993 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0d447640 1 -- 172.21.15.171:0/159188993 >> v1:172.21.15.148:6789/0 conn(0x7f3d08137140 legacy=0x7f3d0813bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471786502 0 0) 0x7f3d0809b5a0 con 0x7f3d08136d60 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0cc46640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0cc46640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0cc46640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0cc46640 10 monclient(hunting): my global_id is 4486 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.788+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cf4003520 con 0x7f3d08136d60 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102889761 0 0) 0x7f3d080a6640 con 0x7f3d081375a0 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient(hunting): my global_id is 4508 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d0809b5a0 con 0x7f3d081375a0 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1585983762 0 0) 0x7f3cf4003520 con 0x7f3d08136d60 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 >> v1:172.21.15.148:6790/0 conn(0x7f3d081375a0 legacy=0x7f3d0813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 >> v1:172.21.15.148:6789/0 conn(0x7f3d08137140 legacy=0x7f3d0813bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: _finish_hunting 0 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 monclient: found mon.noname-c 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d08143b20 con 0x7f3d08136d60 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: _finish_auth 0 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: _check_auth_tickets 2024-09-16T04:07:50.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 5 monclient: authenticate success, global_id 4486 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3cfc001610 con 0x7f3d08136d60 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: dump: 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.801 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3cfc001870 con 0x7f3d08136d60 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 1 -- 172.21.15.171:0/159188993 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3cfc001bc0 con 0x7f3d08136d60 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 4 set_mon_vals no callback set 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0cc46640 10 monclient: dump: 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.802 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 10 monclient: shutdown 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 1 -- 172.21.15.171:0/159188993 >> v1:172.21.15.171:6789/0 conn(0x7f3d08136d60 legacy=0x7f3d0809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 1 -- 172.21.15.171:0/159188993 shutdown_connections 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 1 -- 172.21.15.171:0/159188993 >> 172.21.15.171:0/159188993 conn(0x7f3d080a9ae0 msgr2=0x7f3d080a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:50.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 1 -- 172.21.15.171:0/159188993 shutdown_connections 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.789+0000 7f3d0feac640 1 -- 172.21.15.171:0/159188993 wait complete. 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.790+0000 7f3d0feac640 10 monclient: build_initial_monmap 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.790+0000 7f3d0feac640 10 monclient: monmap: 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:50.804 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.790+0000 7f3d0feac640 1 Processor -- start 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.790+0000 7f3d0feac640 1 -- start start 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.790+0000 7f3d0feac640 10 monclient: init 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient(hunting): picked mon.noname-a con 0x7f3d08136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient(hunting): picked mon.noname-c con 0x7f3d08137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient(hunting): picked mon.noname-b con 0x7f3d081375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d0812eeb0 con 0x7f3d08136d60 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d080a8320 con 0x7f3d081375a0 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d0812e8f0 con 0x7f3d08137140 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient(hunting): authenticate will time out at 846.297149s 2024-09-16T04:07:50.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0dc48640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f3d08136d60 legacy=0x7f3d081cbca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0e449640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3d081375a0 0x7f3d081d43a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40376/0 (socket says 172.21.15.171:40376) 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0e449640 1 -- 172.21.15.171:0/1170851012 learned_addr learned my addr 172.21.15.171:0/1170851012 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807141212 0 0) 0x7f3d080a8320 con 0x7f3d081375a0 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): my global_id is 4489 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cec003520 con 0x7f3d081375a0 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236207609 0 0) 0x7f3d0812e8f0 con 0x7f3d08137140 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:50.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): my global_id is 4511 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d080a8320 con 0x7f3d08137140 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2521672973 0 0) 0x7f3cec003520 con 0x7f3d081375a0 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d0812e8f0 con 0x7f3d081375a0 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d00001d00 con 0x7f3d081375a0 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): dump: 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.807 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2381167048 0 0) 0x7f3d080a8320 con 0x7f3d08137140 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cec003520 con 0x7f3d08137140 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d040040f0 con 0x7f3d08137140 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient(hunting): dump: 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.808 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1469599992 0 0) 0x7f3d0812e8f0 con 0x7f3d081375a0 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 >> v1:172.21.15.148:6790/0 conn(0x7f3d08137140 legacy=0x7f3d081d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 >> v1:172.21.15.148:6789/0 conn(0x7f3d08136d60 legacy=0x7f3d081cbca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 monclient: found mon.b 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:50.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0812ebd0 con 0x7f3d081375a0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 5 monclient: authenticate success, global_id 4489 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient: _renew_subs 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d080a2880 con 0x7f3d081375a0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient: _renew_subs 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3d0feac640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d080a1bd0 con 0x7f3d081375a0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d00003c40 con 0x7f3d081375a0 2024-09-16T04:07:50.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d00004010 con 0x7f3d081375a0 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfbfff640 4 set_mon_vals no callback set 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.791+0000 7f3cfa7fc640 10 monclient: dump: 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:50.811 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.792+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3d000041b0 con 0x7f3d081375a0 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.792+0000 7f3d0feac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.792+0000 7f3d0feac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.792+0000 7f3d0feac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.792+0000 7f3d0feac640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf0005080 con 0x7f3d081375a0 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.792+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 3889+0+0 (unknown 1330234895 0 0) 0x7f3d000472a0 con 0x7f3d081375a0 2024-09-16T04:07:50.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.793+0000 7f3d0dc48640 10 monclient: get_auth_request con 0x7f3cec038f40 auth_method 0 2024-09-16T04:07:50.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.797+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3d00019ce0 con 0x7f3d081375a0 2024-09-16T04:07:50.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.797+0000 7f3cfa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:50.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.797+0000 7f3cfa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:50.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3d0feac640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T04:07:50.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3d0feac640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T04:07:50.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3d0feac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:50.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3d0feac640 1 -- 172.21.15.171:0/1170851012 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) -- 0x7f3cf0005370 con 0x7f3d081375a0 2024-09-16T04:07:50.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3cfa7fc640 1 -- 172.21.15.171:0/1170851012 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v24) ==== 150+0+0 (unknown 4193640499 0 0) 0x7f3d00013d10 con 0x7f3d081375a0 2024-09-16T04:07:50.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3cfa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-16T04:07:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:50.962+0000 7f3cfa7fc640 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-16T04:07:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-16T04:07:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-16T04:07:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.048+0000 7f16c0ae4640 10 monclient: get_monmap_and_config 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.048+0000 7f16c0ae4640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.048+0000 7f16c0ae4640 10 monclient: monmap: 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:51.050032+0000 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:51.050032+0000 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.051 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:51.052 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:51.052 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:51.052 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.048+0000 7f16c0ae4640 1 Processor -- start 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.048+0000 7f16c0ae4640 1 -- start start 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.048+0000 7f16c0ae4640 10 monclient: init 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient(hunting): picked mon.noname-a con 0x7f16bc09f7c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient(hunting): picked mon.noname-c con 0x7f16bc09fba0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient(hunting): picked mon.noname-b con 0x7f16bc1395d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc0a1490 con 0x7f16bc09f7c0 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc09d1b0 con 0x7f16bc09fba0 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc09d490 con 0x7f16bc1395d0 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16c0ae4640 10 monclient(hunting): authenticate will time out at 846.555601s 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b9d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f16bc09fba0 0x7f16bc0a0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40384/0 (socket says 172.21.15.171:40384) 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b9d9b640 1 -- 172.21.15.171:0/1347288193 learned_addr learned my addr 172.21.15.171:0/1347288193 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16ba59c640 1 -- 172.21.15.171:0/1347288193 >> v1:172.21.15.148:6789/0 conn(0x7f16bc09f7c0 legacy=0x7f16bc09de70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601445830 0 0) 0x7f16bc09d1b0 con 0x7f16bc09fba0 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b959a640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b959a640 10 monclient(hunting): my global_id is 4492 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.049+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16a4003520 con 0x7f16bc09fba0 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241291696 0 0) 0x7f16bc09d490 con 0x7f16bc1395d0 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient(hunting): my global_id is 4514 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16bc09d1b0 con 0x7f16bc1395d0 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3511579468 0 0) 0x7f16a4003520 con 0x7f16bc09fba0 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 >> v1:172.21.15.148:6790/0 conn(0x7f16bc1395d0 legacy=0x7f16bc13b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 >> v1:172.21.15.148:6789/0 conn(0x7f16bc09f7c0 legacy=0x7f16bc09de70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 monclient: found mon.noname-c 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc09bca0 con 0x7f16bc09fba0 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 5 monclient: authenticate success, global_id 4492 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16b0001610 con 0x7f16bc09fba0 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: dump: 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.063 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16b0001870 con 0x7f16bc09fba0 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 1 -- 172.21.15.171:0/1347288193 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16b0001c40 con 0x7f16bc09fba0 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 4 set_mon_vals no callback set 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:51.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16b959a640 10 monclient: dump: 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 10 monclient: shutdown 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:51.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 1 -- 172.21.15.171:0/1347288193 >> v1:172.21.15.171:6789/0 conn(0x7f16bc09fba0 legacy=0x7f16bc0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 1 -- 172.21.15.171:0/1347288193 shutdown_connections 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 1 -- 172.21.15.171:0/1347288193 >> 172.21.15.171:0/1347288193 conn(0x7f16bc0a5450 msgr2=0x7f16bc0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 1 -- 172.21.15.171:0/1347288193 shutdown_connections 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.050+0000 7f16c0ae4640 1 -- 172.21.15.171:0/1347288193 wait complete. 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.051+0000 7f16c0ae4640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.051+0000 7f16c0ae4640 10 monclient: monmap: 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:51.066 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.051+0000 7f16c0ae4640 1 Processor -- start 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.051+0000 7f16c0ae4640 1 -- start start 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.051+0000 7f16c0ae4640 10 monclient: init 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient(hunting): picked mon.noname-c con 0x7f16bc09f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient(hunting): picked mon.noname-a con 0x7f16bc09fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient(hunting): picked mon.noname-b con 0x7f16bc1395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc136750 con 0x7f16bc09fba0 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc09c0b0 con 0x7f16bc1395d0 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16bc0727c0 con 0x7f16bc09f7c0 2024-09-16T04:07:51.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16c0ae4640 10 monclient(hunting): authenticate will time out at 846.558423s 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16b9d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f16bc09fba0 legacy=0x7f16bc1d09d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16bad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f16bc1395d0 0x7f16bc1d4100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40396/0 (socket says 172.21.15.171:40396) 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16bad9d640 1 -- 172.21.15.171:0/3923987945 learned_addr learned my addr 172.21.15.171:0/3923987945 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129564644 0 0) 0x7f16bc09c0b0 con 0x7f16bc1395d0 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): my global_id is 4495 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f169c003520 con 0x7f16bc1395d0 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368415556 0 0) 0x7f16bc0727c0 con 0x7f16bc09f7c0 2024-09-16T04:07:51.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): my global_id is 4517 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16bc09c0b0 con 0x7f16bc09f7c0 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2021052008 0 0) 0x7f169c003520 con 0x7f16bc1395d0 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16bc0727c0 con 0x7f16bc1395d0 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16b4001d00 con 0x7f16bc1395d0 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): dump: 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.069 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.070 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.070 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.070 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.070 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.070 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.071 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.071 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.071 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:51.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.052+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3567184185 0 0) 0x7f16bc09c0b0 con 0x7f16bc09f7c0 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f169c003520 con 0x7f16bc09f7c0 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16ac004140 con 0x7f16bc09f7c0 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient(hunting): dump: 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.072 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3610348111 0 0) 0x7f16bc0727c0 con 0x7f16bc1395d0 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 >> v1:172.21.15.148:6790/0 conn(0x7f16bc09f7c0 legacy=0x7f16bc135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 >> v1:172.21.15.148:6789/0 conn(0x7f16bc09fba0 legacy=0x7f16bc1d09d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 monclient: found mon.b 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc124b30 con 0x7f16bc1395d0 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 5 monclient: authenticate success, global_id 4495 2024-09-16T04:07:51.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 10 monclient: _renew_subs 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc124e10 con 0x7f16bc1395d0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 10 monclient: _renew_subs 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16c0ae4640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16bc1220f0 con 0x7f16bc1395d0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16b4003c40 con 0x7f16bc1395d0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16b8d99640 4 set_mon_vals no callback set 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16b4004010 con 0x7f16bc1395d0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 10 monclient: dump: 2024-09-16T04:07:51.074 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.053+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f16b40042f0 con 0x7f16bc1395d0 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.054+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 3889+0+0 (unknown 1330234895 0 0) 0x7f16b40473b0 con 0x7f16bc1395d0 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.054+0000 7f16c0ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.054+0000 7f16ba59c640 10 monclient: get_auth_request con 0x7f169c038f40 auth_method 0 2024-09-16T04:07:51.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.054+0000 7f16c0ae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.054+0000 7f16c0ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.054+0000 7f16c0ae4640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16bc073a70 con 0x7f16bc1395d0 2024-09-16T04:07:51.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.058+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f16bc073a70 con 0x7f16bc1395d0 2024-09-16T04:07:51.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.058+0000 7f16aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.058+0000 7f16aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16c0ae4640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16c0ae4640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16c0ae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16c0ae4640 1 -- 172.21.15.171:0/3923987945 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) -- 0x7f16bc124570 con 0x7f16bc1395d0 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16aaffd640 1 -- 172.21.15.171:0/3923987945 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v24) ==== 146+0+0 (unknown 1613968263 0 0) 0x7f16b4016140 con 0x7f16bc1395d0 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.225+0000 7f16aaffd640 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-16T04:07:51.228 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-16T04:07:51.232 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:51.232 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-16T04:07:51.232 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:51.232 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.310+0000 7f5ca0d4b640 10 monclient: get_monmap_and_config 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.310+0000 7f5ca0d4b640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.310+0000 7f5ca0d4b640 10 monclient: monmap: 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:51.312597+0000 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:51.312597+0000 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.314 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:51.315 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:51.315 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:51.315 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.311+0000 7f5ca0d4b640 1 Processor -- start 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.311+0000 7f5ca0d4b640 1 -- start start 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.311+0000 7f5ca0d4b640 10 monclient: init 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.311+0000 7f5ca0d4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.311+0000 7f5ca0d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9c072aa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9c0702c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9c13b0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c0a0590 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c09f000 con 0x7f5c9c0702c0 2024-09-16T04:07:51.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c0a02b0 con 0x7f5c9c072aa0 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 10 monclient(hunting): authenticate will time out at 846.818066s 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9ad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f5c9c13b0e0 legacy=0x7f5c9c13d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c99d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5c9c0702c0 0x7f5c9c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40412/0 (socket says 172.21.15.171:40412) 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c99d9b640 1 -- 172.21.15.171:0/302234309 learned_addr learned my addr 172.21.15.171:0/302234309 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377100596 0 0) 0x7f5c9c09f000 con 0x7f5c9c0702c0 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): my global_id is 4498 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c84003520 con 0x7f5c9c0702c0 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321934904 0 0) 0x7f5c9c0a02b0 con 0x7f5c9c072aa0 2024-09-16T04:07:51.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient(hunting): my global_id is 4520 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c9c09f000 con 0x7f5c9c072aa0 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1969611543 0 0) 0x7f5c84003520 con 0x7f5c9c0702c0 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 >> v1:172.21.15.148:6790/0 conn(0x7f5c9c072aa0 legacy=0x7f5c9c072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 >> v1:172.21.15.148:6789/0 conn(0x7f5c9c13b0e0 legacy=0x7f5c9c13d4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 monclient: found mon.noname-c 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c09f280 con 0x7f5c9c0702c0 2024-09-16T04:07:51.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c90001610 con 0x7f5c9c0702c0 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 5 monclient: authenticate success, global_id 4498 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: dump: 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c90001930 con 0x7f5c9c0702c0 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 1 -- 172.21.15.171:0/302234309 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c90001d00 con 0x7f5c9c0702c0 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5ca0d4b640 4 set_mon_vals no callback set 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:51.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.312+0000 7f5c9959a640 10 monclient: dump: 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 10 monclient: shutdown 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/302234309 >> v1:172.21.15.171:6789/0 conn(0x7f5c9c0702c0 legacy=0x7f5c9c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/302234309 shutdown_connections 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/302234309 >> 172.21.15.171:0/302234309 conn(0x7f5c9c0a77f0 msgr2=0x7f5c9c0a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/302234309 shutdown_connections 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.313+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/302234309 wait complete. 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient: monmap: 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:51.328 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 1 Processor -- start 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 1 -- start start 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient: init 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9c0702c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c9c072aa0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9c13b0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c121aa0 con 0x7f5c9c072aa0 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c122060 con 0x7f5c9c0702c0 2024-09-16T04:07:51.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9c1204f0 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5ca0d4b640 10 monclient(hunting): authenticate will time out at 846.820874s 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5c9a59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5c9c0702c0 0x7f5c9c136610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40420/0 (socket says 172.21.15.171:40420) 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5c9a59c640 1 -- 172.21.15.171:0/3976847067 learned_addr learned my addr 172.21.15.171:0/3976847067 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.314+0000 7f5c99d9b640 1 -- 172.21.15.171:0/3976847067 >> v1:172.21.15.148:6789/0 conn(0x7f5c9c072aa0 legacy=0x7f5c9c1ce790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538003482 0 0) 0x7f5c9c122060 con 0x7f5c9c0702c0 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): my global_id is 4501 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c7c003520 con 0x7f5c9c0702c0 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931327144 0 0) 0x7f5c9c1204f0 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): my global_id is 4523 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c9c122060 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2465519751 0 0) 0x7f5c7c003520 con 0x7f5c9c0702c0 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c9c1204f0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c94001cf0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): dump: 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.331 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1069720609 0 0) 0x7f5c9c122060 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c7c003520 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c8c004000 con 0x7f5c9c13b0e0 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:51.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient(hunting): dump: 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1696752567 0 0) 0x7f5c9c1204f0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 >> v1:172.21.15.148:6790/0 conn(0x7f5c9c13b0e0 legacy=0x7f5c9c1d1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 >> v1:172.21.15.148:6789/0 conn(0x7f5c9c072aa0 legacy=0x7f5c9c1ce790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:51.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 monclient: found mon.b 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9c120bb0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 5 monclient: authenticate success, global_id 4501 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 10 monclient: _renew_subs 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9c120050 con 0x7f5c9c0702c0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 10 monclient: _renew_subs 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c9c11f3c0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c940020a0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c94003de0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c98d99640 4 set_mon_vals no callback set 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.315+0000 7f5c8affd640 10 monclient: dump: 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.335 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5c94003f80 con 0x7f5c9c0702c0 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5ca0d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5ca0d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5ca0d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c80005080 con 0x7f5c9c0702c0 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 3889+0+0 (unknown 1330234895 0 0) 0x7f5c94047280 con 0x7f5c9c0702c0 2024-09-16T04:07:51.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.316+0000 7f5c99d9b640 10 monclient: get_auth_request con 0x7f5c7c038ef0 auth_method 0 2024-09-16T04:07:51.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.321+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5c940160d0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.321+0000 7f5c8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.321+0000 7f5c8affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:51.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.487+0000 7f5ca0d4b640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T04:07:51.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.487+0000 7f5ca0d4b640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T04:07:51.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.487+0000 7f5ca0d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.487+0000 7f5ca0d4b640 1 -- 172.21.15.171:0/3976847067 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) -- 0x7f5c80005370 con 0x7f5c9c0702c0 2024-09-16T04:07:51.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.488+0000 7f5c8affd640 1 -- 172.21.15.171:0/3976847067 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v24) ==== 149+0+0 (unknown 3442327055 0 0) 0x7f5c940159d0 con 0x7f5c9c0702c0 2024-09-16T04:07:51.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.488+0000 7f5c8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-16T04:07:51.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.488+0000 7f5c8affd640 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-16T04:07:51.494 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-16T04:07:51.494 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:51.495 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.574+0000 7fd80139f640 10 monclient: get_monmap_and_config 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.574+0000 7fd80139f640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.574+0000 7fd80139f640 10 monclient: monmap: 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:51.576387+0000 2024-09-16T04:07:51.577 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:51.576387+0000 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 1 Processor -- start 2024-09-16T04:07:51.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 1 -- start start 2024-09-16T04:07:51.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient: init 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient(hunting): picked mon.noname-c con 0x7fd7fc136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient(hunting): picked mon.noname-a con 0x7fd7fc137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient(hunting): picked mon.noname-b con 0x7fd7fc1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc09b2c0 con 0x7fd7fc137140 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc09b5a0 con 0x7fd7fc136d60 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc0a6640 con 0x7fd7fc1375a0 2024-09-16T04:07:51.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd80139f640 10 monclient(hunting): authenticate will time out at 847.081886s 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd7f3fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd7fc137140 legacy=0x7fd7fc13bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd7faffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd7fc136d60 0x7fd7fc09bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40434/0 (socket says 172.21.15.171:40434) 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.575+0000 7fd7faffd640 1 -- 172.21.15.171:0/3143383000 learned_addr learned my addr 172.21.15.171:0/3143383000 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769156596 0 0) 0x7fd7fc09b5a0 con 0x7fd7fc136d60 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): my global_id is 4504 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7e8003520 con 0x7fd7fc136d60 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260896996 0 0) 0x7fd7fc0a6640 con 0x7fd7fc1375a0 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient(hunting): my global_id is 4526 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7fc09b5a0 con 0x7fd7fc1375a0 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 240643267 0 0) 0x7fd7e8003520 con 0x7fd7fc136d60 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 >> v1:172.21.15.148:6790/0 conn(0x7fd7fc1375a0 legacy=0x7fd7fc13f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 >> v1:172.21.15.148:6789/0 conn(0x7fd7fc137140 legacy=0x7fd7fc13bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 monclient: found mon.noname-c 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc143b20 con 0x7fd7fc136d60 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd80139f640 5 monclient: authenticate success, global_id 4504 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd7ec001610 con 0x7fd7fc136d60 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd80139f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: dump: 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd80139f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7ec0018c0 con 0x7fd7fc136d60 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 1 -- 172.21.15.171:0/3143383000 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd7ec001c90 con 0x7fd7fc136d60 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd7fa7fc640 10 monclient: dump: 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.591 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd80139f640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.576+0000 7fd80139f640 4 set_mon_vals no callback set 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 10 monclient: shutdown 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 1 -- 172.21.15.171:0/3143383000 >> v1:172.21.15.171:6789/0 conn(0x7fd7fc136d60 legacy=0x7fd7fc09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 1 -- 172.21.15.171:0/3143383000 shutdown_connections 2024-09-16T04:07:51.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 1 -- 172.21.15.171:0/3143383000 >> 172.21.15.171:0/3143383000 conn(0x7fd7fc0a9ae0 msgr2=0x7fd7fc0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:51.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 1 -- 172.21.15.171:0/3143383000 shutdown_connections 2024-09-16T04:07:51.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 1 -- 172.21.15.171:0/3143383000 wait complete. 2024-09-16T04:07:51.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.577+0000 7fd80139f640 10 monclient: monmap: 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:51.594 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 1 Processor -- start 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 1 -- start start 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient: init 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient(hunting): picked mon.noname-b con 0x7fd7fc136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient(hunting): picked mon.noname-a con 0x7fd7fc137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient(hunting): picked mon.noname-c con 0x7fd7fc1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc0a87c0 con 0x7fd7fc137140 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc12f300 con 0x7fd7fc136d60 2024-09-16T04:07:51.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7fc12ed40 con 0x7fd7fc1375a0 2024-09-16T04:07:51.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd80139f640 10 monclient(hunting): authenticate will time out at 847.084639s 2024-09-16T04:07:51.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7faffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd7fc136d60 0x7fd7fc129c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40440/0 (socket says 172.21.15.171:40440) 2024-09-16T04:07:51.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7faffd640 1 -- 172.21.15.171:0/2069654984 learned_addr learned my addr 172.21.15.171:0/2069654984 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:51.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f3fff640 1 -- 172.21.15.171:0/2069654984 >> v1:172.21.15.148:6789/0 conn(0x7fd7fc137140 legacy=0x7fd7fc1d0ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2665899456 0 0) 0x7fd7fc12f300 con 0x7fd7fc136d60 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f37fe640 10 monclient(hunting): my global_id is 4507 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.578+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7dc003520 con 0x7fd7fc136d60 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2205725821 0 0) 0x7fd7dc003520 con 0x7fd7fc136d60 2024-09-16T04:07:51.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd7fc12f300 con 0x7fd7fc136d60 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd7ec001c10 con 0x7fd7fc136d60 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): dump: 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.598 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86537934 0 0) 0x7fd7fc12ed40 con 0x7fd7fc1375a0 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient(hunting): my global_id is 4529 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7dc003520 con 0x7fd7fc1375a0 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2643104486 0 0) 0x7fd7fc12f300 con 0x7fd7fc136d60 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 >> v1:172.21.15.148:6790/0 conn(0x7fd7fc1375a0 legacy=0x7fd7fc1d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 >> v1:172.21.15.148:6789/0 conn(0x7fd7fc137140 legacy=0x7fd7fc1d0ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 monclient: found mon.b 2024-09-16T04:07:51.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc09f7f0 con 0x7fd7fc136d60 2024-09-16T04:07:51.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 5 monclient: authenticate success, global_id 4507 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 10 monclient: _renew_subs 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd7fc09e7c0 con 0x7fd7fc136d60 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 10 monclient: _renew_subs 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd7fc12a330 con 0x7fd7fc136d60 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd7ec0018c0 con 0x7fd7fc136d60 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd7ec004ed0 con 0x7fd7fc136d60 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f9ffb640 4 set_mon_vals no callback set 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.579+0000 7fd7f37fe640 10 monclient: dump: 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:51.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd7ec005070 con 0x7fd7fc136d60 2024-09-16T04:07:51.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd80139f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd80139f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd80139f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e0005080 con 0x7fd7fc136d60 2024-09-16T04:07:51.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(24..24 src has 1..24) ==== 3889+0+0 (unknown 1330234895 0 0) 0x7fd7ec046f50 con 0x7fd7fc136d60 2024-09-16T04:07:51.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.580+0000 7fd7f3fff640 10 monclient: get_auth_request con 0x7fd7dc039010 auth_method 0 2024-09-16T04:07:51.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.584+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd7ec015e40 con 0x7fd7fc136d60 2024-09-16T04:07:51.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.584+0000 7fd7f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.584+0000 7fd7f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:51.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.752+0000 7fd80139f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T04:07:51.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.752+0000 7fd80139f640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T04:07:51.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.752+0000 7fd80139f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:51.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:51.752+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) -- 0x7fd7e0005370 con 0x7fd7fc136d60 2024-09-16T04:07:52.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.064+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v25) ==== 143+0+0 (unknown 3047906603 0 0) 0x7fd7ec015740 con 0x7fd7fc136d60 2024-09-16T04:07:52.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.064+0000 7fd7f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T04:07:52.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.064+0000 7fd7f37fe640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-16T04:07:52.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.166+0000 7fd80139f640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T04:07:52.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.166+0000 7fd80139f640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T04:07:52.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.166+0000 7fd80139f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:52.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:52.166+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) -- 0x7fd7e0000c20 con 0x7fd7fc136d60 2024-09-16T04:07:53.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.066+0000 7fd7f37fe640 1 -- 172.21.15.171:0/2069654984 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v26) ==== 143+0+0 (unknown 4160523098 0 0) 0x7fd7ec013aa0 con 0x7fd7fc136d60 2024-09-16T04:07:53.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.066+0000 7fd7f37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-16T04:07:53.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.066+0000 7fd7f37fe640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-16T04:07:53.070 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 3 pg_num_min to 0 2024-09-16T04:07:53.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 >> v1:172.21.15.171:6812/7327273 conn(0x7fd7dc039010 legacy=0x7fd7dc03b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 10 monclient: shutdown 2024-09-16T04:07:53.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:53.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 >> v1:172.21.15.171:6789/0 conn(0x7fd7fc136d60 legacy=0x7fd7fc129c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 shutdown_connections 2024-09-16T04:07:53.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 >> 172.21.15.171:0/2069654984 conn(0x7fd7fc0a9ae0 msgr2=0x7fd7fc072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:53.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 shutdown_connections 2024-09-16T04:07:53.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.069+0000 7fd80139f640 1 -- 172.21.15.171:0/2069654984 wait complete. 2024-09-16T04:07:53.085 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-16T04:07:53.085 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:53.085 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.168+0000 7f1999b33640 10 monclient: get_monmap_and_config 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.168+0000 7f1999b33640 10 monclient: build_initial_monmap 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.168+0000 7f1999b33640 10 monclient: monmap: 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:53.170014+0000 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:53.170014+0000 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:53.171 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.172 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:53.172 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:53.172 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:53.172 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 1 Processor -- start 2024-09-16T04:07:53.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 1 -- start start 2024-09-16T04:07:53.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient: init 2024-09-16T04:07:53.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient(hunting): picked mon.noname-a con 0x7f1994136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient(hunting): picked mon.noname-b con 0x7f1994139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient(hunting): picked mon.noname-c con 0x7f199413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19940a1340 con 0x7f1994136b60 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1994137650 con 0x7f199413d620 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19940a0aa0 con 0x7f1994139b10 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:53.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1999b33640 10 monclient(hunting): authenticate will time out at 848.675867s 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f19937fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f1994136b60 legacy=0x7f1994136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1993fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f199413d620 0x7f199413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40446/0 (socket says 172.21.15.171:40446) 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.169+0000 7f1993fff640 1 -- 172.21.15.171:0/80689880 learned_addr learned my addr 172.21.15.171:0/80689880 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077086170 0 0) 0x7f1994137650 con 0x7f199413d620 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): my global_id is 4510 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1980003520 con 0x7f199413d620 2024-09-16T04:07:53.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 179550421 0 0) 0x7f19940a0aa0 con 0x7f1994139b10 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient(hunting): my global_id is 4532 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1994137650 con 0x7f1994139b10 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2096442457 0 0) 0x7f1980003520 con 0x7f199413d620 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 >> v1:172.21.15.148:6790/0 conn(0x7f1994139b10 legacy=0x7f199413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 >> v1:172.21.15.148:6789/0 conn(0x7f1994136b60 legacy=0x7f1994136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 monclient: found mon.noname-c 2024-09-16T04:07:53.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19940a0550 con 0x7f199413d620 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f198c001990 con 0x7f199413d620 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: dump: 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.185 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1999b33640 5 monclient: authenticate success, global_id 4510 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1999b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:53.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f198c001cb0 con 0x7f199413d620 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 1 -- 172.21.15.171:0/80689880 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f198c002080 con 0x7f199413d620 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1999b33640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1999b33640 4 set_mon_vals no callback set 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.170+0000 7f1992ffd640 10 monclient: dump: 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.187 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 10 monclient: shutdown 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 1 -- 172.21.15.171:0/80689880 >> v1:172.21.15.171:6789/0 conn(0x7f199413d620 legacy=0x7f199413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 1 -- 172.21.15.171:0/80689880 shutdown_connections 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 1 -- 172.21.15.171:0/80689880 >> 172.21.15.171:0/80689880 conn(0x7f19940a98b0 msgr2=0x7f19940a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 1 -- 172.21.15.171:0/80689880 shutdown_connections 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 1 -- 172.21.15.171:0/80689880 wait complete. 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 10 monclient: build_initial_monmap 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.171+0000 7f1999b33640 10 monclient: monmap: 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:53.189 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 1 Processor -- start 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 1 -- start start 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient: init 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:53.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient(hunting): picked mon.noname-a con 0x7f1994136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient(hunting): picked mon.noname-c con 0x7f1994139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient(hunting): picked mon.noname-b con 0x7f199413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1994120d80 con 0x7f1994136b60 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19941cc5c0 con 0x7f199413d620 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19940a6f50 con 0x7f1994139b10 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1999b33640 10 monclient(hunting): authenticate will time out at 848.678624s 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f19937fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f1994136b60 legacy=0x7f19941cb7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1993fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f199413d620 0x7f19941d3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40454/0 (socket says 172.21.15.171:40454) 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1993fff640 1 -- 172.21.15.171:0/2189731627 learned_addr learned my addr 172.21.15.171:0/2189731627 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:53.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1029888557 0 0) 0x7f19941cc5c0 con 0x7f199413d620 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): my global_id is 4513 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1974003550 con 0x7f199413d620 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2848673046 0 0) 0x7f19940a6f50 con 0x7f1994139b10 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 10 monclient(hunting): my global_id is 4535 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.172+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19941cc5c0 con 0x7f1994139b10 2024-09-16T04:07:53.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 391268217 0 0) 0x7f1974003550 con 0x7f199413d620 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19940a6f50 con 0x7f199413d620 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f198c001950 con 0x7f199413d620 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient(hunting): dump: 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:53.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4020878662 0 0) 0x7f19940a6f50 con 0x7f199413d620 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 >> v1:172.21.15.148:6790/0 conn(0x7f1994139b10 legacy=0x7f19941cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 >> v1:172.21.15.148:6789/0 conn(0x7f1994136b60 legacy=0x7f19941cb7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 monclient: found mon.b 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19940a8a90 con 0x7f199413d620 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: _finish_auth 0 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:07:53.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 5 monclient: authenticate success, global_id 4513 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 10 monclient: _renew_subs 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f19940a0240 con 0x7f199413d620 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 10 monclient: _renew_subs 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1999b33640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f199409e2a0 con 0x7f199413d620 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f198c005040 con 0x7f199413d620 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f198c005390 con 0x7f199413d620 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:53.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f1990ff9640 10 monclient: dump: 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.173+0000 7f19927fc640 4 set_mon_vals no callback set 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f198c0107c0 con 0x7f199413d620 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f1999b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f1999b33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f1999b33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f1999b33640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1978005080 con 0x7f199413d620 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(26..26 src has 1..26) ==== 3873+0+0 (unknown 2441009938 0 0) 0x7f198c047140 con 0x7f199413d620 2024-09-16T04:07:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.174+0000 7f19937fe640 10 monclient: get_auth_request con 0x7f1974038bf0 auth_method 0 2024-09-16T04:07:53.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.178+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f198c016180 con 0x7f199413d620 2024-09-16T04:07:53.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.178+0000 7f1990ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:53.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.178+0000 7f1990ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1999b33640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1999b33640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1999b33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1999b33640 1 -- 172.21.15.171:0/2189731627 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) -- 0x7f1978005b80 con 0x7f199413d620 2024-09-16T04:07:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1990ff9640 1 -- 172.21.15.171:0/2189731627 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v26) ==== 158+0+0 (unknown 222113961 0 0) 0x7f198c015a80 con 0x7f199413d620 2024-09-16T04:07:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1990ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-16T04:07:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.347+0000 7f1990ff9640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T04:07:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-16T04:07:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-16T04:07:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.431+0000 7fa7a2477640 10 monclient: get_monmap_and_config 2024-09-16T04:07:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.431+0000 7fa7a2477640 10 monclient: build_initial_monmap 2024-09-16T04:07:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.431+0000 7fa7a2477640 10 monclient: monmap: 2024-09-16T04:07:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:53.433294+0000 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:53.433294+0000 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:53.435 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 1 Processor -- start 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 1 -- start start 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient: init 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient(hunting): picked mon.noname-a con 0x7fa79c09b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:53.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient(hunting): picked mon.noname-c con 0x7fa79c136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient(hunting): picked mon.noname-b con 0x7fa79c13d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c073a20 con 0x7fa79c09b920 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c099920 con 0x7fa79c136da0 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c099c00 con 0x7fa79c13d320 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa7a2477640 10 monclient(hunting): authenticate will time out at 848.938770s 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa79b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa79c136da0 0x7fa79c13bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40466/0 (socket says 172.21.15.171:40466) 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa79b7fe640 1 -- 172.21.15.171:0/1753256413 learned_addr learned my addr 172.21.15.171:0/1753256413 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.432+0000 7fa79bfff640 1 -- 172.21.15.171:0/1753256413 >> v1:172.21.15.148:6789/0 conn(0x7fa79c09b920 legacy=0x7fa79c09bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3631134925 0 0) 0x7fa79c099920 con 0x7fa79c136da0 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient(hunting): my global_id is 4516 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa788003520 con 0x7fa79c136da0 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3535217847 0 0) 0x7fa788003520 con 0x7fa79c136da0 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 >> v1:172.21.15.148:6790/0 conn(0x7fa79c13d320 legacy=0x7fa79c13f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 >> v1:172.21.15.148:6789/0 conn(0x7fa79c09b920 legacy=0x7fa79c09bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 monclient: found mon.noname-c 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c099ee0 con 0x7fa79c136da0 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: _finish_auth 0 2024-09-16T04:07:53.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa784001610 con 0x7fa79c136da0 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: dump: 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.445 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 5 monclient: authenticate success, global_id 4516 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa784001900 con 0x7fa79c136da0 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 1 -- 172.21.15.171:0/1753256413 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa784001d10 con 0x7fa79c136da0 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 4 set_mon_vals no callback set 2024-09-16T04:07:53.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa79affd640 10 monclient: dump: 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.447 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 10 monclient: shutdown 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 1 -- 172.21.15.171:0/1753256413 >> v1:172.21.15.171:6789/0 conn(0x7fa79c136da0 legacy=0x7fa79c13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 1 -- 172.21.15.171:0/1753256413 shutdown_connections 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 1 -- 172.21.15.171:0/1753256413 >> 172.21.15.171:0/1753256413 conn(0x7fa79c0a2150 msgr2=0x7fa79c0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.433+0000 7fa7a2477640 1 -- 172.21.15.171:0/1753256413 shutdown_connections 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.434+0000 7fa7a2477640 1 -- 172.21.15.171:0/1753256413 wait complete. 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.434+0000 7fa7a2477640 10 monclient: build_initial_monmap 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.434+0000 7fa7a2477640 10 monclient: monmap: 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:53.448 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.434+0000 7fa7a2477640 1 Processor -- start 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 1 -- start start 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient: init 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:53.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient(hunting): picked mon.noname-b con 0x7fa79c09b920 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient(hunting): picked mon.noname-a con 0x7fa79c136da0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient(hunting): picked mon.noname-c con 0x7fa79c13d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c12c380 con 0x7fa79c136da0 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c13caf0 con 0x7fa79c09b920 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c13c650 con 0x7fa79c13d320 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa7a2477640 10 monclient(hunting): authenticate will time out at 848.941477s 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa79bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa79c09b920 0x7fa79c133910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40470/0 (socket says 172.21.15.171:40470) 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa79bfff640 1 -- 172.21.15.171:0/3224003050 learned_addr learned my addr 172.21.15.171:0/3224003050 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa79b7fe640 1 -- 172.21.15.171:0/3224003050 >> v1:172.21.15.148:6789/0 conn(0x7fa79c136da0 legacy=0x7fa79c1d08c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:53.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3424710075 0 0) 0x7fa79c13caf0 con 0x7fa79c09b920 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): my global_id is 4519 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa77c003520 con 0x7fa79c09b920 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3958415890 0 0) 0x7fa79c13c650 con 0x7fa79c13d320 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): my global_id is 4541 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa79c13caf0 con 0x7fa79c13d320 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3952272936 0 0) 0x7fa77c003520 con 0x7fa79c09b920 2024-09-16T04:07:53.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa79c13c650 con 0x7fa79c09b920 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa78c0041d0 con 0x7fa79c09b920 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): dump: 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.452 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.435+0000 7fa798ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2164836709 0 0) 0x7fa79c13caf0 con 0x7fa79c13d320 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa77c003520 con 0x7fa79c13d320 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa794001b70 con 0x7fa79c13d320 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient(hunting): dump: 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.453 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1894085261 0 0) 0x7fa79c13c650 con 0x7fa79c09b920 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 >> v1:172.21.15.148:6790/0 conn(0x7fa79c13d320 legacy=0x7fa79c1d3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 >> v1:172.21.15.148:6789/0 conn(0x7fa79c136da0 legacy=0x7fa79c1d08c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 monclient: found mon.b 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c12c0a0 con 0x7fa79c09b920 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: _finish_auth 0 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 5 monclient: authenticate success, global_id 4519 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 10 monclient: _renew_subs 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c12ae10 con 0x7fa79c09b920 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 10 monclient: _renew_subs 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa79c134200 con 0x7fa79c09b920 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa78c004430 con 0x7fa79c09b920 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:53.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa78c004780 con 0x7fa79c09b920 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa79a7fc640 4 set_mon_vals no callback set 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 10 monclient: dump: 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:53.456 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.436+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa78c0049a0 con 0x7fa79c09b920 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.437+0000 7fa7a2477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.437+0000 7fa7a2477640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.437+0000 7fa7a2477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.437+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa780005080 con 0x7fa79c09b920 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.437+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(26..26 src has 1..26) ==== 3873+0+0 (unknown 2441009938 0 0) 0x7fa78c046e80 con 0x7fa79c09b920 2024-09-16T04:07:53.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.437+0000 7fa79b7fe640 10 monclient: get_auth_request con 0x7fa77c038ef0 auth_method 0 2024-09-16T04:07:53.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.441+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa78c015c60 con 0x7fa79c09b920 2024-09-16T04:07:53.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.441+0000 7fa798ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:53.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.441+0000 7fa798ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:53.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.608+0000 7fa7a2477640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T04:07:53.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.608+0000 7fa7a2477640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T04:07:53.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.608+0000 7fa7a2477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:53.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:53.608+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) -- 0x7fa780005370 con 0x7fa79c09b920 2024-09-16T04:07:54.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.074+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v27) ==== 143+0+0 (unknown 2323565336 0 0) 0x7fa78c015560 con 0x7fa79c09b920 2024-09-16T04:07:54.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.074+0000 7fa798ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T04:07:54.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.074+0000 7fa798ff9640 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-16T04:07:54.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.176+0000 7fa7a2477640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T04:07:54.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.176+0000 7fa7a2477640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T04:07:54.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.176+0000 7fa7a2477640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:54.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:54.176+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) -- 0x7fa780000c20 con 0x7fa79c09b920 2024-09-16T04:07:55.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.078+0000 7fa798ff9640 1 -- 172.21.15.171:0/3224003050 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v28) ==== 143+0+0 (unknown 820237781 0 0) 0x7fa78c0198b0 con 0x7fa79c09b920 2024-09-16T04:07:55.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.078+0000 7fa798ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-16T04:07:55.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.078+0000 7fa798ff9640 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-16T04:07:55.082 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 3 pg_num_max to 0 2024-09-16T04:07:55.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 >> v1:172.21.15.171:6812/7327273 conn(0x7fa77c038ef0 legacy=0x7fa77c03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 10 monclient: shutdown 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 >> v1:172.21.15.171:6789/0 conn(0x7fa79c09b920 legacy=0x7fa79c133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 shutdown_connections 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 >> 172.21.15.171:0/3224003050 conn(0x7fa79c0a2150 msgr2=0x7fa79c0a0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.081+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 shutdown_connections 2024-09-16T04:07:55.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.082+0000 7fa7a2477640 1 -- 172.21.15.171:0/3224003050 wait complete. 2024-09-16T04:07:55.096 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-16T04:07:55.096 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:55.096 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.178+0000 7fbb18ad7640 10 monclient: get_monmap_and_config 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.178+0000 7fbb18ad7640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.178+0000 7fbb18ad7640 10 monclient: monmap: 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:55.180232+0000 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:55.180232+0000 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.181 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.182 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:55.182 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:55.182 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:55.182 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 1 Processor -- start 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 1 -- start start 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient: init 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient(hunting): picked mon.noname-b con 0x7fbb141338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient(hunting): picked mon.noname-a con 0x7fbb1409aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient(hunting): picked mon.noname-c con 0x7fbb1413d240 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb141343d0 con 0x7fbb1409aa20 2024-09-16T04:07:55.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb14075db0 con 0x7fbb1413d240 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1409b1e0 con 0x7fbb141338e0 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb18ad7640 10 monclient(hunting): authenticate will time out at 850.685843s 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb12d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbb1413d240 0x7fbb1413f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40472/0 (socket says 172.21.15.171:40472) 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb12d9d640 1 -- 172.21.15.171:0/725343393 learned_addr learned my addr 172.21.15.171:0/725343393 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.179+0000 7fbb11d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbb1409aa20 legacy=0x7fbb1413bb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171295874 0 0) 0x7fbb14075db0 con 0x7fbb1413d240 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): my global_id is 4522 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb00003520 con 0x7fbb1413d240 2024-09-16T04:07:55.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2811549384 0 0) 0x7fbb1409b1e0 con 0x7fbb141338e0 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient(hunting): my global_id is 4544 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb14075db0 con 0x7fbb141338e0 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3302280549 0 0) 0x7fbb00003520 con 0x7fbb1413d240 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 >> v1:172.21.15.148:6790/0 conn(0x7fbb141338e0 legacy=0x7fbb14133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 >> v1:172.21.15.148:6789/0 conn(0x7fbb1409aa20 legacy=0x7fbb1413bb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 monclient: found mon.noname-c 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1409b4c0 con 0x7fbb1413d240 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb18ad7640 5 monclient: authenticate success, global_id 4522 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb18ad7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbb0c001990 con 0x7fbb1413d240 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: dump: 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb18ad7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb0c001c40 con 0x7fbb1413d240 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 1 -- 172.21.15.171:0/725343393 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbb0c002010 con 0x7fbb1413d240 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb18ad7640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb18ad7640 4 set_mon_vals no callback set 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb1159a640 10 monclient: dump: 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.180+0000 7fbb18ad7640 10 monclient: shutdown 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 1 -- 172.21.15.171:0/725343393 >> v1:172.21.15.171:6789/0 conn(0x7fbb1413d240 legacy=0x7fbb1413f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 1 -- 172.21.15.171:0/725343393 shutdown_connections 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 1 -- 172.21.15.171:0/725343393 >> 172.21.15.171:0/725343393 conn(0x7fbb140a28d0 msgr2=0x7fbb140a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 1 -- 172.21.15.171:0/725343393 shutdown_connections 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 1 -- 172.21.15.171:0/725343393 wait complete. 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.181+0000 7fbb18ad7640 10 monclient: monmap: 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:55.195 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 1 Processor -- start 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 1 -- start start 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient: init 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient(hunting): picked mon.noname-b con 0x7fbb1409aa20 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient(hunting): picked mon.noname-a con 0x7fbb141338e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient(hunting): picked mon.noname-c con 0x7fbb1413d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb141cc350 con 0x7fbb141338e0 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1413c260 con 0x7fbb1409aa20 2024-09-16T04:07:55.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb1411df80 con 0x7fbb1413d240 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb18ad7640 10 monclient(hunting): authenticate will time out at 850.688731s 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb1259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbb1409aa20 0x7fbb141325e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40478/0 (socket says 172.21.15.171:40478) 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb1259c640 1 -- 172.21.15.171:0/1443690557 learned_addr learned my addr 172.21.15.171:0/1443690557 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb11d9b640 1 -- 172.21.15.171:0/1443690557 >> v1:172.21.15.148:6789/0 conn(0x7fbb141338e0 legacy=0x7fbb141d0750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.182+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1865500808 0 0) 0x7fbb1413c260 con 0x7fbb1409aa20 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): my global_id is 4525 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaf4003520 con 0x7fbb1409aa20 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3068644673 0 0) 0x7fbaf4003520 con 0x7fbb1409aa20 2024-09-16T04:07:55.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb1413c260 con 0x7fbb1409aa20 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbb04001970 con 0x7fbb1409aa20 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): dump: 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.198 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561457413 0 0) 0x7fbb1411df80 con 0x7fbb1413d240 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient(hunting): my global_id is 4547 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbaf4003520 con 0x7fbb1413d240 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2051219884 0 0) 0x7fbb1413c260 con 0x7fbb1409aa20 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 >> v1:172.21.15.148:6790/0 conn(0x7fbb1413d240 legacy=0x7fbb141d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 >> v1:172.21.15.148:6789/0 conn(0x7fbb141338e0 legacy=0x7fbb141d0750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 monclient: found mon.b 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1409ffa0 con 0x7fbb1409aa20 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 5 monclient: authenticate success, global_id 4525 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 10 monclient: _renew_subs 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb140a59d0 con 0x7fbb1409aa20 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 10 monclient: _renew_subs 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb18ad7640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb14132cf0 con 0x7fbb1409aa20 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb04003940 con 0x7fbb1409aa20 2024-09-16T04:07:55.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbb04003d10 con 0x7fbb1409aa20 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb10d99640 4 set_mon_vals no callback set 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.183+0000 7fbb0affd640 10 monclient: dump: 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.201 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbb04003eb0 con 0x7fbb1409aa20 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb18ad7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb18ad7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb18ad7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb18ad7640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf8005080 con 0x7fbb1409aa20 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(28..28 src has 1..28) ==== 3857+0+0 (unknown 1265567808 0 0) 0x7fbb04046fd0 con 0x7fbb1409aa20 2024-09-16T04:07:55.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.184+0000 7fbb11d9b640 10 monclient: get_auth_request con 0x7fbaf4038fc0 auth_method 0 2024-09-16T04:07:55.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.189+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbb04015dc0 con 0x7fbb1409aa20 2024-09-16T04:07:55.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.189+0000 7fbb0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.189+0000 7fbb0affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.353+0000 7fbb18ad7640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.353+0000 7fbb18ad7640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.353+0000 7fbb18ad7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.353+0000 7fbb18ad7640 1 -- 172.21.15.171:0/1443690557 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) -- 0x7fbaf8005b80 con 0x7fbb1409aa20 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.354+0000 7fbb0affd640 1 -- 172.21.15.171:0/1443690557 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v28) ==== 158+0+0 (unknown 995013874 0 0) 0x7fbb040156c0 con 0x7fbb1409aa20 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.354+0000 7fbb0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.354+0000 7fbb0affd640 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T04:07:55.357 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-16T04:07:55.361 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:55.361 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-16T04:07:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.439+0000 7f428cb66640 10 monclient: get_monmap_and_config 2024-09-16T04:07:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.439+0000 7f428cb66640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.439+0000 7f428cb66640 10 monclient: monmap: 2024-09-16T04:07:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:55.441299+0000 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:55.441299+0000 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:55.443 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 1 Processor -- start 2024-09-16T04:07:55.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 1 -- start start 2024-09-16T04:07:55.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient: init 2024-09-16T04:07:55.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient(hunting): picked mon.noname-a con 0x7f42880a0ae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient(hunting): picked mon.noname-c con 0x7f428809f320 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient(hunting): picked mon.noname-b con 0x7f428813d2c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42880737b0 con 0x7f42880a0ae0 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428806c6d0 con 0x7f428809f320 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428806d980 con 0x7f428813d2c0 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428cb66640 10 monclient(hunting): authenticate will time out at 850.946779s 2024-09-16T04:07:55.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f427ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f428809f320 0x7f428813bba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40492/0 (socket says 172.21.15.171:40492) 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f428659c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f42880a0ae0 legacy=0x7f42880a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.440+0000 7f427ffff640 1 -- 172.21.15.171:0/4147581927 learned_addr learned my addr 172.21.15.171:0/4147581927 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675214665 0 0) 0x7f428806c6d0 con 0x7f428809f320 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient(hunting): my global_id is 4528 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4274003520 con 0x7f428809f320 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 584145578 0 0) 0x7f4274003520 con 0x7f428809f320 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 >> v1:172.21.15.148:6790/0 conn(0x7f428813d2c0 legacy=0x7f428813f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 >> v1:172.21.15.148:6789/0 conn(0x7f42880a0ae0 legacy=0x7f42880a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:55.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 monclient: found mon.noname-c 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428806c950 con 0x7f428809f320 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 5 monclient: authenticate success, global_id 4528 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4270001610 con 0x7f428809f320 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:55.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: dump: 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4270001970 con 0x7f428809f320 2024-09-16T04:07:55.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 1 -- 172.21.15.171:0/4147581927 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4270001d40 con 0x7f428809f320 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 4 set_mon_vals no callback set 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f4285d9b640 10 monclient: dump: 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.459 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 10 monclient: shutdown 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 1 -- 172.21.15.171:0/4147581927 >> v1:172.21.15.171:6789/0 conn(0x7f428809f320 legacy=0x7f428813bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 1 -- 172.21.15.171:0/4147581927 shutdown_connections 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.441+0000 7f428cb66640 1 -- 172.21.15.171:0/4147581927 >> 172.21.15.171:0/4147581927 conn(0x7f42880a77f0 msgr2=0x7f42880a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.442+0000 7f428cb66640 1 -- 172.21.15.171:0/4147581927 shutdown_connections 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.442+0000 7f428cb66640 1 -- 172.21.15.171:0/4147581927 wait complete. 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.442+0000 7f428cb66640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.442+0000 7f428cb66640 10 monclient: monmap: 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 1 Processor -- start 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 1 -- start start 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient: init 2024-09-16T04:07:55.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient(hunting): picked mon.noname-a con 0x7f428809f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient(hunting): picked mon.noname-b con 0x7f42880a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient(hunting): picked mon.noname-c con 0x7f428813d2c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4288127240 con 0x7f428809f320 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4288125df0 con 0x7f42880a0ae0 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4288126070 con 0x7f428813d2c0 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428659c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f428809f320 legacy=0x7f428813bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f428cb66640 10 monclient(hunting): authenticate will time out at 850.949560s 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42880a0ae0 0x7f42881d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40504/0 (socket says 172.21.15.171:40504) 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427ffff640 1 -- 172.21.15.171:0/2178226635 learned_addr learned my addr 172.21.15.171:0/2178226635 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:55.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026856714 0 0) 0x7f4288125df0 con 0x7f42880a0ae0 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): my global_id is 4531 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4268003520 con 0x7f42880a0ae0 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042792534 0 0) 0x7f4288126070 con 0x7f428813d2c0 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 10 monclient(hunting): my global_id is 4553 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4288125df0 con 0x7f428813d2c0 2024-09-16T04:07:55.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2152220945 0 0) 0x7f4268003520 con 0x7f42880a0ae0 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4288126070 con 0x7f42880a0ae0 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.443+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42700015d0 con 0x7f42880a0ae0 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient(hunting): dump: 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.464 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3757793043 0 0) 0x7f4288126070 con 0x7f42880a0ae0 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 >> v1:172.21.15.148:6790/0 conn(0x7f428813d2c0 legacy=0x7f42881d4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 >> v1:172.21.15.148:6789/0 conn(0x7f428809f320 legacy=0x7f428813bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 monclient: found mon.b 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4288127520 con 0x7f42880a0ae0 2024-09-16T04:07:55.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 5 monclient: authenticate success, global_id 4531 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 10 monclient: _renew_subs 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42880a2130 con 0x7f42880a0ae0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 10 monclient: _renew_subs 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4288124880 con 0x7f42880a0ae0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4270004cc0 con 0x7f42880a0ae0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4270004e70 con 0x7f42880a0ae0 2024-09-16T04:07:55.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f428559a640 4 set_mon_vals no callback set 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 10 monclient: dump: 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:55.467 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.444+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f42700102a0 con 0x7f42880a0ae0 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.445+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(28..28 src has 1..28) ==== 3857+0+0 (unknown 1265567808 0 0) 0x7f4270046f00 con 0x7f42880a0ae0 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.445+0000 7f428659c640 10 monclient: get_auth_request con 0x7f4268039130 auth_method 0 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.445+0000 7f428cb66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.445+0000 7f428cb66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.445+0000 7f428cb66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.445+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f426c005080 con 0x7f42880a0ae0 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.449+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4270015db0 con 0x7f42880a0ae0 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.449+0000 7f427effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.449+0000 7f427effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:55.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.615+0000 7f428cb66640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T04:07:55.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.615+0000 7f428cb66640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T04:07:55.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.615+0000 7f428cb66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:55.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:55.615+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) -- 0x7f426c005370 con 0x7f42880a0ae0 2024-09-16T04:07:56.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.081+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v29) ==== 176+0+0 (unknown 1805821260 0 0) 0x7f42700156b0 con 0x7f42880a0ae0 2024-09-16T04:07:56.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.081+0000 7f427effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T04:07:56.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.082+0000 7f427effd640 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-16T04:07:56.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.180+0000 7f428cb66640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.180+0000 7f428cb66640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.180+0000 7f428cb66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.180+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) -- 0x7f426c005ab0 con 0x7f42880a0ae0 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.181+0000 7f427effd640 1 -- 172.21.15.171:0/2178226635 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v29) ==== 183+0+0 (unknown 2197451723 0 0) 0x7f427001e200 con 0x7f42880a0ae0 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.181+0000 7f427effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.181+0000 7f427effd640 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-16T04:07:56.184 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'testpool1234' does not exist 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.182+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 >> v1:172.21.15.171:6812/7327273 conn(0x7f4268039130 legacy=0x7f426803b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.182+0000 7f428cb66640 10 monclient: shutdown 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.182+0000 7f428cb66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.182+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 >> v1:172.21.15.171:6789/0 conn(0x7f42880a0ae0 legacy=0x7f42881d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.183+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 shutdown_connections 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.183+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 >> 172.21.15.171:0/2178226635 conn(0x7f42880a77f0 msgr2=0x7f42880a5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.183+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 shutdown_connections 2024-09-16T04:07:56.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.183+0000 7f428cb66640 1 -- 172.21.15.171:0/2178226635 wait complete. 2024-09-16T04:07:56.195 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T04:07:56.195 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:07:56.195 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-16T04:07:56.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.275+0000 7fd348cc9640 10 monclient: get_monmap_and_config 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.275+0000 7fd348cc9640 10 monclient: build_initial_monmap 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.275+0000 7fd348cc9640 10 monclient: monmap: 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:56.276882+0000 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:56.276882+0000 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:56.278 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.275+0000 7fd348cc9640 1 Processor -- start 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.275+0000 7fd348cc9640 1 -- start start 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.275+0000 7fd348cc9640 10 monclient: init 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient(hunting): picked mon.noname-a con 0x7fd344136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient(hunting): picked mon.noname-b con 0x7fd344139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient(hunting): picked mon.noname-c con 0x7fd34413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3440a1340 con 0x7fd344136b60 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd344137650 con 0x7fd34413d620 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3440a0aa0 con 0x7fd344139b10 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient(hunting): authenticate will time out at 851.782206s 2024-09-16T04:07:56.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd342d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd34413d620 0x7fd34413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40516/0 (socket says 172.21.15.171:40516) 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd342d9d640 1 -- 172.21.15.171:0/4154821298 learned_addr learned my addr 172.21.15.171:0/4154821298 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd34259c640 1 -- 172.21.15.171:0/4154821298 >> v1:172.21.15.148:6789/0 conn(0x7fd344136b60 legacy=0x7fd344136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 162383300 0 0) 0x7fd344137650 con 0x7fd34413d620 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient(hunting): my global_id is 4534 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd330003520 con 0x7fd34413d620 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1433361469 0 0) 0x7fd330003520 con 0x7fd34413d620 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 >> v1:172.21.15.148:6790/0 conn(0x7fd344139b10 legacy=0x7fd34413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 >> v1:172.21.15.148:6789/0 conn(0x7fd344136b60 legacy=0x7fd344136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:07:56.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 monclient: found mon.noname-c 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3440a0550 con 0x7fd34413d620 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: _finish_auth 0 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd33c001990 con 0x7fd34413d620 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 5 monclient: authenticate success, global_id 4534 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: dump: 2024-09-16T04:07:56.288 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd33c001bf0 con 0x7fd34413d620 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:56.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 1 -- 172.21.15.171:0/4154821298 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd33c002040 con 0x7fd34413d620 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd348cc9640 4 set_mon_vals no callback set 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.276+0000 7fd341d9b640 10 monclient: dump: 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.290 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 10 monclient: shutdown 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 1 -- 172.21.15.171:0/4154821298 >> v1:172.21.15.171:6789/0 conn(0x7fd34413d620 legacy=0x7fd34413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 1 -- 172.21.15.171:0/4154821298 shutdown_connections 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 1 -- 172.21.15.171:0/4154821298 >> 172.21.15.171:0/4154821298 conn(0x7fd3440a98b0 msgr2=0x7fd3440a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 1 -- 172.21.15.171:0/4154821298 shutdown_connections 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.277+0000 7fd348cc9640 1 -- 172.21.15.171:0/4154821298 wait complete. 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 10 monclient: build_initial_monmap 2024-09-16T04:07:56.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 10 monclient: monmap: 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 1 Processor -- start 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 1 -- start start 2024-09-16T04:07:56.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 10 monclient: init 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.278+0000 7fd348cc9640 10 monclient(hunting): picked mon.noname-a con 0x7fd344136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient(hunting): picked mon.noname-c con 0x7fd344139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient(hunting): picked mon.noname-b con 0x7fd34413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd34413c010 con 0x7fd344136b60 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd34413ad10 con 0x7fd34413d620 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3440a0240 con 0x7fd344139b10 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd34259c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd344136b60 legacy=0x7fd344139ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient(hunting): authenticate will time out at 851.785091s 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd342d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd34413d620 0x7fd3441d37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40530/0 (socket says 172.21.15.171:40530) 2024-09-16T04:07:56.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd342d9d640 1 -- 172.21.15.171:0/3158545368 learned_addr learned my addr 172.21.15.171:0/3158545368 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186665784 0 0) 0x7fd34413ad10 con 0x7fd34413d620 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): my global_id is 4537 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd324003550 con 0x7fd34413d620 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598589068 0 0) 0x7fd3440a0240 con 0x7fd344139b10 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): my global_id is 4559 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd34413ad10 con 0x7fd344139b10 2024-09-16T04:07:56.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 665938551 0 0) 0x7fd324003550 con 0x7fd34413d620 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3440a0240 con 0x7fd34413d620 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd33c001f40 con 0x7fd34413d620 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): dump: 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.295 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2839277916 0 0) 0x7fd34413ad10 con 0x7fd344139b10 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd324003550 con 0x7fd344139b10 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1382658942 0 0) 0x7fd3440a0240 con 0x7fd34413d620 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 >> v1:172.21.15.148:6790/0 conn(0x7fd344139b10 legacy=0x7fd34413a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 >> v1:172.21.15.148:6789/0 conn(0x7fd344136b60 legacy=0x7fd344139ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 monclient: found mon.b 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:56.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3440a2b30 con 0x7fd34413d620 2024-09-16T04:07:56.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:56.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:56.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:56.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 5 monclient: authenticate success, global_id 4537 2024-09-16T04:07:56.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient: _renew_subs 2024-09-16T04:07:56.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd34409ada0 con 0x7fd34413d620 2024-09-16T04:07:56.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient: _renew_subs 2024-09-16T04:07:56.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd348cc9640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd34413bb70 con 0x7fd34413d620 2024-09-16T04:07:56.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd33c004ff0 con 0x7fd34413d620 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd33c005340 con 0x7fd34413d620 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd33b7fe640 10 monclient: dump: 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.302 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.279+0000 7fd34159a640 4 set_mon_vals no callback set 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd33c0107f0 con 0x7fd34413d620 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd348cc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd348cc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd348cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd348cc9640 1 -- 172.21.15.171:0/3158545368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd328005080 con 0x7fd34413d620 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(29..29 src has 1..29) ==== 3506+0+0 (unknown 2189529227 0 0) 0x7fd33c0470b0 con 0x7fd34413d620 2024-09-16T04:07:56.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.280+0000 7fd34259c640 10 monclient: get_auth_request con 0x7fd324038ee0 auth_method 0 2024-09-16T04:07:56.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.285+0000 7fd33b7fe640 1 -- 172.21.15.171:0/3158545368 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd33c016120 con 0x7fd34413d620 2024-09-16T04:07:56.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.285+0000 7fd33b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.285+0000 7fd33b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:56.456 INFO:tasks.workunit.client.0.smithi171.stderr:foo-rule not valid: foo-rule not one of 'true', 'false' 2024-09-16T04:07:56.456 INFO:tasks.workunit.client.0.smithi171.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-16T04:07:56.456 INFO:tasks.workunit.client.0.smithi171.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] [--yes-i-really-mean-it] : create pool 2024-09-16T04:07:56.457 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: invalid command 2024-09-16T04:07:56.457 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:07:56.457 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.537+0000 7fb0e8191640 10 monclient: get_monmap_and_config 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.537+0000 7fb0e8191640 10 monclient: build_initial_monmap 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.537+0000 7fb0e8191640 10 monclient: monmap: 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:56.538964+0000 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:56.538964+0000 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:56.541 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.537+0000 7fb0e8191640 1 Processor -- start 2024-09-16T04:07:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 1 -- start start 2024-09-16T04:07:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient: init 2024-09-16T04:07:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e0136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e0137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e01375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e009b2c0 con 0x7fb0e0136d60 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e009b5a0 con 0x7fb0e01375a0 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e00a6640 con 0x7fb0e0137140 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e8191640 10 monclient(hunting): authenticate will time out at 852.044516s 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e5f2d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb0e0136d60 legacy=0x7fb0e009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e672e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb0e01375a0 0x7fb0e013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40536/0 (socket says 172.21.15.171:40536) 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e672e640 1 -- 172.21.15.171:0/922366289 learned_addr learned my addr 172.21.15.171:0/922366289 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750849939 0 0) 0x7fb0e009b5a0 con 0x7fb0e01375a0 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e4f2b640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e4f2b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e4f2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e4f2b640 10 monclient(hunting): my global_id is 4540 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.538+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0cc003520 con 0x7fb0e01375a0 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647615936 0 0) 0x7fb0e00a6640 con 0x7fb0e0137140 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient(hunting): my global_id is 4562 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e009b5a0 con 0x7fb0e0137140 2024-09-16T04:07:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1432076483 0 0) 0x7fb0cc003520 con 0x7fb0e01375a0 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 >> v1:172.21.15.148:6790/0 conn(0x7fb0e0137140 legacy=0x7fb0e013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 >> v1:172.21.15.148:6789/0 conn(0x7fb0e0136d60 legacy=0x7fb0e009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: _finish_hunting 0 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 monclient: found mon.noname-c 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e0143b20 con 0x7fb0e01375a0 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: _finish_auth 0 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: _check_auth_tickets 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb0d8001990 con 0x7fb0e01375a0 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 5 monclient: authenticate success, global_id 4540 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: dump: 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0d8001bf0 con 0x7fb0e01375a0 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 1 -- 172.21.15.171:0/922366289 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb0d8001f40 con 0x7fb0e01375a0 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 4 set_mon_vals no callback set 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e4f2b640 10 monclient: dump: 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 10 monclient: shutdown 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 1 -- 172.21.15.171:0/922366289 >> v1:172.21.15.171:6789/0 conn(0x7fb0e01375a0 legacy=0x7fb0e013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 1 -- 172.21.15.171:0/922366289 shutdown_connections 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 1 -- 172.21.15.171:0/922366289 >> 172.21.15.171:0/922366289 conn(0x7fb0e00a9ae0 msgr2=0x7fb0e00a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 1 -- 172.21.15.171:0/922366289 shutdown_connections 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.539+0000 7fb0e8191640 1 -- 172.21.15.171:0/922366289 wait complete. 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.540+0000 7fb0e8191640 10 monclient: build_initial_monmap 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.540+0000 7fb0e8191640 10 monclient: monmap: 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.540+0000 7fb0e8191640 1 Processor -- start 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.540+0000 7fb0e8191640 1 -- start start 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.540+0000 7fb0e8191640 10 monclient: init 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e0136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e0137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e01375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e00a03b0 con 0x7fb0e01375a0 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e012e6f0 con 0x7fb0e0137140 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e012e9d0 con 0x7fb0e0136d60 2024-09-16T04:07:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e8191640 10 monclient(hunting): authenticate will time out at 852.047382s 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e572c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb0e0137140 0x7fb0e01d0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:40542/0 (socket says 172.21.15.171:40542) 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e572c640 1 -- 172.21.15.171:0/2632979915 learned_addr learned my addr 172.21.15.171:0/2632979915 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0e672e640 1 -- 172.21.15.171:0/2632979915 >> v1:172.21.15.148:6789/0 conn(0x7fb0e01375a0 legacy=0x7fb0e01d43a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823730418 0 0) 0x7fb0e012e6f0 con 0x7fb0e0137140 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): my global_id is 4543 2024-09-16T04:07:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c4003520 con 0x7fb0e0137140 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660239533 0 0) 0x7fb0e012e9d0 con 0x7fb0e0136d60 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): my global_id is 4565 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e012e6f0 con 0x7fb0e0136d60 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2024560632 0 0) 0x7fb0c4003520 con 0x7fb0e0137140 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0e012e9d0 con 0x7fb0e0137140 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb0d0001a00 con 0x7fb0e0137140 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): dump: 2024-09-16T04:07:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4082750588 0 0) 0x7fb0e012e6f0 con 0x7fb0e0136d60 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0c4003520 con 0x7fb0e0136d60 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb0dc0040f0 con 0x7fb0e0136d60 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.541+0000 7fb0d67fc640 10 monclient(hunting): dump: 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3608382317 0 0) 0x7fb0e012e9d0 con 0x7fb0e0137140 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 >> v1:172.21.15.148:6790/0 conn(0x7fb0e0136d60 legacy=0x7fb0e01cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 >> v1:172.21.15.148:6789/0 conn(0x7fb0e01375a0 legacy=0x7fb0e01d43a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 monclient: found mon.b 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e009fd70 con 0x7fb0e0137140 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 5 monclient: authenticate success, global_id 4543 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 10 monclient: _renew_subs 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e009f0b0 con 0x7fb0e0137140 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 10 monclient: _renew_subs 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0e009e840 con 0x7fb0e0137140 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0d0001d20 con 0x7fb0e0137140 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb0d0003a60 con 0x7fb0e0137140 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d7fff640 4 set_mon_vals no callback set 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 10 monclient: dump: 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb0d0003c80 con 0x7fb0e0137140 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.542+0000 7fb0e8191640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.543+0000 7fb0e8191640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.543+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c8005080 con 0x7fb0e0137140 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.543+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(29..29 src has 1..29) ==== 3506+0+0 (unknown 2189529227 0 0) 0x7fb0d0046ea0 con 0x7fb0e0137140 2024-09-16T04:07:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.543+0000 7fb0e5f2d640 10 monclient: get_auth_request con 0x7fb0c4038f10 auth_method 0 2024-09-16T04:07:56.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.548+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb0d0015e10 con 0x7fb0e0137140 2024-09-16T04:07:56.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.548+0000 7fb0d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.548+0000 7fb0d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:56.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.713+0000 7fb0e8191640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T04:07:56.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.713+0000 7fb0e8191640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T04:07:56.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.713+0000 7fb0e8191640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:56.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:56.713+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) -- 0x7fb0c8005370 con 0x7fb0e0137140 2024-09-16T04:07:57.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.089+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v30) ==== 154+0+0 (unknown 412941063 0 0) 0x7fb0d0015710 con 0x7fb0e0137140 2024-09-16T04:07:57.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.089+0000 7fb0d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T04:07:57.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.089+0000 7fb0d67fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-16T04:07:57.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0e8191640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0e8191640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0e8191640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) -- 0x7fb0c8001140 con 0x7fb0e0137140 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0d67fc640 1 -- 172.21.15.171:0/2632979915 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v30) ==== 161+0+0 (unknown 1755317401 0 0) 0x7fb0d001e2a0 con 0x7fb0e0137140 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0d67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.196+0000 7fb0d67fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T04:07:57.199 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'foo' already exists 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 >> v1:172.21.15.171:6812/7327273 conn(0x7fb0c4038f10 legacy=0x7fb0c403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 10 monclient: shutdown 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 >> v1:172.21.15.171:6789/0 conn(0x7fb0e0137140 legacy=0x7fb0e01d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 shutdown_connections 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 >> 172.21.15.171:0/2632979915 conn(0x7fb0e00a9ae0 msgr2=0x7fb0e0072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 shutdown_connections 2024-09-16T04:07:57.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.198+0000 7fb0e8191640 1 -- 172.21.15.171:0/2632979915 wait complete. 2024-09-16T04:07:57.209 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-16T04:07:57.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.290+0000 7fb82ae77640 10 monclient: get_monmap_and_config 2024-09-16T04:07:57.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.290+0000 7fb82ae77640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.290+0000 7fb82ae77640 10 monclient: monmap: 2024-09-16T04:07:57.292 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:57.292 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:57.291370+0000 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:57.291370+0000 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:57.293 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:57.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 1 Processor -- start 2024-09-16T04:07:57.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 1 -- start start 2024-09-16T04:07:57.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient: init 2024-09-16T04:07:57.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient(hunting): picked mon.noname-b con 0x7fb824136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient(hunting): picked mon.noname-c con 0x7fb824139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient(hunting): picked mon.noname-a con 0x7fb82413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8240a1340 con 0x7fb82413d620 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb824137650 con 0x7fb824139b10 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8240a0aa0 con 0x7fb824136b60 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb82ae77640 10 monclient(hunting): authenticate will time out at 852.796901s 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb81bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb824139b10 0x7fb82413bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56764/0 (socket says 172.21.15.171:56764) 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb81bfff640 1 -- 172.21.15.171:0/3758978119 learned_addr learned my addr 172.21.15.171:0/3758978119 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.291+0000 7fb829414640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb82413d620 legacy=0x7fb82413fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:57.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2687224809 0 0) 0x7fb824137650 con 0x7fb824139b10 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient(hunting): my global_id is 4546 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb80c003520 con 0x7fb824139b10 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 799619933 0 0) 0x7fb80c003520 con 0x7fb824139b10 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 >> v1:172.21.15.148:6790/0 conn(0x7fb824136b60 legacy=0x7fb824136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 >> v1:172.21.15.148:6789/0 conn(0x7fb82413d620 legacy=0x7fb82413fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8240a0550 con 0x7fb824139b10 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb82ae77640 5 monclient: authenticate success, global_id 4546 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb814001610 con 0x7fb824139b10 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb82ae77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: dump: 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.303 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb82ae77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb814001970 con 0x7fb824139b10 2024-09-16T04:07:57.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb82ae77640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 1 -- 172.21.15.171:0/3758978119 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb8140048a0 con 0x7fb824139b10 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb82ae77640 4 set_mon_vals no callback set 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.292+0000 7fb81b7fe640 10 monclient: dump: 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.305 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 10 monclient: shutdown 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 -- 172.21.15.171:0/3758978119 >> v1:172.21.15.171:6789/0 conn(0x7fb824139b10 legacy=0x7fb82413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 -- 172.21.15.171:0/3758978119 shutdown_connections 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 -- 172.21.15.171:0/3758978119 >> 172.21.15.171:0/3758978119 conn(0x7fb8240a98b0 msgr2=0x7fb8240a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 -- 172.21.15.171:0/3758978119 shutdown_connections 2024-09-16T04:07:57.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 -- 172.21.15.171:0/3758978119 wait complete. 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 10 monclient: monmap: 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 Processor -- start 2024-09-16T04:07:57.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 1 -- start start 2024-09-16T04:07:57.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.293+0000 7fb82ae77640 10 monclient: init 2024-09-16T04:07:57.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:57.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient(hunting): picked mon.noname-b con 0x7fb824136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:57.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient(hunting): picked mon.noname-c con 0x7fb824139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:57.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient(hunting): picked mon.noname-a con 0x7fb82413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb824120920 con 0x7fb82413d620 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82409f430 con 0x7fb824136b60 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb824120c00 con 0x7fb824139b10 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb82ae77640 10 monclient(hunting): authenticate will time out at 852.799596s 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb828c13640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb824136b60 0x7fb82409d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56774/0 (socket says 172.21.15.171:56774) 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb828c13640 1 -- 172.21.15.171:0/948839232 learned_addr learned my addr 172.21.15.171:0/948839232 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb829414640 1 -- 172.21.15.171:0/948839232 >> v1:172.21.15.148:6789/0 conn(0x7fb82413d620 legacy=0x7fb8241d37f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1798675351 0 0) 0x7fb82409f430 con 0x7fb824136b60 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): my global_id is 4549 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb810003550 con 0x7fb824136b60 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222262035 0 0) 0x7fb824120c00 con 0x7fb824139b10 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 10 monclient(hunting): my global_id is 4571 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb82409f430 con 0x7fb824139b10 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.294+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2296173428 0 0) 0x7fb810003550 con 0x7fb824136b60 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb824120c00 con 0x7fb824136b60 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb820001970 con 0x7fb824136b60 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:57.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient(hunting): dump: 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3809792433 0 0) 0x7fb824120c00 con 0x7fb824136b60 2024-09-16T04:07:57.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 >> v1:172.21.15.148:6790/0 conn(0x7fb824139b10 legacy=0x7fb82409dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 >> v1:172.21.15.148:6789/0 conn(0x7fb82413d620 legacy=0x7fb8241d37f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 monclient: found mon.b 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8240a8b70 con 0x7fb824136b60 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 5 monclient: authenticate success, global_id 4549 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 10 monclient: _renew_subs 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb8240a0240 con 0x7fb824136b60 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 10 monclient: _renew_subs 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb82409e380 con 0x7fb824136b60 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8200038c0 con 0x7fb824136b60 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb820003af0 con 0x7fb824136b60 2024-09-16T04:07:57.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb81affd640 4 set_mon_vals no callback set 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.295+0000 7fb8197fa640 10 monclient: dump: 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:57.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb820003c90 con 0x7fb824136b60 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb82ae77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb82ae77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb82ae77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb804005080 con 0x7fb824136b60 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(30..30 src has 1..30) ==== 3848+0+0 (unknown 2043296344 0 0) 0x7fb820046f80 con 0x7fb824136b60 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.296+0000 7fb81bfff640 10 monclient: get_auth_request con 0x7fb810039010 auth_method 0 2024-09-16T04:07:57.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.301+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb820015d80 con 0x7fb824136b60 2024-09-16T04:07:57.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.301+0000 7fb8197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.301+0000 7fb8197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:57.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.467+0000 7fb82ae77640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T04:07:57.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.467+0000 7fb82ae77640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T04:07:57.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.467+0000 7fb82ae77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:57.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:57.467+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) -- 0x7fb804005370 con 0x7fb824136b60 2024-09-16T04:07:59.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.096+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v32) ==== 188+0+0 (unknown 4005810093 0 0) 0x7fb82001ccf0 con 0x7fb824136b60 2024-09-16T04:07:59.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.096+0000 7fb8197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T04:07:59.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.096+0000 7fb8197fa640 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-16T04:07:59.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.212+0000 7fb82ae77640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.212+0000 7fb82ae77640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.212+0000 7fb82ae77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.212+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) -- 0x7fb804005ab0 con 0x7fb824136b60 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.212+0000 7fb8197fa640 1 -- 172.21.15.171:0/948839232 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v32) ==== 195+0+0 (unknown 2585474358 0 0) 0x7fb82001d120 con 0x7fb824136b60 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.213+0000 7fb8197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.213+0000 7fb8197fa640 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-16T04:07:59.215 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'fooo' already exists 2024-09-16T04:07:59.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.214+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 >> v1:172.21.15.171:6812/7327273 conn(0x7fb810039010 legacy=0x7fb81003b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.214+0000 7fb82ae77640 10 monclient: shutdown 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.214+0000 7fb82ae77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.214+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 >> v1:172.21.15.171:6789/0 conn(0x7fb824136b60 legacy=0x7fb82409d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.215+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 shutdown_connections 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.215+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 >> 172.21.15.171:0/948839232 conn(0x7fb8240a98b0 msgr2=0x7fb824139380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.215+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 shutdown_connections 2024-09-16T04:07:59.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.215+0000 7fb82ae77640 1 -- 172.21.15.171:0/948839232 wait complete. 2024-09-16T04:07:59.226 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create foooo 123 2024-09-16T04:07:59.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.304+0000 7fb7bfef8640 10 monclient: get_monmap_and_config 2024-09-16T04:07:59.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.304+0000 7fb7bfef8640 10 monclient: build_initial_monmap 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.304+0000 7fb7bfef8640 10 monclient: monmap: 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:07:59.305735+0000 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:07:59.305735+0000 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:07:59.307 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:59.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.305+0000 7fb7bfef8640 1 Processor -- start 2024-09-16T04:07:59.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.305+0000 7fb7bfef8640 1 -- start start 2024-09-16T04:07:59.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.305+0000 7fb7bfef8640 10 monclient: init 2024-09-16T04:07:59.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient(hunting): picked mon.noname-a con 0x7fb7b8132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient(hunting): picked mon.noname-b con 0x7fb7b809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7b8132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b81332b0 con 0x7fb7b8132a80 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b8134310 con 0x7fb7b8132e60 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b809d980 con 0x7fb7b809ec20 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient(hunting): authenticate will time out at 854.811272s 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7be495640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb7b8132e60 0x7fb7b813c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56790/0 (socket says 172.21.15.171:56790) 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7be495640 1 -- 172.21.15.171:0/2984246813 learned_addr learned my addr 172.21.15.171:0/2984246813 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bdc94640 1 -- 172.21.15.171:0/2984246813 >> v1:172.21.15.148:6789/0 conn(0x7fb7b8132a80 legacy=0x7fb7b81334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340755493 0 0) 0x7fb7b8134310 con 0x7fb7b8132e60 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient(hunting): _negotiate 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient(hunting): my global_id is 4552 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7a4003520 con 0x7fb7b8132e60 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3634759605 0 0) 0x7fb7a4003520 con 0x7fb7b8132e60 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 >> v1:172.21.15.148:6790/0 conn(0x7fb7b809ec20 legacy=0x7fb7b8133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 >> v1:172.21.15.148:6789/0 conn(0x7fb7b8132a80 legacy=0x7fb7b81334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: _finish_hunting 0 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 monclient: found mon.noname-c 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b809dc60 con 0x7fb7b8132e60 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: _finish_auth 0 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: _check_auth_tickets 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 5 monclient: authenticate success, global_id 4552 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7b0001990 con 0x7fb7b8132e60 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: dump: 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:59.317 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7b0001c80 con 0x7fb7b8132e60 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 1 -- 172.21.15.171:0/2984246813 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7b0002090 con 0x7fb7b8132e60 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bfef8640 4 set_mon_vals no callback set 2024-09-16T04:07:59.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.306+0000 7fb7bcc92640 10 monclient: dump: 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:59.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 10 monclient: shutdown 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 1 -- 172.21.15.171:0/2984246813 >> v1:172.21.15.171:6789/0 conn(0x7fb7b8132e60 legacy=0x7fb7b813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 1 -- 172.21.15.171:0/2984246813 shutdown_connections 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 1 -- 172.21.15.171:0/2984246813 >> 172.21.15.171:0/2984246813 conn(0x7fb7b80a52a0 msgr2=0x7fb7b80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 1 -- 172.21.15.171:0/2984246813 shutdown_connections 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.307+0000 7fb7bfef8640 1 -- 172.21.15.171:0/2984246813 wait complete. 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient: build_initial_monmap 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient: monmap: 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:07:59.320 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 1 Processor -- start 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 1 -- start start 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient: init 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient(hunting): picked mon.noname-a con 0x7fb7b809ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7b8132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:07:59.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient(hunting): picked mon.noname-b con 0x7fb7b8132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b8120670 con 0x7fb7b809ec20 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b8120950 con 0x7fb7b8132e60 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b811eb40 con 0x7fb7b8132a80 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bfef8640 10 monclient(hunting): authenticate will time out at 854.813880s 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7bdc94640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb7b809ec20 legacy=0x7fb7b811dd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7be495640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb7b8132e60 0x7fb7b81d1650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56798/0 (socket says 172.21.15.171:56798) 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.308+0000 7fb7be495640 1 -- 172.21.15.171:0/718778954 learned_addr learned my addr 172.21.15.171:0/718778954 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1834612327 0 0) 0x7fb7b8120950 con 0x7fb7b8132e60 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): my global_id is 4555 2024-09-16T04:07:59.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb79c003550 con 0x7fb7b8132e60 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432350425 0 0) 0x7fb7b811eb40 con 0x7fb7b8132a80 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): my global_id is 4577 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7b8120950 con 0x7fb7b8132a80 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 296254963 0 0) 0x7fb79c003550 con 0x7fb7b8132e60 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7b811eb40 con 0x7fb7b8132e60 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7b00017b0 con 0x7fb7b8132e60 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): dump: 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:59.323 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3442370129 0 0) 0x7fb7b811eb40 con 0x7fb7b8132e60 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 >> v1:172.21.15.148:6790/0 conn(0x7fb7b8132a80 legacy=0x7fb7b811e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 >> v1:172.21.15.148:6789/0 conn(0x7fb7b809ec20 legacy=0x7fb7b811dd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:59.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 monclient: found mon.b 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b811ede0 con 0x7fb7b8132e60 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 5 monclient: authenticate success, global_id 4555 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 10 monclient: _renew_subs 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b81319e0 con 0x7fb7b8132e60 2024-09-16T04:07:59.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 10 monclient: _renew_subs 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7b8120390 con 0x7fb7b8132e60 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7b0004ff0 con 0x7fb7b8132e60 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7b0005340 con 0x7fb7b8132e60 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7affff640 4 set_mon_vals no callback set 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.309+0000 7fb7ae7fc640 10 monclient: dump: 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.326 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb7b0010770 con 0x7fb7b8132e60 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7bfef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7bfef8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7bfef8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a0005080 con 0x7fb7b8132e60 2024-09-16T04:07:59.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(32..32 src has 1..32) ==== 4290+0+0 (unknown 1112713647 0 0) 0x7fb7b0047480 con 0x7fb7b8132e60 2024-09-16T04:07:59.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.310+0000 7fb7bdc94640 10 monclient: get_auth_request con 0x7fb79c038fc0 auth_method 0 2024-09-16T04:07:59.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.315+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb7b0016180 con 0x7fb7b8132e60 2024-09-16T04:07:59.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.315+0000 7fb7ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:59.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.315+0000 7fb7ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:59.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.481+0000 7fb7bfef8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T04:07:59.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.481+0000 7fb7bfef8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T04:07:59.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.481+0000 7fb7bfef8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:07:59.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:07:59.481+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) -- 0x7fb7a0005b80 con 0x7fb7b8132e60 2024-09-16T04:08:00.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.107+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v33) ==== 115+0+0 (unknown 565579787 0 0) 0x7fb7b0015a80 con 0x7fb7b8132e60 2024-09-16T04:08:00.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.107+0000 7fb7ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T04:08:00.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.107+0000 7fb7ae7fc640 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-16T04:08:00.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.247+0000 7fb7bfef8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.247+0000 7fb7bfef8640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.247+0000 7fb7bfef8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.247+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) -- 0x7fb7a0001090 con 0x7fb7b8132e60 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.248+0000 7fb7ae7fc640 1 -- 172.21.15.171:0/718778954 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v33) ==== 122+0+0 (unknown 4101563093 0 0) 0x7fb7b001e5d0 con 0x7fb7b8132e60 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.248+0000 7fb7ae7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.248+0000 7fb7ae7fc640 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-16T04:08:00.250 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'foooo' already exists 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 >> v1:172.21.15.171:6812/7327273 conn(0x7fb79c038fc0 legacy=0x7fb79c03b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 10 monclient: shutdown 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 >> v1:172.21.15.171:6789/0 conn(0x7fb7b8132e60 legacy=0x7fb7b81d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 shutdown_connections 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 >> 172.21.15.171:0/718778954 conn(0x7fb7b80a52a0 msgr2=0x7fb7b8131ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 shutdown_connections 2024-09-16T04:08:00.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.250+0000 7fb7bfef8640 1 -- 172.21.15.171:0/718778954 wait complete. 2024-09-16T04:08:00.260 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create foo 123 2024-09-16T04:08:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.338+0000 7f468c798640 10 monclient: get_monmap_and_config 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.338+0000 7f468c798640 10 monclient: build_initial_monmap 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.339+0000 7f468c798640 10 monclient: monmap: 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:00.339770+0000 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:00.339770+0000 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:00.342 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.339+0000 7f468c798640 1 Processor -- start 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.339+0000 7f468c798640 1 -- start start 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.339+0000 7f468c798640 10 monclient: init 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient(hunting): picked mon.noname-a con 0x7f4684072a70 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient(hunting): picked mon.noname-c con 0x7f4684070250 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient(hunting): picked mon.noname-b con 0x7f468413afe0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468409ed80 con 0x7f4684072a70 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46840a0030 con 0x7f4684070250 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46840a0310 con 0x7f468413afe0 2024-09-16T04:08:00.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 10 monclient(hunting): authenticate will time out at 855.845384s 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468a534640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4684072a70 legacy=0x7f4684072e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689d33640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4684070250 0x7f46841398c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56804/0 (socket says 172.21.15.171:56804) 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689d33640 1 -- 172.21.15.171:0/657541880 learned_addr learned my addr 172.21.15.171:0/657541880 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211974437 0 0) 0x7f46840a0030 con 0x7f4684070250 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient(hunting): _negotiate 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient(hunting): my global_id is 4558 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f466c003520 con 0x7f4684070250 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2006920327 0 0) 0x7f466c003520 con 0x7f4684070250 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 >> v1:172.21.15.148:6790/0 conn(0x7f468413afe0 legacy=0x7f468413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 >> v1:172.21.15.148:6789/0 conn(0x7f4684072a70 legacy=0x7f4684072e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient: _finish_hunting 0 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 monclient: found mon.noname-c 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468409f090 con 0x7f4684070250 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient: _finish_auth 0 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient: _check_auth_tickets 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4678001610 con 0x7f4684070250 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 5 monclient: authenticate success, global_id 4558 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f468c798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.340+0000 7f4689532640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:00.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 10 monclient: dump: 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4678001970 con 0x7f4684070250 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:00.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 1 -- 172.21.15.171:0/657541880 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f46780048a0 con 0x7f4684070250 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 4 set_mon_vals no callback set 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f4689532640 10 monclient: dump: 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.353 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 10 monclient: shutdown 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 1 -- 172.21.15.171:0/657541880 >> v1:172.21.15.171:6789/0 conn(0x7f4684070250 legacy=0x7f46841398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 1 -- 172.21.15.171:0/657541880 shutdown_connections 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 1 -- 172.21.15.171:0/657541880 >> 172.21.15.171:0/657541880 conn(0x7f46840a7520 msgr2=0x7f46840a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 1 -- 172.21.15.171:0/657541880 shutdown_connections 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.341+0000 7f468c798640 1 -- 172.21.15.171:0/657541880 wait complete. 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient: build_initial_monmap 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient: monmap: 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:00.354 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 1 Processor -- start 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 1 -- start start 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient: init 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient(hunting): picked mon.noname-a con 0x7f4684070250 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient(hunting): picked mon.noname-c con 0x7f4684072a70 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:00.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient(hunting): picked mon.noname-b con 0x7f468413afe0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4684139e40 con 0x7f4684070250 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4684126e90 con 0x7f468413afe0 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468413ac50 con 0x7f4684072a70 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.342+0000 7f468c798640 10 monclient(hunting): authenticate will time out at 855.847984s 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468a534640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4684070250 legacy=0x7f4684139370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468ad35640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f468413afe0 0x7f46841d3af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56818/0 (socket says 172.21.15.171:56818) 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468ad35640 1 -- 172.21.15.171:0/3779674404 learned_addr learned my addr 172.21.15.171:0/3779674404 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895381953 0 0) 0x7f4684126e90 con 0x7f468413afe0 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): my global_id is 4561 2024-09-16T04:08:00.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4670003520 con 0x7f468413afe0 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051364924 0 0) 0x7f468413ac50 con 0x7f4684072a70 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): my global_id is 4583 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4684126e90 con 0x7f4684072a70 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3764435631 0 0) 0x7f4670003520 con 0x7f468413afe0 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f468413ac50 con 0x7f468413afe0 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f467c001940 con 0x7f468413afe0 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.357 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1056369420 0 0) 0x7f4684126e90 con 0x7f4684072a70 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4670003520 con 0x7f4684072a70 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4678001430 con 0x7f4684072a70 2024-09-16T04:08:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2722782975 0 0) 0x7f468413ac50 con 0x7f468413afe0 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 >> v1:172.21.15.148:6790/0 conn(0x7f4684072a70 legacy=0x7f46841d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 >> v1:172.21.15.148:6789/0 conn(0x7f4684070250 legacy=0x7f4684139370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 monclient: found mon.b 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468413a700 con 0x7f468413afe0 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 5 monclient: authenticate success, global_id 4561 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 10 monclient: _renew_subs 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f468409e0d0 con 0x7f468413afe0 2024-09-16T04:08:00.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 10 monclient: _renew_subs 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f467c001cf0 con 0x7f468413afe0 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4684126bb0 con 0x7f468413afe0 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f467c003a60 con 0x7f468413afe0 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4682ffd640 10 monclient: dump: 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.361 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.343+0000 7f4688d31640 4 set_mon_vals no callback set 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f467c003c80 con 0x7f468413afe0 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f468c798640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f468c798640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f468c798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46840008d0 con 0x7f468413afe0 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(33..33 src has 1..33) ==== 4634+0+0 (unknown 4221952135 0 0) 0x7f467c047120 con 0x7f468413afe0 2024-09-16T04:08:00.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.344+0000 7f468a534640 10 monclient: get_auth_request con 0x7f4670038ec0 auth_method 0 2024-09-16T04:08:00.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.349+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f467c015cf0 con 0x7f468413afe0 2024-09-16T04:08:00.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.349+0000 7f4682ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:00.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.349+0000 7f4682ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.513+0000 7f468c798640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.513+0000 7f468c798640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.513+0000 7f468c798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.513+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) -- 0x7f4684139a80 con 0x7f468413afe0 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.514+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v33) ==== 118+0+0 (unknown 389084578 0 0) 0x7f467c0155f0 con 0x7f468413afe0 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.514+0000 7f4682ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T04:08:00.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.514+0000 7f4682ffd640 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-16T04:08:00.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f468c798640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T04:08:00.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f468c798640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T04:08:00.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f468c798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) -- 0x7f468406cbd0 con 0x7f468413afe0 2024-09-16T04:08:00.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f4682ffd640 1 -- 172.21.15.171:0/3779674404 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v33) ==== 118+0+0 (unknown 389084578 0 0) 0x7f467c019940 con 0x7f468413afe0 2024-09-16T04:08:00.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f4682ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-16T04:08:00.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.611+0000 7f4682ffd640 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-16T04:08:00.614 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'foo' already exists 2024-09-16T04:08:00.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 >> v1:172.21.15.171:6812/7327273 conn(0x7f4670038ec0 legacy=0x7f467003b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 10 monclient: shutdown 2024-09-16T04:08:00.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:00.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 >> v1:172.21.15.171:6789/0 conn(0x7f468413afe0 legacy=0x7f46841d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 shutdown_connections 2024-09-16T04:08:00.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 >> 172.21.15.171:0/3779674404 conn(0x7f46840a7520 msgr2=0x7f46840a3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:00.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 shutdown_connections 2024-09-16T04:08:00.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.613+0000 7f468c798640 1 -- 172.21.15.171:0/3779674404 wait complete. 2024-09-16T04:08:00.624 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-16T04:08:00.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.704+0000 7f59d3e2e640 10 monclient: get_monmap_and_config 2024-09-16T04:08:00.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.704+0000 7f59d3e2e640 10 monclient: build_initial_monmap 2024-09-16T04:08:00.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.704+0000 7f59d3e2e640 10 monclient: monmap: 2024-09-16T04:08:00.706 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:00.706 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.706 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:00.705391+0000 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:00.705391+0000 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:00.707 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 1 Processor -- start 2024-09-16T04:08:00.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 1 -- start start 2024-09-16T04:08:00.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient: init 2024-09-16T04:08:00.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:00.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:00.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc1338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc133cc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc1340d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc09b420 con 0x7f59cc1340d0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc0753d0 con 0x7f59cc133cc0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc09b140 con 0x7f59cc1338e0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d3e2e640 10 monclient(hunting): authenticate will time out at 856.210877s 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d13c9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f59cc133cc0 0x7f59cc13baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56828/0 (socket says 172.21.15.171:56828) 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d13c9640 1 -- 172.21.15.171:0/2629850930 learned_addr learned my addr 172.21.15.171:0/2629850930 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.705+0000 7f59d23cb640 1 -- 172.21.15.171:0/2629850930 >> v1:172.21.15.148:6789/0 conn(0x7f59cc1340d0 legacy=0x7f59cc13f220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666800066 0 0) 0x7f59cc0753d0 con 0x7f59cc133cc0 2024-09-16T04:08:00.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient(hunting): _negotiate 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient(hunting): my global_id is 4564 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b8003520 con 0x7f59cc133cc0 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2959841990 0 0) 0x7f59b8003520 con 0x7f59cc133cc0 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 >> v1:172.21.15.148:6790/0 conn(0x7f59cc1338e0 legacy=0x7f59cc09be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 >> v1:172.21.15.148:6789/0 conn(0x7f59cc1340d0 legacy=0x7f59cc13f220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: _finish_hunting 0 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 monclient: found mon.noname-c 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc09b700 con 0x7f59cc133cc0 2024-09-16T04:08:00.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: _finish_auth 0 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: _check_auth_tickets 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59bc001610 con 0x7f59cc133cc0 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: dump: 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d3e2e640 5 monclient: authenticate success, global_id 4564 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d3e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59bc001900 con 0x7f59cc133cc0 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 1 -- 172.21.15.171:0/2629850930 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59bc001d10 con 0x7f59cc133cc0 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d3e2e640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d3e2e640 4 set_mon_vals no callback set 2024-09-16T04:08:00.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.706+0000 7f59d0bc8640 10 monclient: dump: 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 10 monclient: shutdown 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 1 -- 172.21.15.171:0/2629850930 >> v1:172.21.15.171:6789/0 conn(0x7f59cc133cc0 legacy=0x7f59cc13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 1 -- 172.21.15.171:0/2629850930 shutdown_connections 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 1 -- 172.21.15.171:0/2629850930 >> 172.21.15.171:0/2629850930 conn(0x7f59cc0a6460 msgr2=0x7f59cc0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 1 -- 172.21.15.171:0/2629850930 shutdown_connections 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 1 -- 172.21.15.171:0/2629850930 wait complete. 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 10 monclient: build_initial_monmap 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 10 monclient: monmap: 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:00.723 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.707+0000 7f59d3e2e640 1 Processor -- start 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 1 -- start start 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient: init 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:00.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc1338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc133cc0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc1340d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc1362b0 con 0x7f59cc1340d0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc136590 con 0x7f59cc1338e0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc132bb0 con 0x7f59cc133cc0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d1bca640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f59cc1338e0 0x7f59cc131d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56838/0 (socket says 172.21.15.171:56838) 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d1bca640 1 -- 172.21.15.171:0/1978872701 learned_addr learned my addr 172.21.15.171:0/1978872701 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d3e2e640 10 monclient(hunting): authenticate will time out at 856.213576s 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59d23cb640 1 -- 172.21.15.171:0/1978872701 >> v1:172.21.15.148:6789/0 conn(0x7f59cc1340d0 legacy=0x7f59cc1d39a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159223618 0 0) 0x7f59cc136590 con 0x7f59cc1338e0 2024-09-16T04:08:00.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): my global_id is 4567 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b0003520 con 0x7f59cc1338e0 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3637879439 0 0) 0x7f59cc132bb0 con 0x7f59cc133cc0 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): my global_id is 4589 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59cc136590 con 0x7f59cc133cc0 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2394145048 0 0) 0x7f59b0003520 con 0x7f59cc1338e0 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59cc132bb0 con 0x7f59cc1338e0 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59c80018d0 con 0x7f59cc1338e0 2024-09-16T04:08:00.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): dump: 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.727 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.708+0000 7f59c27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1993948178 0 0) 0x7f59cc132bb0 con 0x7f59cc1338e0 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 >> v1:172.21.15.148:6790/0 conn(0x7f59cc133cc0 legacy=0x7f59cc1324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 >> v1:172.21.15.148:6789/0 conn(0x7f59cc1340d0 legacy=0x7f59cc1d39a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 monclient: found mon.b 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc136b50 con 0x7f59cc1338e0 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:00.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 5 monclient: authenticate success, global_id 4567 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 10 monclient: _renew_subs 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59cc136870 con 0x7f59cc1338e0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 10 monclient: _renew_subs 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f59cc1402f0 con 0x7f59cc1338e0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59c8003610 con 0x7f59cc1338e0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59c8003960 con 0x7f59cc1338e0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c3fff640 4 set_mon_vals no callback set 2024-09-16T04:08:00.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 10 monclient: dump: 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.709+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f59c8003b00 con 0x7f59cc1338e0 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.710+0000 7f59d3e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.710+0000 7f59d3e2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:00.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.710+0000 7f59d3e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.710+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b4005080 con 0x7f59cc1338e0 2024-09-16T04:08:00.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.710+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(33..33 src has 1..33) ==== 4634+0+0 (unknown 4221952135 0 0) 0x7f59c80471b0 con 0x7f59cc1338e0 2024-09-16T04:08:00.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.710+0000 7f59d13c9640 10 monclient: get_auth_request con 0x7f59b0038be0 auth_method 0 2024-09-16T04:08:00.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.714+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f59c80198b0 con 0x7f59cc1338e0 2024-09-16T04:08:00.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.714+0000 7f59c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:00.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.714+0000 7f59c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:00.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.877+0000 7f59d3e2e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T04:08:00.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.877+0000 7f59d3e2e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T04:08:00.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.877+0000 7f59d3e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:00.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:00.877+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) -- 0x7f59b4005370 con 0x7f59cc1338e0 2024-09-16T04:08:01.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.116+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v34) ==== 152+0+0 (unknown 4035156538 0 0) 0x7f59c801e150 con 0x7f59cc1338e0 2024-09-16T04:08:01.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.116+0000 7f59c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T04:08:01.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.116+0000 7f59c27fc640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-16T04:08:01.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.247+0000 7f59d3e2e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T04:08:01.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.247+0000 7f59d3e2e640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T04:08:01.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.247+0000 7f59d3e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:01.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:01.247+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) -- 0x7f59b4001140 con 0x7f59cc1338e0 2024-09-16T04:08:02.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.120+0000 7f59c27fc640 1 -- 172.21.15.171:0/1978872701 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v35) ==== 152+0+0 (unknown 1997080733 0 0) 0x7f59c8044b80 con 0x7f59cc1338e0 2024-09-16T04:08:02.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.120+0000 7f59c27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-16T04:08:02.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.120+0000 7f59c27fc640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-16T04:08:02.123 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 4 size to 1 2024-09-16T04:08:02.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.126+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 >> v1:172.21.15.171:6812/7327273 conn(0x7f59b0038be0 legacy=0x7f59b003b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:02.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.127+0000 7f59d3e2e640 10 monclient: shutdown 2024-09-16T04:08:02.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.127+0000 7f59d3e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:02.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.127+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 >> v1:172.21.15.171:6789/0 conn(0x7f59cc1338e0 legacy=0x7f59cc131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:02.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.127+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 shutdown_connections 2024-09-16T04:08:02.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.127+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 >> 172.21.15.171:0/1978872701 conn(0x7f59cc0a6460 msgr2=0x7f59cc0a4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:02.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.128+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 shutdown_connections 2024-09-16T04:08:02.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.128+0000 7f59d3e2e640 1 -- 172.21.15.171:0/1978872701 wait complete. 2024-09-16T04:08:02.156 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_config_value foo min_size 1 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:+ local pool_name config_opt expected_val val 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:+ pool_name=foo 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:+ config_opt=min_size 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_val=1 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:++ get_config_value_or_die foo min_size 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:++ local pool_name config_opt raw val 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:++ pool_name=foo 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:++ config_opt=min_size 2024-09-16T04:08:02.157 INFO:tasks.workunit.client.0.smithi171.stderr:+++ ceph osd pool get foo min_size 2024-09-16T04:08:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:++ raw='min_size: 1' 2024-09-16T04:08:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T04:08:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:+++ echo min_size: 1 2024-09-16T04:08:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T04:08:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:++ raw=min_size:1 2024-09-16T04:08:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:+++ echo min_size:1 2024-09-16T04:08:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:+++ cut -f2 -d: 2024-09-16T04:08:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:++ val=1 2024-09-16T04:08:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:++ echo 1 2024-09-16T04:08:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:++ return 0 2024-09-16T04:08:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:+ val=1 2024-09-16T04:08:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 1 != \1 ]] 2024-09-16T04:08:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set foo size 4 2024-09-16T04:08:02.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.607+0000 7f5866118640 10 monclient: get_monmap_and_config 2024-09-16T04:08:02.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.607+0000 7f5866118640 10 monclient: build_initial_monmap 2024-09-16T04:08:02.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.607+0000 7f5866118640 10 monclient: monmap: 2024-09-16T04:08:02.609 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:02.609 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:02.608347+0000 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:02.608347+0000 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:02.610 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:02.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 1 Processor -- start 2024-09-16T04:08:02.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 1 -- start start 2024-09-16T04:08:02.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient: init 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient(hunting): picked mon.noname-a con 0x7f586009b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient(hunting): picked mon.noname-b con 0x7f5860136da0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient(hunting): picked mon.noname-c con 0x7f586013d320 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5860073a20 con 0x7f586009b920 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5860099920 con 0x7f586013d320 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5860099c00 con 0x7f5860136da0 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f5866118640 10 monclient(hunting): authenticate will time out at 858.113807s 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f585ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f586013d320 0x7f586013f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56866/0 (socket says 172.21.15.171:56866) 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f585f7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f586009b920 legacy=0x7f586009bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:02.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.608+0000 7f585ffff640 1 -- 172.21.15.171:0/4045044933 learned_addr learned my addr 172.21.15.171:0/4045044933 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597141401 0 0) 0x7f5860099920 con 0x7f586013d320 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): my global_id is 4576 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5848003520 con 0x7f586013d320 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573596538 0 0) 0x7f5860099c00 con 0x7f5860136da0 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient(hunting): my global_id is 4598 2024-09-16T04:08:02.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5860099920 con 0x7f5860136da0 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4248973064 0 0) 0x7f5848003520 con 0x7f586013d320 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 >> v1:172.21.15.148:6790/0 conn(0x7f5860136da0 legacy=0x7f586013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 >> v1:172.21.15.148:6789/0 conn(0x7f586009b920 legacy=0x7f586009bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5860099ee0 con 0x7f586013d320 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 5 monclient: authenticate success, global_id 4576 2024-09-16T04:08:02.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5858001990 con 0x7f586013d320 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: dump: 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:02.621 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5858001c40 con 0x7f586013d320 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 4 set_mon_vals no callback set 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 1 -- 172.21.15.171:0/4045044933 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5858002010 con 0x7f586013d320 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f585e7fc640 10 monclient: dump: 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:02.622 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 10 monclient: shutdown 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.609+0000 7f5866118640 1 -- 172.21.15.171:0/4045044933 >> v1:172.21.15.171:6789/0 conn(0x7f586013d320 legacy=0x7f586013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 1 -- 172.21.15.171:0/4045044933 shutdown_connections 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 1 -- 172.21.15.171:0/4045044933 >> 172.21.15.171:0/4045044933 conn(0x7f58600a2150 msgr2=0x7f58600a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 1 -- 172.21.15.171:0/4045044933 shutdown_connections 2024-09-16T04:08:02.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 1 -- 172.21.15.171:0/4045044933 wait complete. 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 10 monclient: build_initial_monmap 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 10 monclient: monmap: 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:02.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.610+0000 7f5866118640 1 Processor -- start 2024-09-16T04:08:02.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 1 -- start start 2024-09-16T04:08:02.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient: init 2024-09-16T04:08:02.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:02.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:02.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient(hunting): picked mon.noname-a con 0x7f586009b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:02.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient(hunting): picked mon.noname-b con 0x7f5860136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:02.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient(hunting): picked mon.noname-c con 0x7f586013d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:02.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f586012be60 con 0x7f586009b920 2024-09-16T04:08:02.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f586013b730 con 0x7f5860136da0 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f586009c660 con 0x7f586013d320 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5866118640 10 monclient(hunting): authenticate will time out at 858.116651s 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f585effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5860136da0 0x7f586013b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56878/0 (socket says 172.21.15.171:56878) 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f585effd640 1 -- 172.21.15.171:0/4016718507 learned_addr learned my addr 172.21.15.171:0/4016718507 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f585f7fe640 1 -- 172.21.15.171:0/4016718507 >> v1:172.21.15.148:6789/0 conn(0x7f586009b920 legacy=0x7f586013a910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554578331 0 0) 0x7f586013b730 con 0x7f5860136da0 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5847fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5847fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5847fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:02.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5847fff640 10 monclient(hunting): my global_id is 4579 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.611+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f584c003520 con 0x7f5860136da0 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3390597022 0 0) 0x7f584c003520 con 0x7f5860136da0 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f586013b730 con 0x7f5860136da0 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5854001910 con 0x7f5860136da0 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): dump: 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:02.629 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251259675 0 0) 0x7f586009c660 con 0x7f586013d320 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient(hunting): my global_id is 4601 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f584c003520 con 0x7f586013d320 2024-09-16T04:08:02.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3084213100 0 0) 0x7f586013b730 con 0x7f5860136da0 2024-09-16T04:08:02.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 >> v1:172.21.15.148:6790/0 conn(0x7f586013d320 legacy=0x7f58601d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:02.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 >> v1:172.21.15.148:6789/0 conn(0x7f586009b920 legacy=0x7f586013a910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:02.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:02.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 monclient: found mon.b 2024-09-16T04:08:02.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:02.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586013c510 con 0x7f5860136da0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 5 monclient: authenticate success, global_id 4579 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 10 monclient: _renew_subs 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f586012bb80 con 0x7f5860136da0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 10 monclient: _renew_subs 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f586012a9b0 con 0x7f5860136da0 2024-09-16T04:08:02.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5854001d20 con 0x7f5860136da0 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5854003ae0 con 0x7f5860136da0 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f5847fff640 10 monclient: dump: 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:02.633 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.612+0000 7f585dffb640 4 set_mon_vals no callback set 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5854003d00 con 0x7f5860136da0 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f5866118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f5866118640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f5866118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586013cc40 con 0x7f5860136da0 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(35..35 src has 1..35) ==== 4634+0+0 (unknown 2356578225 0 0) 0x7f5854047300 con 0x7f5860136da0 2024-09-16T04:08:02.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.613+0000 7f585f7fe640 10 monclient: get_auth_request con 0x7f584c038f90 auth_method 0 2024-09-16T04:08:02.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.617+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5854015e50 con 0x7f5860136da0 2024-09-16T04:08:02.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.617+0000 7f5847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:02.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.617+0000 7f5847fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:02.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.781+0000 7f5866118640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T04:08:02.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.781+0000 7f5866118640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T04:08:02.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.781+0000 7f5866118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:02.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:02.781+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) -- 0x7f586006c630 con 0x7f5860136da0 2024-09-16T04:08:03.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.122+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v36) ==== 122+0+0 (unknown 3848321630 0 0) 0x7f5854015750 con 0x7f5860136da0 2024-09-16T04:08:03.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.122+0000 7f5847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T04:08:03.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.122+0000 7f5847fff640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-16T04:08:03.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.236+0000 7f5866118640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T04:08:03.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.236+0000 7f5866118640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T04:08:03.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.236+0000 7f5866118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:03.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:03.236+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) -- 0x7f58601d4f70 con 0x7f5860136da0 2024-09-16T04:08:04.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.123+0000 7f5847fff640 1 -- 172.21.15.171:0/4016718507 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v37) ==== 122+0+0 (unknown 1743069073 0 0) 0x7f585401e300 con 0x7f5860136da0 2024-09-16T04:08:04.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.124+0000 7f5847fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-16T04:08:04.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.124+0000 7f5847fff640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-16T04:08:04.126 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 4 size to 4 2024-09-16T04:08:04.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.126+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 >> v1:172.21.15.171:6812/7327273 conn(0x7f584c038f90 legacy=0x7f584c03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:04.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.126+0000 7f5866118640 10 monclient: shutdown 2024-09-16T04:08:04.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.126+0000 7f5866118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:04.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.126+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 >> v1:172.21.15.171:6789/0 conn(0x7f5860136da0 legacy=0x7f586013b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:04.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.126+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 shutdown_connections 2024-09-16T04:08:04.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.126+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 >> 172.21.15.171:0/4016718507 conn(0x7f58600a2150 msgr2=0x7f58600a0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:04.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.127+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 shutdown_connections 2024-09-16T04:08:04.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.127+0000 7f5866118640 1 -- 172.21.15.171:0/4016718507 wait complete. 2024-09-16T04:08:04.148 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_config_value foo min_size 2 2024-09-16T04:08:04.148 INFO:tasks.workunit.client.0.smithi171.stderr:+ local pool_name config_opt expected_val val 2024-09-16T04:08:04.148 INFO:tasks.workunit.client.0.smithi171.stderr:+ pool_name=foo 2024-09-16T04:08:04.148 INFO:tasks.workunit.client.0.smithi171.stderr:+ config_opt=min_size 2024-09-16T04:08:04.148 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_val=2 2024-09-16T04:08:04.148 INFO:tasks.workunit.client.0.smithi171.stderr:++ get_config_value_or_die foo min_size 2024-09-16T04:08:04.149 INFO:tasks.workunit.client.0.smithi171.stderr:++ local pool_name config_opt raw val 2024-09-16T04:08:04.149 INFO:tasks.workunit.client.0.smithi171.stderr:++ pool_name=foo 2024-09-16T04:08:04.149 INFO:tasks.workunit.client.0.smithi171.stderr:++ config_opt=min_size 2024-09-16T04:08:04.149 INFO:tasks.workunit.client.0.smithi171.stderr:+++ ceph osd pool get foo min_size 2024-09-16T04:08:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:++ raw='min_size: 2' 2024-09-16T04:08:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T04:08:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:+++ echo min_size: 2 2024-09-16T04:08:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T04:08:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:++ raw=min_size:2 2024-09-16T04:08:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:+++ echo min_size:2 2024-09-16T04:08:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:+++ cut -f2 -d: 2024-09-16T04:08:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:++ val=2 2024-09-16T04:08:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:++ echo 2 2024-09-16T04:08:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:++ return 0 2024-09-16T04:08:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:+ val=2 2024-09-16T04:08:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 2 != \2 ]] 2024-09-16T04:08:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set foo size 10 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.632+0000 7f78ddf2a640 10 monclient: get_monmap_and_config 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.632+0000 7f78ddf2a640 10 monclient: build_initial_monmap 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.632+0000 7f78ddf2a640 10 monclient: monmap: 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:04.633221+0000 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:04.633221+0000 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:04.642 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 1 Processor -- start 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 1 -- start start 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient: init 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient(hunting): picked mon.noname-c con 0x7f78d8136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient(hunting): picked mon.noname-a con 0x7f78d8137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient(hunting): picked mon.noname-b con 0x7f78d81375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:04.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d809b2c0 con 0x7f78d8137140 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d809b5a0 con 0x7f78d8136d60 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d80a6640 con 0x7f78d81375a0 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78ddf2a640 10 monclient(hunting): authenticate will time out at 860.138819s 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78d77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78d8136d60 0x7f78d809bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56910/0 (socket says 172.21.15.171:56910) 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78d77fe640 1 -- 172.21.15.171:0/4061967820 learned_addr learned my addr 172.21.15.171:0/4061967820 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.633+0000 7f78d6ffd640 1 -- 172.21.15.171:0/4061967820 >> v1:172.21.15.148:6789/0 conn(0x7f78d8137140 legacy=0x7f78d813bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815064109 0 0) 0x7f78d809b5a0 con 0x7f78d8136d60 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:04.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): my global_id is 4588 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c0003520 con 0x7f78d8136d60 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1073624132 0 0) 0x7f78d80a6640 con 0x7f78d81375a0 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient(hunting): my global_id is 4610 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d809b5a0 con 0x7f78d81375a0 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3254122883 0 0) 0x7f78c0003520 con 0x7f78d8136d60 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 >> v1:172.21.15.148:6790/0 conn(0x7f78d81375a0 legacy=0x7f78d813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 >> v1:172.21.15.148:6789/0 conn(0x7f78d8137140 legacy=0x7f78d813bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:04.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d8143b20 con 0x7f78d8136d60 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78c8001610 con 0x7f78d8136d60 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 5 monclient: authenticate success, global_id 4588 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:04.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: dump: 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78c80018c0 con 0x7f78d8136d60 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:04.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 4 set_mon_vals no callback set 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 1 -- 172.21.15.171:0/4061967820 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78c8001c90 con 0x7f78d8136d60 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78d67fc640 10 monclient: dump: 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:04.648 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 10 monclient: shutdown 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.634+0000 7f78ddf2a640 1 -- 172.21.15.171:0/4061967820 >> v1:172.21.15.171:6789/0 conn(0x7f78d8136d60 legacy=0x7f78d809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 1 -- 172.21.15.171:0/4061967820 shutdown_connections 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 1 -- 172.21.15.171:0/4061967820 >> 172.21.15.171:0/4061967820 conn(0x7f78d80a9ae0 msgr2=0x7f78d80a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 1 -- 172.21.15.171:0/4061967820 shutdown_connections 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 1 -- 172.21.15.171:0/4061967820 wait complete. 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 10 monclient: build_initial_monmap 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 10 monclient: monmap: 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:04.649 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.635+0000 7f78ddf2a640 1 Processor -- start 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 1 -- start start 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient: init 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:04.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient(hunting): picked mon.noname-c con 0x7f78d8136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient(hunting): picked mon.noname-a con 0x7f78d8137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient(hunting): picked mon.noname-b con 0x7f78d81375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d80a87c0 con 0x7f78d8137140 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d812f300 con 0x7f78d81375a0 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d812ed40 con 0x7f78d8136d60 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78ddf2a640 10 monclient(hunting): authenticate will time out at 860.141551s 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78d7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78d81375a0 0x7f78d81d4620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56926/0 (socket says 172.21.15.171:56926) 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78d7fff640 1 -- 172.21.15.171:0/665060380 learned_addr learned my addr 172.21.15.171:0/665060380 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78d6ffd640 1 -- 172.21.15.171:0/665060380 >> v1:172.21.15.148:6789/0 conn(0x7f78d8137140 legacy=0x7f78d81d0ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156764851 0 0) 0x7f78d812f300 con 0x7f78d81375a0 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78bffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:04.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78bffff640 10 monclient(hunting): my global_id is 4591 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.636+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c4003520 con 0x7f78d81375a0 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2644709542 0 0) 0x7f78d812ed40 con 0x7f78d8136d60 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): my global_id is 4613 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d812f300 con 0x7f78d8136d60 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2515618352 0 0) 0x7f78c4003520 con 0x7f78d81375a0 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78d812ed40 con 0x7f78d81375a0 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78d0001bb0 con 0x7f78d81375a0 2024-09-16T04:08:04.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): dump: 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:04.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 590620640 0 0) 0x7f78d812ed40 con 0x7f78d81375a0 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 >> v1:172.21.15.148:6790/0 conn(0x7f78d8136d60 legacy=0x7f78d8129c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 >> v1:172.21.15.148:6789/0 conn(0x7f78d8137140 legacy=0x7f78d81d0ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 monclient: found mon.b 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d809f7f0 con 0x7f78d81375a0 2024-09-16T04:08:04.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 5 monclient: authenticate success, global_id 4591 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 10 monclient: _renew_subs 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78d809e7c0 con 0x7f78d81375a0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 10 monclient: _renew_subs 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78d812a330 con 0x7f78d81375a0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78d0003990 con 0x7f78d81375a0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78d0003ce0 con 0x7f78d81375a0 2024-09-16T04:08:04.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78d5ffb640 4 set_mon_vals no callback set 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.637+0000 7f78bffff640 10 monclient: dump: 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:04.656 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f78d0003e80 con 0x7f78d81375a0 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(37..37 src has 1..37) ==== 4634+0+0 (unknown 335752939 0 0) 0x7f78d0047340 con 0x7f78d81375a0 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78d77fe640 10 monclient: get_auth_request con 0x7f78c4039000 auth_method 0 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78ddf2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78ddf2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78ddf2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:04.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.638+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b8005080 con 0x7f78d81375a0 2024-09-16T04:08:04.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.642+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f78d0015e90 con 0x7f78d81375a0 2024-09-16T04:08:04.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.642+0000 7f78bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:04.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.642+0000 7f78bffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:04.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.808+0000 7f78ddf2a640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T04:08:04.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.808+0000 7f78ddf2a640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T04:08:04.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.808+0000 7f78ddf2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:04.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:04.808+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) -- 0x7f78b8005b80 con 0x7f78d81375a0 2024-09-16T04:08:05.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.146+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v38) ==== 124+0+0 (unknown 2690976392 0 0) 0x7f78d0015790 con 0x7f78d81375a0 2024-09-16T04:08:05.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.146+0000 7f78bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T04:08:05.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.146+0000 7f78bffff640 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-16T04:08:05.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.300+0000 7f78ddf2a640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T04:08:05.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.300+0000 7f78ddf2a640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T04:08:05.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.300+0000 7f78ddf2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:05.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:05.300+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) -- 0x7f78b8005760 con 0x7f78d81375a0 2024-09-16T04:08:06.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.151+0000 7f78bffff640 1 -- 172.21.15.171:0/665060380 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v39) ==== 124+0+0 (unknown 3564249015 0 0) 0x7f78d001e340 con 0x7f78d81375a0 2024-09-16T04:08:06.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.151+0000 7f78bffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-16T04:08:06.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.151+0000 7f78bffff640 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-16T04:08:06.153 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 4 size to 10 2024-09-16T04:08:06.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.154+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 >> v1:172.21.15.171:6812/7327273 conn(0x7f78c4039000 legacy=0x7f78c403b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.154+0000 7f78ddf2a640 10 monclient: shutdown 2024-09-16T04:08:06.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.154+0000 7f78ddf2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:06.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.154+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 >> v1:172.21.15.171:6789/0 conn(0x7f78d81375a0 legacy=0x7f78d81d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.155+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 shutdown_connections 2024-09-16T04:08:06.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.155+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 >> 172.21.15.171:0/665060380 conn(0x7f78d80a9ae0 msgr2=0x7f78d8072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.161+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 shutdown_connections 2024-09-16T04:08:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.162+0000 7f78ddf2a640 1 -- 172.21.15.171:0/665060380 wait complete. 2024-09-16T04:08:06.186 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_config_value foo min_size 5 2024-09-16T04:08:06.186 INFO:tasks.workunit.client.0.smithi171.stderr:+ local pool_name config_opt expected_val val 2024-09-16T04:08:06.186 INFO:tasks.workunit.client.0.smithi171.stderr:+ pool_name=foo 2024-09-16T04:08:06.186 INFO:tasks.workunit.client.0.smithi171.stderr:+ config_opt=min_size 2024-09-16T04:08:06.186 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_val=5 2024-09-16T04:08:06.187 INFO:tasks.workunit.client.0.smithi171.stderr:++ get_config_value_or_die foo min_size 2024-09-16T04:08:06.187 INFO:tasks.workunit.client.0.smithi171.stderr:++ local pool_name config_opt raw val 2024-09-16T04:08:06.187 INFO:tasks.workunit.client.0.smithi171.stderr:++ pool_name=foo 2024-09-16T04:08:06.187 INFO:tasks.workunit.client.0.smithi171.stderr:++ config_opt=min_size 2024-09-16T04:08:06.187 INFO:tasks.workunit.client.0.smithi171.stderr:+++ ceph osd pool get foo min_size 2024-09-16T04:08:06.588 INFO:tasks.workunit.client.0.smithi171.stderr:++ raw='min_size: 5' 2024-09-16T04:08:06.589 INFO:tasks.workunit.client.0.smithi171.stderr:++ [[ 0 -ne 0 ]] 2024-09-16T04:08:06.589 INFO:tasks.workunit.client.0.smithi171.stderr:+++ echo min_size: 5 2024-09-16T04:08:06.589 INFO:tasks.workunit.client.0.smithi171.stderr:+++ sed -e 's/[{} "]//g' 2024-09-16T04:08:06.590 INFO:tasks.workunit.client.0.smithi171.stderr:++ raw=min_size:5 2024-09-16T04:08:06.590 INFO:tasks.workunit.client.0.smithi171.stderr:+++ echo min_size:5 2024-09-16T04:08:06.591 INFO:tasks.workunit.client.0.smithi171.stderr:+++ cut -f2 -d: 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:++ val=5 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:++ echo 5 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:++ return 0 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:+ val=5 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 5 != \5 ]] 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-16T04:08:06.592 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:06.593 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set foo size 0 2024-09-16T04:08:06.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.669+0000 7fbaeb523640 10 monclient: get_monmap_and_config 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.669+0000 7fbaeb523640 10 monclient: build_initial_monmap 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.669+0000 7fbaeb523640 10 monclient: monmap: 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:06.670331+0000 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:06.670331+0000 2024-09-16T04:08:06.672 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 1 Processor -- start 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 1 -- start start 2024-09-16T04:08:06.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient: init 2024-09-16T04:08:06.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient(hunting): picked mon.noname-c con 0x7fbae40a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient(hunting): picked mon.noname-a con 0x7fbae409f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient(hunting): picked mon.noname-b con 0x7fbae413d2c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae40737b0 con 0x7fbae409f320 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae406c6d0 con 0x7fbae40a0ae0 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae406d980 con 0x7fbae413d2c0 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbaeb523640 10 monclient(hunting): authenticate will time out at 862.175876s 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbae92bf640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbae40a0ae0 0x7fbae40a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56950/0 (socket says 172.21.15.171:56950) 2024-09-16T04:08:06.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbae92bf640 1 -- 172.21.15.171:0/529233660 learned_addr learned my addr 172.21.15.171:0/529233660 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.670+0000 7fbae8abe640 1 -- 172.21.15.171:0/529233660 >> v1:172.21.15.148:6789/0 conn(0x7fbae409f320 legacy=0x7fbae413bba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 881418192 0 0) 0x7fbae406c6d0 con 0x7fbae40a0ae0 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient(hunting): my global_id is 4600 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad0003520 con 0x7fbae40a0ae0 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 256326580 0 0) 0x7fbad0003520 con 0x7fbae40a0ae0 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 >> v1:172.21.15.148:6790/0 conn(0x7fbae413d2c0 legacy=0x7fbae413f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 >> v1:172.21.15.148:6789/0 conn(0x7fbae409f320 legacy=0x7fbae413bba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:06.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 monclient: found mon.noname-c 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae406c950 con 0x7fbae40a0ae0 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: _finish_auth 0 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbad4001610 con 0x7fbae40a0ae0 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: dump: 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.683 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbaeb523640 5 monclient: authenticate success, global_id 4600 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbaeb523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbad4001970 con 0x7fbae40a0ae0 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:06.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbaeb523640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 1 -- 172.21.15.171:0/529233660 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbad4001d40 con 0x7fbae40a0ae0 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbaeb523640 4 set_mon_vals no callback set 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbadbfff640 10 monclient: dump: 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.685 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbaeb523640 10 monclient: shutdown 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.671+0000 7fbaeb523640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 1 -- 172.21.15.171:0/529233660 >> v1:172.21.15.171:6789/0 conn(0x7fbae40a0ae0 legacy=0x7fbae40a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 1 -- 172.21.15.171:0/529233660 shutdown_connections 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 1 -- 172.21.15.171:0/529233660 >> 172.21.15.171:0/529233660 conn(0x7fbae40a77f0 msgr2=0x7fbae40a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 1 -- 172.21.15.171:0/529233660 shutdown_connections 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 1 -- 172.21.15.171:0/529233660 wait complete. 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 10 monclient: build_initial_monmap 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 10 monclient: monmap: 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:06.686 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.672+0000 7fbaeb523640 1 Processor -- start 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 1 -- start start 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient: init 2024-09-16T04:08:06.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient(hunting): picked mon.noname-a con 0x7fbae409f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient(hunting): picked mon.noname-c con 0x7fbae40a0ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient(hunting): picked mon.noname-b con 0x7fbae413d2c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae4127680 con 0x7fbae409f320 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae40a3f30 con 0x7fbae413d2c0 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbae92bf640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbae409f320 legacy=0x7fbae41cb6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae409e1b0 con 0x7fbae40a0ae0 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbaeb523640 10 monclient(hunting): authenticate will time out at 862.178572s 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbae9ac0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbae413d2c0 0x7fbae41d3db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56966/0 (socket says 172.21.15.171:56966) 2024-09-16T04:08:06.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbae9ac0640 1 -- 172.21.15.171:0/1690344137 learned_addr learned my addr 172.21.15.171:0/1690344137 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961004891 0 0) 0x7fbae40a3f30 con 0x7fbae413d2c0 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): my global_id is 4603 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbac8003520 con 0x7fbae413d2c0 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2400758492 0 0) 0x7fbae409e1b0 con 0x7fbae40a0ae0 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:06.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 10 monclient(hunting): my global_id is 4625 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae40a3f30 con 0x7fbae40a0ae0 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.673+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3284152703 0 0) 0x7fbac8003520 con 0x7fbae413d2c0 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbae409e1b0 con 0x7fbae413d2c0 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbadc001cc0 con 0x7fbae413d2c0 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.690 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2860702012 0 0) 0x7fbae40a3f30 con 0x7fbae40a0ae0 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbac8003520 con 0x7fbae40a0ae0 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbae00040f0 con 0x7fbae40a0ae0 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:06.691 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1360870904 0 0) 0x7fbae409e1b0 con 0x7fbae413d2c0 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 >> v1:172.21.15.148:6790/0 conn(0x7fbae40a0ae0 legacy=0x7fbae41d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 >> v1:172.21.15.148:6789/0 conn(0x7fbae409f320 legacy=0x7fbae41cb6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:06.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 monclient: found mon.b 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae41270c0 con 0x7fbae413d2c0 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 5 monclient: authenticate success, global_id 4603 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 10 monclient: _renew_subs 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbae41273a0 con 0x7fbae413d2c0 2024-09-16T04:08:06.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbadc002070 con 0x7fbae413d2c0 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbadc003de0 con 0x7fbae413d2c0 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbad9ffb640 10 monclient: dump: 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.694 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbadb7fe640 4 set_mon_vals no callback set 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 10 monclient: _renew_subs 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.674+0000 7fbaeb523640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbae4124790 con 0x7fbae413d2c0 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbadc003f80 con 0x7fbae413d2c0 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbaeb523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbaeb523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbaeb523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbaeb523640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae4125390 con 0x7fbae413d2c0 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7fbadc047480 con 0x7fbae413d2c0 2024-09-16T04:08:06.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.675+0000 7fbae92bf640 10 monclient: get_auth_request con 0x7fbac8038f10 auth_method 0 2024-09-16T04:08:06.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.680+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbadc0047c0 con 0x7fbae413d2c0 2024-09-16T04:08:06.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.680+0000 7fbad9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:06.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.680+0000 7fbad9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbaeb523640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbaeb523640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbaeb523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbaeb523640 1 -- 172.21.15.171:0/1690344137 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) -- 0x7fbae406df70 con 0x7fbae413d2c0 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbad9ffb640 1 -- 172.21.15.171:0/1690344137 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v39) ==== 136+0+0 (unknown 1829963189 0 0) 0x7fbadc019bc0 con 0x7fbae413d2c0 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbad9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-16T04:08:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.843+0000 7fbad9ffb640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T04:08:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T04:08:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-16T04:08:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set foo size 20 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.927+0000 7f33c1642640 10 monclient: get_monmap_and_config 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.927+0000 7f33c1642640 10 monclient: build_initial_monmap 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.927+0000 7f33c1642640 10 monclient: monmap: 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:06.928168+0000 2024-09-16T04:08:06.929 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:06.928168+0000 2024-09-16T04:08:06.930 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:06.930 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.930 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:06.930 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:06.930 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:06.930 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.927+0000 7f33c1642640 1 Processor -- start 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 1 -- start start 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient: init 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient(hunting): picked mon.noname-b con 0x7f33bc136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient(hunting): picked mon.noname-c con 0x7f33bc137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient(hunting): picked mon.noname-a con 0x7f33bc1375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:06.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc09b2c0 con 0x7f33bc1375a0 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc09b5a0 con 0x7f33bc137140 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc0a6640 con 0x7f33bc136d60 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33c1642640 10 monclient(hunting): authenticate will time out at 862.433634s 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33ba7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f33bc137140 0x7f33bc13bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56968/0 (socket says 172.21.15.171:56968) 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33bb7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f33bc1375a0 legacy=0x7f33bc13f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33ba7fc640 1 -- 172.21.15.171:0/1221861647 learned_addr learned my addr 172.21.15.171:0/1221861647 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33ba7fc640 1 -- 172.21.15.171:0/1221861647 >> v1:172.21.15.171:6789/0 conn(0x7f33bc137140 legacy=0x7f33bc13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33ba7fc640 1 -- 172.21.15.171:0/1221861647 >> v1:172.21.15.171:6789/0 conn(0x7f33bc137140 legacy=0x7f33bc13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:06.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.928+0000 7f33ba7fc640 1 --1- 172.21.15.171:0/1221861647 >> v1:172.21.15.171:6789/0 conn(0x7f33bc137140 0x7f33bc13bca0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 590223517 0 0) 0x7f33bc0a6640 con 0x7f33bc136d60 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient(hunting): my global_id is 4628 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33a8003520 con 0x7f33bc136d60 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4101727982 0 0) 0x7f33a8003520 con 0x7f33bc136d60 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 >> v1:172.21.15.171:6789/0 conn(0x7f33bc137140 legacy=0x7f33bc13bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 >> v1:172.21.15.148:6789/0 conn(0x7f33bc1375a0 legacy=0x7f33bc13f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 monclient: found mon.noname-b 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:06.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc143b20 con 0x7f33bc136d60 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33ac001610 con 0x7f33bc136d60 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33c1642640 5 monclient: authenticate success, global_id 4628 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33c1642640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: dump: 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.941 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33c1642640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33ac001940 con 0x7f33bc136d60 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33c1642640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33c1642640 4 set_mon_vals no callback set 2024-09-16T04:08:06.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 1 -- 172.21.15.171:0/1221861647 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33ac001d10 con 0x7f33bc136d60 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.929+0000 7f33b9ffb640 10 monclient: dump: 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.943 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 10 monclient: shutdown 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 1 -- 172.21.15.171:0/1221861647 >> v1:172.21.15.148:6790/0 conn(0x7f33bc136d60 legacy=0x7f33bc09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 1 -- 172.21.15.171:0/1221861647 shutdown_connections 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 1 -- 172.21.15.171:0/1221861647 >> 172.21.15.171:0/1221861647 conn(0x7f33bc0a9ae0 msgr2=0x7f33bc0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 1 -- 172.21.15.171:0/1221861647 shutdown_connections 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 1 -- 172.21.15.171:0/1221861647 wait complete. 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 10 monclient: build_initial_monmap 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.930+0000 7f33c1642640 10 monclient: monmap: 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:06.944 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 1 Processor -- start 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 1 -- start start 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient: init 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient(hunting): picked mon.noname-b con 0x7f33bc136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient(hunting): picked mon.noname-a con 0x7f33bc137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient(hunting): picked mon.noname-c con 0x7f33bc1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc13a6a0 con 0x7f33bc137140 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc13f960 con 0x7f33bc136d60 2024-09-16T04:08:06.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33bc138e10 con 0x7f33bc1375a0 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33c1642640 10 monclient(hunting): authenticate will time out at 862.436802s 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33baffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f33bc136d60 0x7f33bc137a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56972/0 (socket says 172.21.15.171:56972) 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33baffd640 1 -- 172.21.15.171:0/2560601494 learned_addr learned my addr 172.21.15.171:0/2560601494 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.931+0000 7f33ba7fc640 1 -- 172.21.15.171:0/2560601494 >> v1:172.21.15.148:6789/0 conn(0x7f33bc137140 legacy=0x7f33bc138180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 748095551 0 0) 0x7f33bc13f960 con 0x7f33bc136d60 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:06.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): my global_id is 4606 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f339c003520 con 0x7f33bc136d60 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1440765965 0 0) 0x7f33bc138e10 con 0x7f33bc1375a0 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): my global_id is 4631 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33bc13f960 con 0x7f33bc1375a0 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1793463240 0 0) 0x7f339c003520 con 0x7f33bc136d60 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33bc138e10 con 0x7f33bc136d60 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33ac001690 con 0x7f33bc136d60 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:06.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): dump: 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 70463062 0 0) 0x7f33bc13f960 con 0x7f33bc1375a0 2024-09-16T04:08:06.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f339c003520 con 0x7f33bc1375a0 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33b40043b0 con 0x7f33bc1375a0 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient(hunting): dump: 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.950 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1055051824 0 0) 0x7f33bc138e10 con 0x7f33bc136d60 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 >> v1:172.21.15.148:6790/0 conn(0x7f33bc1375a0 legacy=0x7f33bc1d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 >> v1:172.21.15.148:6789/0 conn(0x7f33bc137140 legacy=0x7f33bc138180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:06.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 monclient: found mon.b 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc13a980 con 0x7f33bc136d60 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 5 monclient: authenticate success, global_id 4606 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 10 monclient: _renew_subs 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f33bc13a3c0 con 0x7f33bc136d60 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 10 monclient: _renew_subs 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33c1642640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f33bc138890 con 0x7f33bc136d60 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33ac001940 con 0x7f33bc136d60 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33ac003bd0 con 0x7f33bc136d60 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b97fa640 4 set_mon_vals no callback set 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.932+0000 7f33b37fe640 10 monclient: dump: 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:06.954 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f33ac003df0 con 0x7f33bc136d60 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33c1642640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33c1642640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33c1642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33c1642640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a0005080 con 0x7f33bc136d60 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7f33ac0472f0 con 0x7f33bc136d60 2024-09-16T04:08:06.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.933+0000 7f33ba7fc640 10 monclient: get_auth_request con 0x7f339c038ec0 auth_method 0 2024-09-16T04:08:06.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.938+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f33ac015de0 con 0x7f33bc136d60 2024-09-16T04:08:06.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.938+0000 7f33b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:06.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:06.938+0000 7f33b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:07.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33c1642640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T04:08:07.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33c1642640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T04:08:07.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33c1642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33c1642640 1 -- 172.21.15.171:0/2560601494 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) -- 0x7f33a0005b80 con 0x7f33bc136d60 2024-09-16T04:08:07.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33b37fe640 1 -- 172.21.15.171:0/2560601494 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v39) ==== 137+0+0 (unknown 1510511677 0 0) 0x7f33ac01e290 con 0x7f33bc136d60 2024-09-16T04:08:07.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-16T04:08:07.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.103+0000 7f33b37fe640 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-16T04:08:07.106 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-16T04:08:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool delete foo 2024-09-16T04:08:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo 2024-09-16T04:08:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.187+0000 7f92d679d640 10 monclient: get_monmap_and_config 2024-09-16T04:08:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.187+0000 7f92d679d640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.187+0000 7f92d679d640 10 monclient: monmap: 2024-09-16T04:08:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:07.188591+0000 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:07.188591+0000 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:07.190 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.188+0000 7f92d679d640 1 Processor -- start 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.188+0000 7f92d679d640 1 -- start start 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.188+0000 7f92d679d640 10 monclient: init 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient(hunting): picked mon.noname-a con 0x7f92d00a2a30 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient(hunting): picked mon.noname-c con 0x7f92d0136d20 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient(hunting): picked mon.noname-b con 0x7f92d013d270 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d0137130 con 0x7f92d00a2a30 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d009fec0 con 0x7f92d0136d20 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d00a0a30 con 0x7f92d013d270 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient(hunting): authenticate will time out at 862.694137s 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d579b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f92d00a2a30 legacy=0x7f92d00a2e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d4f9a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f92d0136d20 0x7f92d013bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56984/0 (socket says 172.21.15.171:56984) 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d4f9a640 1 -- 172.21.15.171:0/3105854310 learned_addr learned my addr 172.21.15.171:0/3105854310 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456207661 0 0) 0x7f92d009fec0 con 0x7f92d0136d20 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): my global_id is 4609 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92b8003520 con 0x7f92d0136d20 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847293695 0 0) 0x7f92d00a0a30 con 0x7f92d013d270 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient(hunting): my global_id is 4634 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92d009fec0 con 0x7f92d013d270 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2337195702 0 0) 0x7f92b8003520 con 0x7f92d0136d20 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 >> v1:172.21.15.148:6790/0 conn(0x7f92d013d270 legacy=0x7f92d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 >> v1:172.21.15.148:6789/0 conn(0x7f92d00a2a30 legacy=0x7f92d00a2e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 monclient: found mon.noname-c 2024-09-16T04:08:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d00a0d10 con 0x7f92d0136d20 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 5 monclient: authenticate success, global_id 4609 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92c4001610 con 0x7f92d0136d20 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: dump: 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92c4001870 con 0x7f92d0136d20 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 1 -- 172.21.15.171:0/3105854310 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92c4001bc0 con 0x7f92d0136d20 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92d679d640 4 set_mon_vals no callback set 2024-09-16T04:08:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.189+0000 7f92cffff640 10 monclient: dump: 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 10 monclient: shutdown 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 1 -- 172.21.15.171:0/3105854310 >> v1:172.21.15.171:6789/0 conn(0x7f92d0136d20 legacy=0x7f92d013bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 1 -- 172.21.15.171:0/3105854310 shutdown_connections 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 1 -- 172.21.15.171:0/3105854310 >> 172.21.15.171:0/3105854310 conn(0x7f92d00a9390 msgr2=0x7f92d00a97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 1 -- 172.21.15.171:0/3105854310 shutdown_connections 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 1 -- 172.21.15.171:0/3105854310 wait complete. 2024-09-16T04:08:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.190+0000 7f92d679d640 10 monclient: monmap: 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 1 Processor -- start 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 1 -- start start 2024-09-16T04:08:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient: init 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient(hunting): picked mon.noname-a con 0x7f92d00a2a30 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient(hunting): picked mon.noname-c con 0x7f92d0136d20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient(hunting): picked mon.noname-b con 0x7f92d013d270 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d0120a80 con 0x7f92d00a2a30 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d011f8b0 con 0x7f92d013d270 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d0120d60 con 0x7f92d0136d20 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d679d640 10 monclient(hunting): authenticate will time out at 862.696708s 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d579b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f92d00a2a30 legacy=0x7f92d0135310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d5f9c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f92d013d270 0x7f92d01d3fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56992/0 (socket says 172.21.15.171:56992) 2024-09-16T04:08:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92d5f9c640 1 -- 172.21.15.171:0/2763218904 learned_addr learned my addr 172.21.15.171:0/2763218904 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679367854 0 0) 0x7f92d011f8b0 con 0x7f92d013d270 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92cdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92cdffb640 10 monclient(hunting): my global_id is 4612 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.191+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92bc003520 con 0x7f92d013d270 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220765738 0 0) 0x7f92d0120d60 con 0x7f92d0136d20 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): my global_id is 4637 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92d011f8b0 con 0x7f92d0136d20 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2552397857 0 0) 0x7f92bc003520 con 0x7f92d013d270 2024-09-16T04:08:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92d0120d60 con 0x7f92d013d270 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92c0001cb0 con 0x7f92d013d270 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): dump: 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.210 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2575849931 0 0) 0x7f92d0120d60 con 0x7f92d013d270 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.148:6790/0 conn(0x7f92d0136d20 legacy=0x7f92d01d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.148:6789/0 conn(0x7f92d00a2a30 legacy=0x7f92d0135310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 monclient: found mon.b 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d013cd20 con 0x7f92d013d270 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 5 monclient: authenticate success, global_id 4612 2024-09-16T04:08:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 10 monclient: _renew_subs 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92d01207a0 con 0x7f92d013d270 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92c0002080 con 0x7f92d013d270 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92c0003de0 con 0x7f92d013d270 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cdffb640 10 monclient: dump: 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 10 monclient: _renew_subs 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92cf7fe640 4 set_mon_vals no callback set 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.192+0000 7f92d679d640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f92d013b4c0 con 0x7f92d013d270 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d5f9c640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.171:6789/0 conn(0x7f92d013d270 legacy=0x7f92d01d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f92c0003f80 con 0x7f92d013d270 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d5f9c640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.171:6789/0 conn(0x7f92d013d270 legacy=0x7f92d01d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d5f9c640 1 --1- 172.21.15.171:0/2763218904 >> v1:172.21.15.171:6789/0 conn(0x7f92d013d270 0x7f92d01d3fe0 :-1 s=OPENED pgs=209 cs=1 l=1).handle_message read tag failed 2024-09-16T04:08:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d5f9c640 1 --1- 172.21.15.171:0/2763218904 >> v1:172.21.15.171:6789/0 conn(0x7f92d013d270 0x7f92d01d3fe0 :-1 s=OPENED pgs=209 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.171:6789/0 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.171:6789/0 conn(0x7f92d013d270 legacy=0x7f92d01d3fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient(hunting): picked mon.a con 0x7f92d00a2a30 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient(hunting): picked mon.c con 0x7f92d0136d20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient(hunting): picked mon.b con 0x7f92bc043e20 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92d0120d60 con 0x7f92d00a2a30 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92d0120a80 con 0x7f92bc043e20 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92d011f8b0 con 0x7f92d0136d20 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92cdffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d4f9a640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.148:6789/0 conn(0x7f92d00a2a30 legacy=0x7f92bc03e870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d679d640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.193+0000 7f92d579b640 10 monclient(hunting): get_auth_request con 0x7f92bc038b90 auth_method 0 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697384641 0 0) 0x7f92d0120a80 con 0x7f92bc043e20 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f92bc048f40 con 0x7f92bc043e20 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361918068 0 0) 0x7f92d011f8b0 con 0x7f92d0136d20 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.148:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f92d0120a80 con 0x7f92d0136d20 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 2917273341 0 0) 0x7f92bc048f40 con 0x7f92bc043e20 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.148:6790/0 conn(0x7f92d0136d20 legacy=0x7f92bc042290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 >> v1:172.21.15.148:6789/0 conn(0x7f92d00a2a30 legacy=0x7f92bc03e870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 monclient: found mon.b 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f92bc003520 con 0x7f92bc043e20 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d0120fa0 con 0x7f92bc043e20 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d0120d60 con 0x7f92bc043e20 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92c8005320 con 0x7f92bc043e20 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: dump: 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.217 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92c8005180 con 0x7f92bc043e20 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.194+0000 7f92ceffd640 4 set_mon_vals no callback set 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.195+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f92c8005590 con 0x7f92bc043e20 2024-09-16T04:08:07.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.195+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 6 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7f92c8048730 con 0x7f92bc043e20 2024-09-16T04:08:07.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.199+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f92c801f810 con 0x7f92bc043e20 2024-09-16T04:08:07.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.199+0000 7f92cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.199+0000 7f92cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:07.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.203+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f92c8044320 con 0x7f92bc043e20 2024-09-16T04:08:07.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.203+0000 7f92cdffb640 10 monclient: handle_mon_command_ack 1 not found 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92d679d640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92d679d640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92d679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92d679d640 1 -- 172.21.15.171:0/2763218904 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) -- 0x7f92d006bc80 con 0x7f92bc043e20 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92cdffb640 1 -- 172.21.15.171:0/2763218904 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v39) ==== 274+0+0 (unknown 4272810089 0 0) 0x7f92c8046240 con 0x7f92bc043e20 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.367+0000 7f92cdffb640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T04:08:07.369 INFO:tasks.workunit.client.0.smithi171.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T04:08:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-16T04:08:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo foo 2024-09-16T04:08:07.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.450+0000 7fd5a4923640 10 monclient: get_monmap_and_config 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.450+0000 7fd5a4923640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.450+0000 7fd5a4923640 10 monclient: monmap: 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:07.451729+0000 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:07.451729+0000 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:07.453 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.451+0000 7fd5a4923640 1 Processor -- start 2024-09-16T04:08:07.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.451+0000 7fd5a4923640 1 -- start start 2024-09-16T04:08:07.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.451+0000 7fd5a4923640 10 monclient: init 2024-09-16T04:08:07.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a0072a70 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a0070250 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a013afe0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a009ed80 con 0x7fd5a0072a70 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a00a0030 con 0x7fd5a0070250 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a00a0310 con 0x7fd5a013afe0 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient(hunting): authenticate will time out at 862.957354s 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59e59c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd5a0072a70 legacy=0x7fd5a0072e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd5a0070250 0x7fd5a01398c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52170/0 (socket says 172.21.15.171:52170) 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59dd9b640 1 -- 172.21.15.171:0/1483010699 learned_addr learned my addr 172.21.15.171:0/1483010699 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596529913 0 0) 0x7fd5a00a0030 con 0x7fd5a0070250 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): my global_id is 4615 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd588003520 con 0x7fd5a0070250 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958291799 0 0) 0x7fd5a00a0310 con 0x7fd5a013afe0 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient(hunting): my global_id is 4640 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5a00a0030 con 0x7fd5a013afe0 2024-09-16T04:08:07.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3334674792 0 0) 0x7fd588003520 con 0x7fd5a0070250 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 >> v1:172.21.15.148:6790/0 conn(0x7fd5a013afe0 legacy=0x7fd5a013d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 >> v1:172.21.15.148:6789/0 conn(0x7fd5a0072a70 legacy=0x7fd5a0072e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 monclient: found mon.noname-c 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a009f090 con 0x7fd5a0070250 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd594001610 con 0x7fd5a0070250 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 5 monclient: authenticate success, global_id 4615 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: dump: 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd594001930 con 0x7fd5a0070250 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd59d59a640 1 -- 172.21.15.171:0/1483010699 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd594001d00 con 0x7fd5a0070250 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.452+0000 7fd5a4923640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 4 set_mon_vals no callback set 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd59d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd59d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd59d59a640 10 monclient: dump: 2024-09-16T04:08:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 10 monclient: shutdown 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 1 -- 172.21.15.171:0/1483010699 >> v1:172.21.15.171:6789/0 conn(0x7fd5a0070250 legacy=0x7fd5a01398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 1 -- 172.21.15.171:0/1483010699 shutdown_connections 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 1 -- 172.21.15.171:0/1483010699 >> 172.21.15.171:0/1483010699 conn(0x7fd5a00a7520 msgr2=0x7fd5a00a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 1 -- 172.21.15.171:0/1483010699 shutdown_connections 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.453+0000 7fd5a4923640 1 -- 172.21.15.171:0/1483010699 wait complete. 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.454+0000 7fd5a4923640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.454+0000 7fd5a4923640 10 monclient: monmap: 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.454+0000 7fd5a4923640 1 Processor -- start 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.454+0000 7fd5a4923640 1 -- start start 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.454+0000 7fd5a4923640 10 monclient: init 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient(hunting): picked mon.noname-a con 0x7fd5a0070250 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient(hunting): picked mon.noname-b con 0x7fd5a0072a70 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient(hunting): picked mon.noname-c con 0x7fd5a013afe0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a01cc1a0 con 0x7fd5a0070250 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a01271b0 con 0x7fd5a0072a70 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a013a900 con 0x7fd5a013afe0 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd5a4923640 10 monclient(hunting): authenticate will time out at 862.960244s 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59e59c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd5a0070250 legacy=0x7fd5a01397f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59dd9b640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fd5a0072a70 legacy=0x7fd5a01d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59dd9b640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7fd5a0072a70 legacy=0x7fd5a01d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd5a0072a70 0x7fd5a01d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd5a0072a70 0x7fd5a01d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T04:08:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.171:6789/0 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59ed9d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd5a013afe0 0x7fd5a01d3cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.171:51446/0 (socket says 172.21.15.171:51446) 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd59ed9d640 1 -- 172.21.15.171:0/1556749334 learned_addr learned my addr 172.21.15.171:0/1556749334 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796395897 0 0) 0x7fd5a013a900 con 0x7fd5a013afe0 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): my global_id is 4643 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd580003520 con 0x7fd5a013afe0 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4125209378 0 0) 0x7fd580003520 con 0x7fd5a013afe0 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5a013a900 con 0x7fd5a013afe0 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd598001c50 con 0x7fd5a013afe0 2024-09-16T04:08:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): dump: 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.455+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1772355369 0 0) 0x7fd5a013a900 con 0x7fd5a013afe0 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 >> v1:172.21.15.171:6789/0 conn(0x7fd5a0072a70 legacy=0x7fd5a01d05c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 >> v1:172.21.15.148:6789/0 conn(0x7fd5a0070250 legacy=0x7fd5a01397f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 1 monclient: found mon.c 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a009e0d0 con 0x7fd5a013afe0 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 5 monclient: authenticate success, global_id 4643 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 monclient: _renew_subs 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a0126bf0 con 0x7fd5a013afe0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 monclient: _renew_subs 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5a0125750 con 0x7fd5a013afe0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd598002060 con 0x7fd5a013afe0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd598003de0 con 0x7fd5a013afe0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd59cd99640 4 set_mon_vals no callback set 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd58effd640 10 monclient: dump: 2024-09-16T04:08:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.456+0000 7fd5a4923640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd584005080 con 0x7fd5a013afe0 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.457+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd598003f80 con 0x7fd5a013afe0 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.457+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7fd598047530 con 0x7fd5a013afe0 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.457+0000 7fd59e59c640 10 monclient: get_auth_request con 0x7fd580038ec0 auth_method 0 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.462+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd598016020 con 0x7fd5a013afe0 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.462+0000 7fd58effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.462+0000 7fd58effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:07.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.628+0000 7fd5a4923640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T04:08:07.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.628+0000 7fd5a4923640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T04:08:07.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.628+0000 7fd5a4923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:07.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.628+0000 7fd5a4923640 1 -- 172.21.15.171:0/1556749334 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) -- 0x7fd584005b80 con 0x7fd5a013afe0 2024-09-16T04:08:07.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.629+0000 7fd58effd640 1 -- 172.21.15.171:0/1556749334 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v39) ==== 290+0+0 (unknown 2710066426 0 0) 0x7fd598015920 con 0x7fd5a013afe0 2024-09-16T04:08:07.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.629+0000 7fd58effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-16T04:08:07.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.629+0000 7fd58effd640 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T04:08:07.631 INFO:tasks.workunit.client.0.smithi171.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-16T04:08:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-16T04:08:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo foo --force 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.714+0000 7f40370d7640 10 monclient: get_monmap_and_config 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.714+0000 7f40370d7640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.714+0000 7f40370d7640 10 monclient: monmap: 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:07.714939+0000 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:07.714939+0000 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.716 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:07.717 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:07.717 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.714+0000 7f40370d7640 1 Processor -- start 2024-09-16T04:08:07.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.714+0000 7f40370d7640 1 -- start start 2024-09-16T04:08:07.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.714+0000 7f40370d7640 10 monclient: init 2024-09-16T04:08:07.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient(hunting): picked mon.noname-a con 0x7f4030136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient(hunting): picked mon.noname-b con 0x7f4030137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient(hunting): picked mon.noname-c con 0x7f40301375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403009b2c0 con 0x7f4030136d60 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403009b5a0 con 0x7f40301375a0 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40300a6640 con 0x7f4030137140 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f40370d7640 10 monclient(hunting): authenticate will time out at 863.220456s 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f4034e73640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4030136d60 legacy=0x7f403009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f4035674640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f40301375a0 0x7f403013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52190/0 (socket says 172.21.15.171:52190) 2024-09-16T04:08:07.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f4035674640 1 -- 172.21.15.171:0/1045581444 learned_addr learned my addr 172.21.15.171:0/1045581444 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4675412 0 0) 0x7f403009b5a0 con 0x7f40301375a0 2024-09-16T04:08:07.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): my global_id is 4618 2024-09-16T04:08:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4018003520 con 0x7f40301375a0 2024-09-16T04:08:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174581661 0 0) 0x7f40300a6640 con 0x7f4030137140 2024-09-16T04:08:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 10 monclient(hunting): my global_id is 4646 2024-09-16T04:08:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f403009b5a0 con 0x7f4030137140 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.715+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2984216568 0 0) 0x7f4018003520 con 0x7f40301375a0 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 >> v1:172.21.15.148:6790/0 conn(0x7f4030137140 legacy=0x7f403013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 >> v1:172.21.15.148:6789/0 conn(0x7f4030136d60 legacy=0x7f403009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4030143b20 con 0x7f40301375a0 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4020001590 con 0x7f40301375a0 2024-09-16T04:08:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 5 monclient: authenticate success, global_id 4618 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: dump: 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.730 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40200017f0 con 0x7f40301375a0 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 1 -- 172.21.15.171:0/1045581444 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4020001b40 con 0x7f40301375a0 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 4 set_mon_vals no callback set 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f402f7fe640 10 monclient: dump: 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 10 monclient: shutdown 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 1 -- 172.21.15.171:0/1045581444 >> v1:172.21.15.171:6789/0 conn(0x7f40301375a0 legacy=0x7f403013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 1 -- 172.21.15.171:0/1045581444 shutdown_connections 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 1 -- 172.21.15.171:0/1045581444 >> 172.21.15.171:0/1045581444 conn(0x7f40300a9ae0 msgr2=0x7f40300a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 1 -- 172.21.15.171:0/1045581444 shutdown_connections 2024-09-16T04:08:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.716+0000 7f40370d7640 1 -- 172.21.15.171:0/1045581444 wait complete. 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.717+0000 7f40370d7640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.717+0000 7f40370d7640 10 monclient: monmap: 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:07.733 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.717+0000 7f40370d7640 1 Processor -- start 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.717+0000 7f40370d7640 1 -- start start 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.717+0000 7f40370d7640 10 monclient: init 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient(hunting): picked mon.noname-a con 0x7f4030136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient(hunting): picked mon.noname-c con 0x7f4030137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient(hunting): picked mon.noname-b con 0x7f40301375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4030131d20 con 0x7f4030136d60 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4030142c70 con 0x7f40301375a0 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4030131760 con 0x7f4030137140 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient(hunting): authenticate will time out at 863.223185s 2024-09-16T04:08:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f4034e73640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4030136d60 legacy=0x7f403012e220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f4035674640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f40301375a0 0x7f40301d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52192/0 (socket says 172.21.15.171:52192) 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f4035674640 1 -- 172.21.15.171:0/2880067974 learned_addr learned my addr 172.21.15.171:0/2880067974 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835824148 0 0) 0x7f4030142c70 con 0x7f40301375a0 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): my global_id is 4621 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f401c003520 con 0x7f40301375a0 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700936302 0 0) 0x7f4030131760 con 0x7f4030137140 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): my global_id is 4649 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4030142c70 con 0x7f4030137140 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1389842221 0 0) 0x7f401c003520 con 0x7f40301375a0 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4030131760 con 0x7f40301375a0 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40200015d0 con 0x7f40301375a0 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): dump: 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2924035675 0 0) 0x7f4030142c70 con 0x7f4030137140 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f401c003520 con 0x7f4030137140 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40240018b0 con 0x7f4030137140 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient(hunting): dump: 2024-09-16T04:08:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4027611580 0 0) 0x7f4030131760 con 0x7f40301375a0 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 >> v1:172.21.15.148:6790/0 conn(0x7f4030137140 legacy=0x7f403012e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 >> v1:172.21.15.148:6789/0 conn(0x7f4030136d60 legacy=0x7f403012e220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 monclient: found mon.b 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4030131a40 con 0x7f40301375a0 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f402d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 5 monclient: authenticate success, global_id 4621 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient: _renew_subs 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f403012fe90 con 0x7f40301375a0 2024-09-16T04:08:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient: _renew_subs 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.718+0000 7f40370d7640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f403012f220 con 0x7f40301375a0 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4020001770 con 0x7f40301375a0 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402effd640 4 set_mon_vals no callback set 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4020004c30 con 0x7f40301375a0 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 10 monclient: dump: 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4020004e50 con 0x7f40301375a0 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.719+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7f40200470a0 con 0x7f40301375a0 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.720+0000 7f4034e73640 10 monclient: get_auth_request con 0x7f401c038e70 auth_method 0 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.720+0000 7f40370d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.720+0000 7f40370d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.720+0000 7f40370d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.720+0000 7f40370d7640 1 -- 172.21.15.171:0/2880067974 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4010005080 con 0x7f40301375a0 2024-09-16T04:08:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.724+0000 7f402d7fa640 1 -- 172.21.15.171:0/2880067974 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f40200198c0 con 0x7f40301375a0 2024-09-16T04:08:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.724+0000 7f402d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.724+0000 7f402d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:07.890 INFO:tasks.workunit.client.0.smithi171.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-16T04:08:07.890 INFO:tasks.workunit.client.0.smithi171.stderr:Invalid command: unused arguments: ['--force'] 2024-09-16T04:08:07.890 INFO:tasks.workunit.client.0.smithi171.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T04:08:07.891 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: invalid command 2024-09-16T04:08:07.892 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:07.892 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T04:08:07.893 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:07.893 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-16T04:08:07.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.971+0000 7fd4a4786640 10 monclient: get_monmap_and_config 2024-09-16T04:08:07.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.971+0000 7fd4a4786640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.971+0000 7fd4a4786640 10 monclient: monmap: 2024-09-16T04:08:07.973 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.973 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.973 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:07.972432+0000 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:07.972432+0000 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:07.974 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 1 Processor -- start 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 1 -- start start 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient: init 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c09ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient(hunting): picked mon.noname-a con 0x7fd49c0a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c0a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c0a0f20 con 0x7fd49c0a0370 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c0a1200 con 0x7fd49c09ff90 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c09bca0 con 0x7fd49c0a0800 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.972+0000 7fd4a4786640 10 monclient(hunting): authenticate will time out at 863.477921s 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a2522640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd49c09ff90 0x7fd49c09f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52200/0 (socket says 172.21.15.171:52200) 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a2522640 1 -- 172.21.15.171:0/1258584130 learned_addr learned my addr 172.21.15.171:0/1258584130 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1d21640 1 -- 172.21.15.171:0/1258584130 >> v1:172.21.15.148:6789/0 conn(0x7fd49c0a0370 legacy=0x7fd49c139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182338478 0 0) 0x7fd49c0a1200 con 0x7fd49c09ff90 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): my global_id is 4624 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd488003520 con 0x7fd49c09ff90 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2202236207 0 0) 0x7fd49c09bca0 con 0x7fd49c0a0800 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient(hunting): my global_id is 4652 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd49c0a1200 con 0x7fd49c0a0800 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 413590362 0 0) 0x7fd488003520 con 0x7fd49c09ff90 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 >> v1:172.21.15.148:6790/0 conn(0x7fd49c0a0800 legacy=0x7fd49c13d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 >> v1:172.21.15.148:6789/0 conn(0x7fd49c0a0370 legacy=0x7fd49c139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 monclient: found mon.noname-c 2024-09-16T04:08:07.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c09e830 con 0x7fd49c09ff90 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd498001610 con 0x7fd49c09ff90 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a4786640 5 monclient: authenticate success, global_id 4624 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a4786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: dump: 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.984 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a4786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd498001930 con 0x7fd49c09ff90 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a4786640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:07.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 1 -- 172.21.15.171:0/1258584130 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd498001d00 con 0x7fd49c09ff90 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a4786640 4 set_mon_vals no callback set 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.973+0000 7fd4a1520640 10 monclient: dump: 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.986 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 10 monclient: shutdown 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 1 -- 172.21.15.171:0/1258584130 >> v1:172.21.15.171:6789/0 conn(0x7fd49c09ff90 legacy=0x7fd49c09f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 1 -- 172.21.15.171:0/1258584130 shutdown_connections 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 1 -- 172.21.15.171:0/1258584130 >> 172.21.15.171:0/1258584130 conn(0x7fd49c0a5450 msgr2=0x7fd49c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 1 -- 172.21.15.171:0/1258584130 shutdown_connections 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 1 -- 172.21.15.171:0/1258584130 wait complete. 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 10 monclient: build_initial_monmap 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.974+0000 7fd4a4786640 10 monclient: monmap: 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:07.988 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.988 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 1 Processor -- start 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 1 -- start start 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient: init 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:07.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c09ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient(hunting): picked mon.noname-a con 0x7fd49c0a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c0a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c134630 con 0x7fd49c0a0370 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c13e540 con 0x7fd49c0a0800 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c13b990 con 0x7fd49c09ff90 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a4786640 10 monclient(hunting): authenticate will time out at 863.480578s 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a1d21640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd49c0a0370 legacy=0x7fd49c13b280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:07.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a2d23640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd49c0a0800 0x7fd49c1d1940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52210/0 (socket says 172.21.15.171:52210) 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd4a2d23640 1 -- 172.21.15.171:0/1030214647 learned_addr learned my addr 172.21.15.171:0/1030214647 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3965219836 0 0) 0x7fd49c13e540 con 0x7fd49c0a0800 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): my global_id is 4627 2024-09-16T04:08:07.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd480003520 con 0x7fd49c0a0800 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815888497 0 0) 0x7fd49c13b990 con 0x7fd49c09ff90 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 10 monclient(hunting): my global_id is 4655 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.975+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd49c13e540 con 0x7fd49c09ff90 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 338485856 0 0) 0x7fd480003520 con 0x7fd49c0a0800 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd49c13b990 con 0x7fd49c0a0800 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd494001c50 con 0x7fd49c0a0800 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:07.992 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.993 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.993 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.993 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.993 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.993 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1711984187 0 0) 0x7fd49c13b990 con 0x7fd49c0a0800 2024-09-16T04:08:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 >> v1:172.21.15.148:6790/0 conn(0x7fd49c09ff90 legacy=0x7fd49c13ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 >> v1:172.21.15.148:6789/0 conn(0x7fd49c0a0370 legacy=0x7fd49c13b280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 monclient: found mon.b 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c11b6f0 con 0x7fd49c0a0800 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 5 monclient: authenticate success, global_id 4627 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 10 monclient: _renew_subs 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd49c13d0f0 con 0x7fd49c0a0800 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 10 monclient: _renew_subs 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a4786640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd49c134230 con 0x7fd49c0a0800 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd494002020 con 0x7fd49c0a0800 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd4a0d1f640 4 set_mon_vals no callback set 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd494003de0 con 0x7fd49c0a0800 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.976+0000 7fd492ffd640 10 monclient: dump: 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd494004000 con 0x7fd49c0a0800 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd4a4786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd4a4786640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd4a4786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd4a4786640 1 -- 172.21.15.171:0/1030214647 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd484005080 con 0x7fd49c0a0800 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7fd494047470 con 0x7fd49c0a0800 2024-09-16T04:08:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.977+0000 7fd4a2522640 10 monclient: get_auth_request con 0x7fd480038bc0 auth_method 0 2024-09-16T04:08:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.981+0000 7fd492ffd640 1 -- 172.21.15.171:0/1030214647 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd494016060 con 0x7fd49c0a0800 2024-09-16T04:08:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.981+0000 7fd492ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:07.981+0000 7fd492ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:08.151 INFO:tasks.workunit.client.0.smithi171.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-16T04:08:08.151 INFO:tasks.workunit.client.0.smithi171.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-16T04:08:08.151 INFO:tasks.workunit.client.0.smithi171.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T04:08:08.151 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: invalid command 2024-09-16T04:08:08.153 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:08.153 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T04:08:08.153 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:08.153 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.237+0000 7f59aba82640 10 monclient: get_monmap_and_config 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.237+0000 7f59aba82640 10 monclient: build_initial_monmap 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.237+0000 7f59aba82640 10 monclient: monmap: 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:08.238453+0000 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:08.238453+0000 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.240 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:08.241 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:08.241 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:08.241 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 1 Processor -- start 2024-09-16T04:08:08.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 1 -- start start 2024-09-16T04:08:08.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 10 monclient: init 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 10 monclient(hunting): picked mon.noname-b con 0x7f59a4136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 10 monclient(hunting): picked mon.noname-c con 0x7f59a4137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.238+0000 7f59aba82640 10 monclient(hunting): picked mon.noname-a con 0x7f59a41375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a409b2c0 con 0x7f59a41375a0 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a409b5a0 con 0x7f59a4137140 2024-09-16T04:08:08.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a40a6640 con 0x7f59a4136d60 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 10 monclient(hunting): authenticate will time out at 863.744070s 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a901d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f59a4137140 0x7f59a413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52212/0 (socket says 172.21.15.171:52212) 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a901d640 1 -- 172.21.15.171:0/2937012281 learned_addr learned my addr 172.21.15.171:0/2937012281 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aa01f640 1 -- 172.21.15.171:0/2937012281 >> v1:172.21.15.148:6789/0 conn(0x7f59a41375a0 legacy=0x7f59a413f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288117169 0 0) 0x7f59a409b5a0 con 0x7f59a4137140 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): my global_id is 4630 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5990003520 con 0x7f59a4137140 2024-09-16T04:08:08.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1678523828 0 0) 0x7f59a40a6640 con 0x7f59a4136d60 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient(hunting): my global_id is 4658 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a409b5a0 con 0x7f59a4136d60 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3962607753 0 0) 0x7f5990003520 con 0x7f59a4137140 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 >> v1:172.21.15.148:6790/0 conn(0x7f59a4136d60 legacy=0x7f59a409bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 >> v1:172.21.15.148:6789/0 conn(0x7f59a41375a0 legacy=0x7f59a413f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: _finish_hunting 0 2024-09-16T04:08:08.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 monclient: found mon.noname-c 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a4143b20 con 0x7f59a4137140 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: _finish_auth 0 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: _check_auth_tickets 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5994001610 con 0x7f59a4137140 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 5 monclient: authenticate success, global_id 4630 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:08.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: dump: 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59940018c0 con 0x7f59a4137140 2024-09-16T04:08:08.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 1 -- 172.21.15.171:0/2937012281 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5994001c90 con 0x7f59a4137140 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59aba82640 4 set_mon_vals no callback set 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.239+0000 7f59a881c640 10 monclient: dump: 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.253 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 10 monclient: shutdown 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 1 -- 172.21.15.171:0/2937012281 >> v1:172.21.15.171:6789/0 conn(0x7f59a4137140 legacy=0x7f59a413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 1 -- 172.21.15.171:0/2937012281 shutdown_connections 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 1 -- 172.21.15.171:0/2937012281 >> 172.21.15.171:0/2937012281 conn(0x7f59a40a9ae0 msgr2=0x7f59a40a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 1 -- 172.21.15.171:0/2937012281 shutdown_connections 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 1 -- 172.21.15.171:0/2937012281 wait complete. 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 10 monclient: build_initial_monmap 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.240+0000 7f59aba82640 10 monclient: monmap: 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:08.254 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 1 Processor -- start 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 1 -- start start 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient: init 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:08.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient(hunting): picked mon.noname-b con 0x7f59a4136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient(hunting): picked mon.noname-c con 0x7f59a4137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient(hunting): picked mon.noname-a con 0x7f59a41375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a40a88f0 con 0x7f59a41375a0 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a40a83a0 con 0x7f59a4136d60 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a412ea40 con 0x7f59a4137140 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59a981e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f59a4136d60 0x7f59a41cbca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52222/0 (socket says 172.21.15.171:52222) 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aba82640 10 monclient(hunting): authenticate will time out at 863.746720s 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59a981e640 1 -- 172.21.15.171:0/1211429719 learned_addr learned my addr 172.21.15.171:0/1211429719 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f59aa01f640 1 -- 172.21.15.171:0/1211429719 >> v1:172.21.15.148:6789/0 conn(0x7f59a41375a0 legacy=0x7f59a41d4410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:08.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217250932 0 0) 0x7f59a40a83a0 con 0x7f59a4136d60 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f599a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f599a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f599a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f599a7fc640 10 monclient(hunting): my global_id is 4633 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.241+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5988003520 con 0x7f59a4136d60 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194896601 0 0) 0x7f59a412ea40 con 0x7f59a4137140 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): my global_id is 4661 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a40a83a0 con 0x7f59a4137140 2024-09-16T04:08:08.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2884718093 0 0) 0x7f5988003520 con 0x7f59a4136d60 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59a412ea40 con 0x7f59a4136d60 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59a0001970 con 0x7f59a4136d60 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): dump: 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.258 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3881860215 0 0) 0x7f59a40a83a0 con 0x7f59a4137140 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5988003520 con 0x7f59a4137140 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5994001a70 con 0x7f59a4137140 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient(hunting): dump: 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.259 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1563811639 0 0) 0x7f59a412ea40 con 0x7f59a4136d60 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 >> v1:172.21.15.148:6790/0 conn(0x7f59a4137140 legacy=0x7f59a41d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 >> v1:172.21.15.148:6789/0 conn(0x7f59a41375a0 legacy=0x7f59a41d4410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 monclient: found mon.b 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a412ed20 con 0x7f59a4136d60 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 5 monclient: authenticate success, global_id 4633 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 10 monclient: _renew_subs 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59a412f000 con 0x7f59a4136d60 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 10 monclient: _renew_subs 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f59aba82640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f59a409e570 con 0x7f59a4136d60 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59a0001d20 con 0x7f59a4136d60 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:08.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59a0003a60 con 0x7f59a4136d60 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599bfff640 4 set_mon_vals no callback set 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.242+0000 7f599a7fc640 10 monclient: dump: 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.262 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f59a0003c00 con 0x7f59a4136d60 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f59aba82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f59aba82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f59aba82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f59aba82640 1 -- 172.21.15.171:0/1211429719 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f598c005080 con 0x7f59a4136d60 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7f59a0047150 con 0x7f59a4136d60 2024-09-16T04:08:08.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.243+0000 7f59a901d640 10 monclient: get_auth_request con 0x7f5988038ef0 auth_method 0 2024-09-16T04:08:08.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.247+0000 7f599a7fc640 1 -- 172.21.15.171:0/1211429719 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f59a00198b0 con 0x7f59a4136d60 2024-09-16T04:08:08.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.247+0000 7f599a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:08.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.247+0000 7f599a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:08.420 INFO:tasks.workunit.client.0.smithi171.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-16T04:08:08.421 INFO:tasks.workunit.client.0.smithi171.stderr:Invalid command: unused arguments: ['foo'] 2024-09-16T04:08:08.421 INFO:tasks.workunit.client.0.smithi171.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-16T04:08:08.421 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: invalid command 2024-09-16T04:08:08.423 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:08.423 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-16T04:08:08.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.502+0000 7f0c9dbb7640 10 monclient: get_monmap_and_config 2024-09-16T04:08:08.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.502+0000 7f0c9dbb7640 10 monclient: build_initial_monmap 2024-09-16T04:08:08.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.502+0000 7f0c9dbb7640 10 monclient: monmap: 2024-09-16T04:08:08.504 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:08.504 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.504 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:08.502849+0000 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:08.502849+0000 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:08.505 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.502+0000 7f0c9dbb7640 1 Processor -- start 2024-09-16T04:08:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.502+0000 7f0c9dbb7640 1 -- start start 2024-09-16T04:08:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.502+0000 7f0c9dbb7640 10 monclient: init 2024-09-16T04:08:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient(hunting): picked mon.noname-b con 0x7f0c980a39e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient(hunting): picked mon.noname-a con 0x7f0c980a44d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient(hunting): picked mon.noname-c con 0x7f0c980a1c90 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c980a4f90 con 0x7f0c980a44d0 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c980a0a30 con 0x7f0c980a1c90 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c980a0ff0 con 0x7f0c980a39e0 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 10 monclient(hunting): authenticate will time out at 864.008351s 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c96ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f0c980a44d0 legacy=0x7f0c9813bd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c97fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0c980a1c90 0x7f0c9813f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52226/0 (socket says 172.21.15.171:52226) 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c97fff640 1 -- 172.21.15.171:0/1027054433 learned_addr learned my addr 172.21.15.171:0/1027054433 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677160237 0 0) 0x7f0c980a0a30 con 0x7f0c980a1c90 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): my global_id is 4636 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c80003520 con 0x7f0c980a1c90 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802759094 0 0) 0x7f0c980a0ff0 con 0x7f0c980a39e0 2024-09-16T04:08:08.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient(hunting): my global_id is 4664 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c980a0a30 con 0x7f0c980a39e0 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3868881455 0 0) 0x7f0c80003520 con 0x7f0c980a1c90 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 >> v1:172.21.15.148:6790/0 conn(0x7f0c980a39e0 legacy=0x7f0c980a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 >> v1:172.21.15.148:6789/0 conn(0x7f0c980a44d0 legacy=0x7f0c9813bd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c98073820 con 0x7f0c980a1c90 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 5 monclient: authenticate success, global_id 4636 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0c90001990 con 0x7f0c980a1c90 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.503+0000 7f0c9dbb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: dump: 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c90001cb0 con 0x7f0c980a1c90 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 1 -- 172.21.15.171:0/1027054433 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0c90002080 con 0x7f0c980a1c90 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 4 set_mon_vals no callback set 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c967fc640 10 monclient: dump: 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.519 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 10 monclient: shutdown 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/1027054433 >> v1:172.21.15.171:6789/0 conn(0x7f0c980a1c90 legacy=0x7f0c9813f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/1027054433 shutdown_connections 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/1027054433 >> 172.21.15.171:0/1027054433 conn(0x7f0c980a9860 msgr2=0x7f0c980a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/1027054433 shutdown_connections 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.504+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/1027054433 wait complete. 2024-09-16T04:08:08.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.505+0000 7f0c9dbb7640 10 monclient: build_initial_monmap 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.505+0000 7f0c9dbb7640 10 monclient: monmap: 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:08.521 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.505+0000 7f0c9dbb7640 1 Processor -- start 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.505+0000 7f0c9dbb7640 1 -- start start 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.505+0000 7f0c9dbb7640 10 monclient: init 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient(hunting): picked mon.noname-c con 0x7f0c980a1c90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient(hunting): picked mon.noname-b con 0x7f0c980a39e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient(hunting): picked mon.noname-a con 0x7f0c980a44d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c98133f70 con 0x7f0c980a44d0 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c981354f0 con 0x7f0c980a39e0 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c981357d0 con 0x7f0c980a1c90 2024-09-16T04:08:08.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c97fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f0c980a44d0 legacy=0x7f0c981d3c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c96ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0c980a39e0 0x7f0c98132700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52242/0 (socket says 172.21.15.171:52242) 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c96ffd640 1 -- 172.21.15.171:0/2631033052 learned_addr learned my addr 172.21.15.171:0/2631033052 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient(hunting): authenticate will time out at 864.011312s 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036997004 0 0) 0x7f0c981354f0 con 0x7f0c980a39e0 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): my global_id is 4639 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c84003520 con 0x7f0c980a39e0 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330016307 0 0) 0x7f0c981357d0 con 0x7f0c980a1c90 2024-09-16T04:08:08.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): my global_id is 4667 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c981354f0 con 0x7f0c980a1c90 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1033044276 0 0) 0x7f0c84003520 con 0x7f0c980a39e0 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c981357d0 con 0x7f0c980a39e0 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0c8c004180 con 0x7f0c980a39e0 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): dump: 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.524 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3887500105 0 0) 0x7f0c981354f0 con 0x7f0c980a1c90 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c84003520 con 0x7f0c980a1c90 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0c88001870 con 0x7f0c980a1c90 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient(hunting): dump: 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2787793033 0 0) 0x7f0c981357d0 con 0x7f0c980a39e0 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 >> v1:172.21.15.148:6790/0 conn(0x7f0c980a1c90 legacy=0x7f0c98131ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:08.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 >> v1:172.21.15.148:6789/0 conn(0x7f0c980a44d0 legacy=0x7f0c981d3c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 monclient: found mon.b 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c98132e10 con 0x7f0c980a39e0 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 5 monclient: authenticate success, global_id 4639 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient: _renew_subs 2024-09-16T04:08:08.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.506+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c98135d90 con 0x7f0c980a39e0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c9dbb7640 10 monclient: _renew_subs 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c9dbb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c980a83f0 con 0x7f0c980a39e0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0c8c0043e0 con 0x7f0c980a39e0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0c8c004730 con 0x7f0c980a39e0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c95ffb640 4 set_mon_vals no callback set 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 10 monclient: dump: 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:08.528 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0c8c0048d0 con 0x7f0c980a39e0 2024-09-16T04:08:08.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.507+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(39..39 src has 1..39) ==== 4634+0+0 (unknown 1674594781 0 0) 0x7f0c8c047100 con 0x7f0c980a39e0 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.508+0000 7f0c977fe640 10 monclient: get_auth_request con 0x7f0c84038f40 auth_method 0 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.508+0000 7f0c9dbb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.508+0000 7f0c9dbb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.508+0000 7f0c9dbb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.508+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c78005080 con 0x7f0c980a39e0 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.512+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0c8c015b90 con 0x7f0c980a39e0 2024-09-16T04:08:08.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.512+0000 7f0c7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:08.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.512+0000 7f0c7ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:08.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.678+0000 7f0c9dbb7640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:08.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.678+0000 7f0c9dbb7640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:08.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.678+0000 7f0c9dbb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:08.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:08.678+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0c78005370 con 0x7f0c980a39e0 2024-09-16T04:08:09.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.168+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v40) ==== 155+0+0 (unknown 672496452 0 0) 0x7f0c8c015490 con 0x7f0c980a39e0 2024-09-16T04:08:09.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.168+0000 7f0c7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:09.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.168+0000 7f0c7ffff640 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-16T04:08:09.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c9dbb7640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c9dbb7640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c9dbb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0c780008d0 con 0x7f0c980a39e0 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c7ffff640 1 -- 172.21.15.171:0/2631033052 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v40) ==== 162+0+0 (unknown 3953006692 0 0) 0x7f0c8c0197e0 con 0x7f0c980a39e0 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c7ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.282+0000 7f0c7ffff640 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-16T04:08:09.285 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'foooo' does not exist 2024-09-16T04:08:09.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 >> v1:172.21.15.171:6812/7327273 conn(0x7f0c84038f40 legacy=0x7f0c8403b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:09.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 10 monclient: shutdown 2024-09-16T04:08:09.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:09.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 >> v1:172.21.15.171:6789/0 conn(0x7f0c980a39e0 legacy=0x7f0c98132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:09.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 shutdown_connections 2024-09-16T04:08:09.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 >> 172.21.15.171:0/2631033052 conn(0x7f0c980a9860 msgr2=0x7f0c98072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:09.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 shutdown_connections 2024-09-16T04:08:09.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.285+0000 7f0c9dbb7640 1 -- 172.21.15.171:0/2631033052 wait complete. 2024-09-16T04:08:09.296 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.374+0000 7f2945db9640 10 monclient: get_monmap_and_config 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.374+0000 7f2945db9640 10 monclient: build_initial_monmap 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.374+0000 7f2945db9640 10 monclient: monmap: 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:09.375240+0000 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:09.375240+0000 2024-09-16T04:08:09.376 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:09.377 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:09.377 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:09.377 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:09.377 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:09.377 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 1 Processor -- start 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 1 -- start start 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient: init 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient(hunting): picked mon.noname-b con 0x7f29401338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient(hunting): picked mon.noname-a con 0x7f2940133cc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient(hunting): picked mon.noname-c con 0x7f29401340d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294009b420 con 0x7f2940133cc0 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29400753d0 con 0x7f29401340d0 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294009b140 con 0x7f29401338e0 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f2945db9640 10 monclient(hunting): authenticate will time out at 864.880723s 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f293effd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2940133cc0 legacy=0x7f294013baf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f293ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f29401340d0 0x7f294013f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52256/0 (socket says 172.21.15.171:52256) 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.375+0000 7f293ffff640 1 -- 172.21.15.171:0/2200983344 learned_addr learned my addr 172.21.15.171:0/2200983344 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 181094333 0 0) 0x7f29400753d0 con 0x7f29401340d0 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:09.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient(hunting): my global_id is 4642 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2928003520 con 0x7f29401340d0 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1458536975 0 0) 0x7f2928003520 con 0x7f29401340d0 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 >> v1:172.21.15.148:6790/0 conn(0x7f29401338e0 legacy=0x7f294009be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 >> v1:172.21.15.148:6789/0 conn(0x7f2940133cc0 legacy=0x7f294013baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294009b700 con 0x7f29401340d0 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 5 monclient: authenticate success, global_id 4642 2024-09-16T04:08:09.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2938001990 con 0x7f29401340d0 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: dump: 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:09.387 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2938001c80 con 0x7f29401340d0 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 4 set_mon_vals no callback set 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 1 -- 172.21.15.171:0/2200983344 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2938002050 con 0x7f29401340d0 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f293e7fc640 10 monclient: dump: 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:09.388 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 10 monclient: shutdown 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 1 -- 172.21.15.171:0/2200983344 >> v1:172.21.15.171:6789/0 conn(0x7f29401340d0 legacy=0x7f294013f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 1 -- 172.21.15.171:0/2200983344 shutdown_connections 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 1 -- 172.21.15.171:0/2200983344 >> 172.21.15.171:0/2200983344 conn(0x7f29400a6460 msgr2=0x7f29400a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:09.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.376+0000 7f2945db9640 1 -- 172.21.15.171:0/2200983344 shutdown_connections 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.377+0000 7f2945db9640 1 -- 172.21.15.171:0/2200983344 wait complete. 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.377+0000 7f2945db9640 10 monclient: build_initial_monmap 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.377+0000 7f2945db9640 10 monclient: monmap: 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:09.390 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.377+0000 7f2945db9640 1 Processor -- start 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 1 -- start start 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient: init 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient(hunting): picked mon.noname-a con 0x7f29401338e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient(hunting): picked mon.noname-c con 0x7f2940133cc0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient(hunting): picked mon.noname-b con 0x7f29401340d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2940131920 con 0x7f29401338e0 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2940131dc0 con 0x7f29401340d0 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29401345a0 con 0x7f2940133cc0 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:09.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2945db9640 10 monclient(hunting): authenticate will time out at 864.883453s 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f293f7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f29401338e0 legacy=0x7f294012ff40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f293ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f29401340d0 0x7f29401d3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52260/0 (socket says 172.21.15.171:52260) 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f293ffff640 1 -- 172.21.15.171:0/419759013 learned_addr learned my addr 172.21.15.171:0/419759013 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687941382 0 0) 0x7f2940131dc0 con 0x7f29401340d0 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): my global_id is 4645 2024-09-16T04:08:09.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f292c003520 con 0x7f29401340d0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199478127 0 0) 0x7f29401345a0 con 0x7f2940133cc0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): my global_id is 4673 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2940131dc0 con 0x7f2940133cc0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2708333120 0 0) 0x7f292c003520 con 0x7f29401340d0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29401345a0 con 0x7f29401340d0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f29380017b0 con 0x7f29401340d0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): dump: 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:09.393 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.378+0000 7f2927fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:09.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3844377902 0 0) 0x7f29401345a0 con 0x7f29401340d0 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 >> v1:172.21.15.148:6790/0 conn(0x7f2940133cc0 legacy=0x7f2940130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 >> v1:172.21.15.148:6789/0 conn(0x7f29401338e0 legacy=0x7f294012ff40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 monclient: found mon.b 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294009d0c0 con 0x7f29401340d0 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 5 monclient: authenticate success, global_id 4645 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 10 monclient: _renew_subs 2024-09-16T04:08:09.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2940134b20 con 0x7f29401340d0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 10 monclient: _renew_subs 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29401326a0 con 0x7f29401340d0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2938001c80 con 0x7f29401340d0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2938005190 con 0x7f29401340d0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f293dffb640 4 set_mon_vals no callback set 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.379+0000 7f2927fff640 10 monclient: dump: 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:09.396 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f29380105c0 con 0x7f29401340d0 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f2945db9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f2945db9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f2945db9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2920005080 con 0x7f29401340d0 2024-09-16T04:08:09.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(40..40 src has 1..40) ==== 4290+0+0 (unknown 971780243 0 0) 0x7f29380473a0 con 0x7f29401340d0 2024-09-16T04:08:09.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.380+0000 7f293f7fe640 10 monclient: get_auth_request con 0x7f292c038fb0 auth_method 0 2024-09-16T04:08:09.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.384+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f29380160a0 con 0x7f29401340d0 2024-09-16T04:08:09.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.384+0000 7f2927fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:09.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.384+0000 7f2927fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:09.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.566+0000 7f2945db9640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:09.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.566+0000 7f2945db9640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:09.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.566+0000 7f2945db9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:09.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:09.566+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2920005370 con 0x7f29401340d0 2024-09-16T04:08:10.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.172+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v41) ==== 152+0+0 (unknown 3899742253 0 0) 0x7f29380159a0 con 0x7f29401340d0 2024-09-16T04:08:10.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.172+0000 7f2927fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.172+0000 7f2927fff640 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2945db9640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2945db9640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2945db9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2920001140 con 0x7f29401340d0 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2927fff640 1 -- 172.21.15.171:0/419759013 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v41) ==== 159+0+0 (unknown 3902454184 0 0) 0x7f293801e4f0 con 0x7f29401340d0 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2927fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.281+0000 7f2927fff640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T04:08:10.284 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'fooo' does not exist 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 >> v1:172.21.15.171:6812/7327273 conn(0x7f292c038fb0 legacy=0x7f292c03b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 10 monclient: shutdown 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 >> v1:172.21.15.171:6789/0 conn(0x7f29401340d0 legacy=0x7f29401d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 shutdown_connections 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 >> 172.21.15.171:0/419759013 conn(0x7f29400a6460 msgr2=0x7f294012b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:10.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 shutdown_connections 2024-09-16T04:08:10.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.283+0000 7f2945db9640 1 -- 172.21.15.171:0/419759013 wait complete. 2024-09-16T04:08:10.294 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T04:08:10.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.372+0000 7f902c152640 10 monclient: get_monmap_and_config 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.372+0000 7f902c152640 10 monclient: build_initial_monmap 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.372+0000 7f902c152640 10 monclient: monmap: 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:10.373705+0000 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:10.373705+0000 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:10.375 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:10.376 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:10.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.373+0000 7f902c152640 1 Processor -- start 2024-09-16T04:08:10.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.373+0000 7f902c152640 1 -- start start 2024-09-16T04:08:10.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.373+0000 7f902c152640 10 monclient: init 2024-09-16T04:08:10.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:10.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:10.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient(hunting): picked mon.noname-c con 0x7f90240a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:10.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient(hunting): picked mon.noname-b con 0x7f902409f320 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:10.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient(hunting): picked mon.noname-a con 0x7f902413d2c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:10.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90240737b0 con 0x7f902413d2c0 2024-09-16T04:08:10.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f902406c6d0 con 0x7f90240a0ae0 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f902406d980 con 0x7f902409f320 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient(hunting): authenticate will time out at 865.879223s 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9029eee640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f90240a0ae0 0x7f90240a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52268/0 (socket says 172.21.15.171:52268) 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9029eee640 1 -- 172.21.15.171:0/3434107518 learned_addr learned my addr 172.21.15.171:0/3434107518 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902a6ef640 1 -- 172.21.15.171:0/3434107518 >> v1:172.21.15.148:6789/0 conn(0x7f902413d2c0 legacy=0x7f902413f6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192531467 0 0) 0x7f902406c6d0 con 0x7f90240a0ae0 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient(hunting): _negotiate 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient(hunting): my global_id is 4648 2024-09-16T04:08:10.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9010003520 con 0x7f90240a0ae0 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2461522095 0 0) 0x7f9010003520 con 0x7f90240a0ae0 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 >> v1:172.21.15.148:6790/0 conn(0x7f902409f320 legacy=0x7f902413bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 >> v1:172.21.15.148:6789/0 conn(0x7f902413d2c0 legacy=0x7f902413f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: _finish_hunting 0 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 monclient: found mon.noname-c 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902406c950 con 0x7f90240a0ae0 2024-09-16T04:08:10.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: _finish_auth 0 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: _check_auth_tickets 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 5 monclient: authenticate success, global_id 4648 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9014001610 con 0x7f90240a0ae0 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:10.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: dump: 2024-09-16T04:08:10.390 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:10.390 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9014001970 con 0x7f90240a0ae0 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:10.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 1 -- 172.21.15.171:0/3434107518 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f90140048a0 con 0x7f90240a0ae0 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f902c152640 4 set_mon_vals no callback set 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.374+0000 7f9028eec640 10 monclient: dump: 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 10 monclient: shutdown 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 1 -- 172.21.15.171:0/3434107518 >> v1:172.21.15.171:6789/0 conn(0x7f90240a0ae0 legacy=0x7f90240a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 1 -- 172.21.15.171:0/3434107518 shutdown_connections 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 1 -- 172.21.15.171:0/3434107518 >> 172.21.15.171:0/3434107518 conn(0x7f90240a77f0 msgr2=0x7f90240a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 1 -- 172.21.15.171:0/3434107518 shutdown_connections 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.375+0000 7f902c152640 1 -- 172.21.15.171:0/3434107518 wait complete. 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient: build_initial_monmap 2024-09-16T04:08:10.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient: monmap: 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 1 Processor -- start 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 1 -- start start 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient: init 2024-09-16T04:08:10.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient(hunting): picked mon.noname-b con 0x7f902409f320 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient(hunting): picked mon.noname-c con 0x7f90240a0ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient(hunting): picked mon.noname-a con 0x7f902413d2c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9024127160 con 0x7f902413d2c0 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9024127440 con 0x7f902409f320 2024-09-16T04:08:10.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90240a2210 con 0x7f90240a0ae0 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f9029eee640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f902409f320 0x7f90241cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52270/0 (socket says 172.21.15.171:52270) 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f9029eee640 1 -- 172.21.15.171:0/2698734199 learned_addr learned my addr 172.21.15.171:0/2698734199 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.376+0000 7f902c152640 10 monclient(hunting): authenticate will time out at 865.881971s 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902a6ef640 1 -- 172.21.15.171:0/2698734199 >> v1:172.21.15.148:6789/0 conn(0x7f902413d2c0 legacy=0x7f90241d3f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2074214191 0 0) 0x7f9024127440 con 0x7f902409f320 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): my global_id is 4651 2024-09-16T04:08:10.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9008003520 con 0x7f902409f320 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997850703 0 0) 0x7f90240a2210 con 0x7f90240a0ae0 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): my global_id is 4679 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9024127440 con 0x7f90240a0ae0 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2588934477 0 0) 0x7f9008003520 con 0x7f902409f320 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90240a2210 con 0x7f902409f320 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9014001430 con 0x7f902409f320 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:10.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): dump: 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:10.398 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2298665186 0 0) 0x7f90240a2210 con 0x7f902409f320 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 >> v1:172.21.15.148:6790/0 conn(0x7f90240a0ae0 legacy=0x7f90241d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 >> v1:172.21.15.148:6789/0 conn(0x7f902413d2c0 legacy=0x7f90241d3f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 monclient: found mon.b 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90240a1d50 con 0x7f902409f320 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 5 monclient: authenticate success, global_id 4651 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 10 monclient: _renew_subs 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9024126e80 con 0x7f902409f320 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 10 monclient: _renew_subs 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f90241244c0 con 0x7f902409f320 2024-09-16T04:08:10.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f90140016e0 con 0x7f902409f320 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9014005000 con 0x7f902409f320 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901bfff640 4 set_mon_vals no callback set 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.377+0000 7f901a7fc640 10 monclient: dump: 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:10.401 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9014010430 con 0x7f902409f320 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f902c152640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f902c152640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f902c152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f900c005080 con 0x7f902409f320 2024-09-16T04:08:10.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(41..41 src has 1..41) ==== 3940+0+0 (unknown 874842820 0 0) 0x7f9014046f50 con 0x7f902409f320 2024-09-16T04:08:10.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.378+0000 7f90296ed640 10 monclient: get_auth_request con 0x7f9008038fb0 auth_method 0 2024-09-16T04:08:10.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.383+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9014015db0 con 0x7f902409f320 2024-09-16T04:08:10.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.383+0000 7f901a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:10.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.383+0000 7f901a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:10.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.557+0000 7f902c152640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.557+0000 7f902c152640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.557+0000 7f902c152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:10.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.557+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f900c005370 con 0x7f902409f320 2024-09-16T04:08:10.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.976+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v42) ==== 149+0+0 (unknown 3756339892 0 0) 0x7f90140156b0 con 0x7f902409f320 2024-09-16T04:08:10.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.976+0000 7f901a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:10.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:10.976+0000 7f901a7fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-16T04:08:11.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.090+0000 7f902c152640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.090+0000 7f902c152640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.090+0000 7f902c152640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.090+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f900c001140 con 0x7f902409f320 2024-09-16T04:08:11.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.091+0000 7f901a7fc640 1 -- 172.21.15.171:0/2698734199 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v42) ==== 156+0+0 (unknown 426725196 0 0) 0x7f901401e200 con 0x7f902409f320 2024-09-16T04:08:11.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.091+0000 7f901a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.091+0000 7f901a7fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T04:08:11.093 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'foo' does not exist 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 >> v1:172.21.15.171:6812/7327273 conn(0x7f9008038fb0 legacy=0x7f900803b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 10 monclient: shutdown 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 >> v1:172.21.15.171:6789/0 conn(0x7f902409f320 legacy=0x7f90241cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 shutdown_connections 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 >> 172.21.15.171:0/2698734199 conn(0x7f90240a77f0 msgr2=0x7f902409ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 shutdown_connections 2024-09-16T04:08:11.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.093+0000 7f902c152640 1 -- 172.21.15.171:0/2698734199 wait complete. 2024-09-16T04:08:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-16T04:08:11.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.184+0000 7f320baa6640 10 monclient: get_monmap_and_config 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.184+0000 7f320baa6640 10 monclient: build_initial_monmap 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.184+0000 7f320baa6640 10 monclient: monmap: 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:11.184844+0000 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:11.184844+0000 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:11.186 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.184+0000 7f320baa6640 1 Processor -- start 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.184+0000 7f320baa6640 1 -- start start 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.184+0000 7f320baa6640 10 monclient: init 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient(hunting): picked mon.noname-a con 0x7f32040a39e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient(hunting): picked mon.noname-c con 0x7f32040a44d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:11.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient(hunting): picked mon.noname-b con 0x7f32040a1c90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32040a4f90 con 0x7f32040a39e0 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32040a0a30 con 0x7f32040a44d0 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32040a0ff0 con 0x7f32040a1c90 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 10 monclient(hunting): authenticate will time out at 866.690342s 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3209842640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f32040a39e0 legacy=0x7f32040a3dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3209041640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f32040a44d0 0x7f320413bd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52284/0 (socket says 172.21.15.171:52284) 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3209041640 1 -- 172.21.15.171:0/1860864416 learned_addr learned my addr 172.21.15.171:0/1860864416 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907275178 0 0) 0x7f32040a0a30 con 0x7f32040a44d0 2024-09-16T04:08:11.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): my global_id is 4654 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31ec003520 con 0x7f32040a44d0 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578699996 0 0) 0x7f32040a0ff0 con 0x7f32040a1c90 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient(hunting): my global_id is 4682 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32040a0a30 con 0x7f32040a1c90 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2717452282 0 0) 0x7f31ec003520 con 0x7f32040a44d0 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 >> v1:172.21.15.148:6790/0 conn(0x7f32040a1c90 legacy=0x7f320413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 >> v1:172.21.15.148:6789/0 conn(0x7f32040a39e0 legacy=0x7f32040a3dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: _finish_hunting 0 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 monclient: found mon.noname-c 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3204073820 con 0x7f32040a44d0 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: _finish_auth 0 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: _check_auth_tickets 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 5 monclient: authenticate success, global_id 4654 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31f8001610 con 0x7f32040a44d0 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:11.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f3208840640 10 monclient: dump: 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.185+0000 7f320baa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31f8001930 con 0x7f32040a44d0 2024-09-16T04:08:11.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f3208840640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f3208840640 1 -- 172.21.15.171:0/1860864416 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31f8001d00 con 0x7f32040a44d0 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f3208840640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f3208840640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f3208840640 10 monclient: dump: 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.199 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 4 set_mon_vals no callback set 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 10 monclient: shutdown 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 1 -- 172.21.15.171:0/1860864416 >> v1:172.21.15.171:6789/0 conn(0x7f32040a44d0 legacy=0x7f320413bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 1 -- 172.21.15.171:0/1860864416 shutdown_connections 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 1 -- 172.21.15.171:0/1860864416 >> 172.21.15.171:0/1860864416 conn(0x7f32040a9860 msgr2=0x7f32040a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 1 -- 172.21.15.171:0/1860864416 shutdown_connections 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.186+0000 7f320baa6640 1 -- 172.21.15.171:0/1860864416 wait complete. 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient: build_initial_monmap 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient: monmap: 2024-09-16T04:08:11.200 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:11.201 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 1 Processor -- start 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 1 -- start start 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient: init 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient(hunting): picked mon.noname-a con 0x7f32040a1c90 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient(hunting): picked mon.noname-c con 0x7f32040a39e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient(hunting): picked mon.noname-b con 0x7f32040a44d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3204133600 con 0x7f32040a1c90 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32041355b0 con 0x7f32040a44d0 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3204135890 con 0x7f32040a39e0 2024-09-16T04:08:11.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.187+0000 7f320baa6640 10 monclient(hunting): authenticate will time out at 866.692981s 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f3209842640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f32040a1c90 legacy=0x7f3204132110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320a043640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f32040a44d0 0x7f32041d3c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52290/0 (socket says 172.21.15.171:52290) 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320a043640 1 -- 172.21.15.171:0/122189941 learned_addr learned my addr 172.21.15.171:0/122189941 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2408212363 0 0) 0x7f32041355b0 con 0x7f32040a44d0 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): my global_id is 4657 2024-09-16T04:08:11.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31f0003520 con 0x7f32040a44d0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224920996 0 0) 0x7f3204135890 con 0x7f32040a39e0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): my global_id is 4685 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32041355b0 con 0x7f32040a39e0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1508464648 0 0) 0x7f31f0003520 con 0x7f32040a44d0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3204135890 con 0x7f32040a44d0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31f4001b90 con 0x7f32040a44d0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:11.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): dump: 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 750287133 0 0) 0x7f32041355b0 con 0x7f32040a39e0 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31f0003520 con 0x7f32040a39e0 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31f8001430 con 0x7f32040a39e0 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient(hunting): dump: 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 480098001 0 0) 0x7f3204135890 con 0x7f32040a44d0 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 >> v1:172.21.15.148:6790/0 conn(0x7f32040a39e0 legacy=0x7f3204132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 >> v1:172.21.15.148:6789/0 conn(0x7f32040a1c90 legacy=0x7f3204132110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 monclient: found mon.b 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3204132f30 con 0x7f32040a44d0 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:11.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 5 monclient: authenticate success, global_id 4657 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 10 monclient: _renew_subs 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3204135e50 con 0x7f32040a44d0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 10 monclient: _renew_subs 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f32040a82a0 con 0x7f32040a44d0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31f4003990 con 0x7f32040a44d0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31f4003d60 con 0x7f32040a44d0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:11.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f3203fff640 4 set_mon_vals no callback set 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.188+0000 7f32027fc640 10 monclient: dump: 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.209 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f31f4003f00 con 0x7f32040a44d0 2024-09-16T04:08:11.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f320baa6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f320baa6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f320baa6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e4005080 con 0x7f32040a44d0 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7f31f4046ff0 con 0x7f32040a44d0 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.189+0000 7f3209842640 10 monclient: get_auth_request con 0x7f31f0038ea0 auth_method 0 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.194+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f31f4015f70 con 0x7f32040a44d0 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.194+0000 7f32027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.194+0000 7f32027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f320baa6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f320baa6640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f320baa6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f31e4005370 con 0x7f32040a44d0 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v42) ==== 156+0+0 (unknown 426725196 0 0) 0x7f31f4015870 con 0x7f32040a44d0 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f32027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.379+0000 7f32027fc640 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-16T04:08:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f320baa6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f320baa6640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f320baa6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f31e4001140 con 0x7f32040a44d0 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f32027fc640 1 -- 172.21.15.171:0/122189941 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v42) ==== 156+0+0 (unknown 426725196 0 0) 0x7f31f401e420 con 0x7f32040a44d0 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f32027fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.477+0000 7f32027fc640 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-16T04:08:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'foo' does not exist 2024-09-16T04:08:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 >> v1:172.21.15.171:6812/7327273 conn(0x7f31f0038ea0 legacy=0x7f31f003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 10 monclient: shutdown 2024-09-16T04:08:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 >> v1:172.21.15.171:6789/0 conn(0x7f32040a44d0 legacy=0x7f32041d3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 shutdown_connections 2024-09-16T04:08:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 >> 172.21.15.171:0/122189941 conn(0x7f32040a9860 msgr2=0x7f3204072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 shutdown_connections 2024-09-16T04:08:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.479+0000 7f320baa6640 1 -- 172.21.15.171:0/122189941 wait complete. 2024-09-16T04:08:11.490 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T04:08:11.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.569+0000 7f5a3f186640 10 monclient: get_monmap_and_config 2024-09-16T04:08:11.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.569+0000 7f5a3f186640 10 monclient: build_initial_monmap 2024-09-16T04:08:11.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.569+0000 7f5a3f186640 10 monclient: monmap: 2024-09-16T04:08:11.571 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:11.570454+0000 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:11.570454+0000 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:11.572 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 1 Processor -- start 2024-09-16T04:08:11.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 1 -- start start 2024-09-16T04:08:11.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 10 monclient: init 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 10 monclient(hunting): picked mon.noname-b con 0x7f5a381338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 10 monclient(hunting): picked mon.noname-a con 0x7f5a38133cc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 10 monclient(hunting): picked mon.noname-c con 0x7f5a381340d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3809b420 con 0x7f5a38133cc0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a380753d0 con 0x7f5a381340d0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.570+0000 7f5a3f186640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3809b140 con 0x7f5a381338e0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 10 monclient(hunting): authenticate will time out at 867.076017s 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2ffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f5a38133cc0 legacy=0x7f5a3813baf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3d723640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5a381340d0 0x7f5a3813f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52306/0 (socket says 172.21.15.171:52306) 2024-09-16T04:08:11.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3d723640 1 -- 172.21.15.171:0/3189815812 learned_addr learned my addr 172.21.15.171:0/3189815812 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543321142 0 0) 0x7f5a380753d0 con 0x7f5a381340d0 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): my global_id is 4660 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a20003520 con 0x7f5a381340d0 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070485962 0 0) 0x7f5a3809b140 con 0x7f5a381338e0 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient(hunting): my global_id is 4688 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a380753d0 con 0x7f5a381338e0 2024-09-16T04:08:11.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1931073744 0 0) 0x7f5a20003520 con 0x7f5a381340d0 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 >> v1:172.21.15.148:6790/0 conn(0x7f5a381338e0 legacy=0x7f5a3809be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 >> v1:172.21.15.148:6789/0 conn(0x7f5a38133cc0 legacy=0x7f5a3813baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3809b700 con 0x7f5a381340d0 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a30001990 con 0x7f5a381340d0 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 5 monclient: authenticate success, global_id 4660 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:11.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: dump: 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a30001c40 con 0x7f5a381340d0 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 1 -- 172.21.15.171:0/3189815812 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a30002010 con 0x7f5a381340d0 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a3f186640 4 set_mon_vals no callback set 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.571+0000 7f5a2f7fe640 10 monclient: dump: 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.584 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 10 monclient: shutdown 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 1 -- 172.21.15.171:0/3189815812 >> v1:172.21.15.171:6789/0 conn(0x7f5a381340d0 legacy=0x7f5a3813f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 1 -- 172.21.15.171:0/3189815812 shutdown_connections 2024-09-16T04:08:11.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 1 -- 172.21.15.171:0/3189815812 >> 172.21.15.171:0/3189815812 conn(0x7f5a380a6460 msgr2=0x7f5a380a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 1 -- 172.21.15.171:0/3189815812 shutdown_connections 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 1 -- 172.21.15.171:0/3189815812 wait complete. 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 10 monclient: build_initial_monmap 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.572+0000 7f5a3f186640 10 monclient: monmap: 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:11.586 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 1 Processor -- start 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 1 -- start start 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient: init 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient(hunting): picked mon.noname-a con 0x7f5a381338e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient(hunting): picked mon.noname-b con 0x7f5a38133cc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:11.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient(hunting): picked mon.noname-c con 0x7f5a381340d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a38131bb0 con 0x7f5a381338e0 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a381346d0 con 0x7f5a38133cc0 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a381349b0 con 0x7f5a381340d0 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3f186640 10 monclient(hunting): authenticate will time out at 867.078720s 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a3cf22640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f5a381338e0 legacy=0x7f5a3812fc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5a38133cc0 0x7f5a38130360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52314/0 (socket says 172.21.15.171:52314) 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2ffff640 1 -- 172.21.15.171:0/1794092898 learned_addr learned my addr 172.21.15.171:0/1794092898 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2759771906 0 0) 0x7f5a381346d0 con 0x7f5a38133cc0 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2d7fa640 10 monclient(hunting): my global_id is 4663 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.573+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a24003550 con 0x7f5a38133cc0 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367226871 0 0) 0x7f5a381349b0 con 0x7f5a381340d0 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): my global_id is 4691 2024-09-16T04:08:11.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a381346d0 con 0x7f5a381340d0 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2005895836 0 0) 0x7f5a24003550 con 0x7f5a38133cc0 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a381349b0 con 0x7f5a38133cc0 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a280041d0 con 0x7f5a38133cc0 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): dump: 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.590 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1511901449 0 0) 0x7f5a381346d0 con 0x7f5a381340d0 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a24003550 con 0x7f5a381340d0 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a30001df0 con 0x7f5a381340d0 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient(hunting): dump: 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.591 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4139572895 0 0) 0x7f5a381349b0 con 0x7f5a38133cc0 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 >> v1:172.21.15.148:6790/0 conn(0x7f5a381340d0 legacy=0x7f5a381d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 >> v1:172.21.15.148:6789/0 conn(0x7f5a381338e0 legacy=0x7f5a3812fc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 monclient: found mon.b 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:11.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a38130a70 con 0x7f5a38133cc0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 5 monclient: authenticate success, global_id 4663 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 10 monclient: _renew_subs 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a380a1a80 con 0x7f5a38133cc0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 10 monclient: _renew_subs 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a38131dc0 con 0x7f5a38133cc0 2024-09-16T04:08:11.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a28004480 con 0x7f5a38133cc0 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a28004850 con 0x7f5a38133cc0 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2effd640 4 set_mon_vals no callback set 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.574+0000 7f5a2d7fa640 10 monclient: dump: 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5a28004ab0 con 0x7f5a38133cc0 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a3f186640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a3f186640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a3f186640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a18005080 con 0x7f5a38133cc0 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7f5a28046e60 con 0x7f5a38133cc0 2024-09-16T04:08:11.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.575+0000 7f5a3cf22640 10 monclient: get_auth_request con 0x7f5a24038ed0 auth_method 0 2024-09-16T04:08:11.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.580+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5a28015d00 con 0x7f5a38133cc0 2024-09-16T04:08:11.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.580+0000 7f5a2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.580+0000 7f5a2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:11.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.776+0000 7f5a3f186640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.776+0000 7f5a3f186640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.776+0000 7f5a3f186640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.776+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5a18005370 con 0x7f5a38133cc0 2024-09-16T04:08:11.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.777+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v42) ==== 159+0+0 (unknown 784008757 0 0) 0x7f5a28015600 con 0x7f5a38133cc0 2024-09-16T04:08:11.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.777+0000 7f5a2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.777+0000 7f5a2d7fa640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T04:08:11.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a3f186640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a3f186640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a3f186640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5a18001140 con 0x7f5a38133cc0 2024-09-16T04:08:11.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a2d7fa640 1 -- 172.21.15.171:0/1794092898 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v42) ==== 159+0+0 (unknown 784008757 0 0) 0x7f5a28019950 con 0x7f5a38133cc0 2024-09-16T04:08:11.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a2d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:11.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.878+0000 7f5a2d7fa640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T04:08:11.881 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'fooo' does not exist 2024-09-16T04:08:11.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.880+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 >> v1:172.21.15.171:6812/7327273 conn(0x7f5a24038ed0 legacy=0x7f5a2403b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.880+0000 7f5a3f186640 10 monclient: shutdown 2024-09-16T04:08:11.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.880+0000 7f5a3f186640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:11.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.880+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 >> v1:172.21.15.171:6789/0 conn(0x7f5a38133cc0 legacy=0x7f5a38130360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.880+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 shutdown_connections 2024-09-16T04:08:11.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.880+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 >> 172.21.15.171:0/1794092898 conn(0x7f5a380a6460 msgr2=0x7f5a380a4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:11.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.881+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 shutdown_connections 2024-09-16T04:08:11.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.881+0000 7f5a3f186640 1 -- 172.21.15.171:0/1794092898 wait complete. 2024-09-16T04:08:11.893 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-16T04:08:11.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.971+0000 7fd3a279f640 10 monclient: get_monmap_and_config 2024-09-16T04:08:11.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.971+0000 7fd3a279f640 10 monclient: build_initial_monmap 2024-09-16T04:08:11.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.971+0000 7fd3a279f640 10 monclient: monmap: 2024-09-16T04:08:11.973 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:11.973 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:11.972196+0000 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:11.972196+0000 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:11.974 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 1 Processor -- start 2024-09-16T04:08:11.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 1 -- start start 2024-09-16T04:08:11.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient: init 2024-09-16T04:08:11.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:11.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:11.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient(hunting): picked mon.noname-c con 0x7fd39c133950 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:11.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient(hunting): picked mon.noname-b con 0x7fd39c137920 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:11.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient(hunting): picked mon.noname-a con 0x7fd39c13b430 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:11.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c076200 con 0x7fd39c13b430 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c0a0260 con 0x7fd39c133950 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c075880 con 0x7fd39c137920 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.972+0000 7fd3a279f640 10 monclient(hunting): authenticate will time out at 867.477971s 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd39c133950 0x7fd39c133d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52320/0 (socket says 172.21.15.171:52320) 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd3a0d3c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd39c13b430 legacy=0x7fd39c13d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39bfff640 1 -- 172.21.15.171:0/1307727108 learned_addr learned my addr 172.21.15.171:0/1307727108 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:11.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732931398 0 0) 0x7fd39c0a0260 con 0x7fd39c133950 2024-09-16T04:08:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): my global_id is 4666 2024-09-16T04:08:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd388003520 con 0x7fd39c133950 2024-09-16T04:08:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827958225 0 0) 0x7fd39c075880 con 0x7fd39c137920 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient(hunting): my global_id is 4694 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd39c0a0260 con 0x7fd39c137920 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2357913878 0 0) 0x7fd388003520 con 0x7fd39c133950 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 >> v1:172.21.15.148:6790/0 conn(0x7fd39c137920 legacy=0x7fd39c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 >> v1:172.21.15.148:6789/0 conn(0x7fd39c13b430 legacy=0x7fd39c13d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c09da90 con 0x7fd39c133950 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd38c001610 con 0x7fd39c133950 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd3a279f640 5 monclient: authenticate success, global_id 4666 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd3a279f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: dump: 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.987 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd3a279f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd38c0018c0 con 0x7fd39c133950 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 1 -- 172.21.15.171:0/1307727108 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd38c001c90 con 0x7fd39c133950 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd39b7fe640 10 monclient: dump: 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.989 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd3a279f640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.973+0000 7fd3a279f640 4 set_mon_vals no callback set 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 10 monclient: shutdown 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 1 -- 172.21.15.171:0/1307727108 >> v1:172.21.15.171:6789/0 conn(0x7fd39c133950 legacy=0x7fd39c133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 1 -- 172.21.15.171:0/1307727108 shutdown_connections 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 1 -- 172.21.15.171:0/1307727108 >> 172.21.15.171:0/1307727108 conn(0x7fd39c0a6460 msgr2=0x7fd39c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 1 -- 172.21.15.171:0/1307727108 shutdown_connections 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 1 -- 172.21.15.171:0/1307727108 wait complete. 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 10 monclient: build_initial_monmap 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.974+0000 7fd3a279f640 10 monclient: monmap: 2024-09-16T04:08:11.990 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 1 Processor -- start 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 1 -- start start 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient: init 2024-09-16T04:08:11.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient(hunting): picked mon.noname-a con 0x7fd39c133950 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient(hunting): picked mon.noname-c con 0x7fd39c137920 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient(hunting): picked mon.noname-b con 0x7fd39c13b430 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c11d220 con 0x7fd39c133950 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c11d7e0 con 0x7fd39c13b430 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c13ad30 con 0x7fd39c137920 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd39bfff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd39c133950 legacy=0x7fd39c1c9790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a279f640 10 monclient(hunting): authenticate will time out at 867.480704s 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a0d3c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd39c13b430 0x7fd39c1d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52330/0 (socket says 172.21.15.171:52330) 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3a0d3c640 1 -- 172.21.15.171:0/1061462008 learned_addr learned my addr 172.21.15.171:0/1061462008 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609854412 0 0) 0x7fd39c11d7e0 con 0x7fd39c13b430 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3997fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3997fa640 10 monclient(hunting): my global_id is 4669 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.975+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd37c003520 con 0x7fd39c13b430 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607443057 0 0) 0x7fd39c13ad30 con 0x7fd39c137920 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): my global_id is 4697 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd39c11d7e0 con 0x7fd39c137920 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2846804429 0 0) 0x7fd37c003520 con 0x7fd39c13b430 2024-09-16T04:08:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd39c13ad30 con 0x7fd39c13b430 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd394004410 con 0x7fd39c13b430 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): dump: 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.994 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3692573633 0 0) 0x7fd39c11d7e0 con 0x7fd39c137920 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd37c003520 con 0x7fd39c137920 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd3840018b0 con 0x7fd39c137920 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient(hunting): dump: 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3015210095 0 0) 0x7fd39c13ad30 con 0x7fd39c13b430 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 >> v1:172.21.15.148:6790/0 conn(0x7fd39c137920 legacy=0x7fd39c1c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 >> v1:172.21.15.148:6789/0 conn(0x7fd39c133950 legacy=0x7fd39c1c9790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 monclient: found mon.b 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c072690 con 0x7fd39c13b430 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 5 monclient: authenticate success, global_id 4669 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 10 monclient: _renew_subs 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd39c072ae0 con 0x7fd39c13b430 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 10 monclient: _renew_subs 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd39c071a00 con 0x7fd39c13b430 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3940046c0 con 0x7fd39c13b430 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd394004a90 con 0x7fd39c13b430 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd39affd640 4 set_mon_vals no callback set 2024-09-16T04:08:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.976+0000 7fd3997fa640 10 monclient: dump: 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd394004cf0 con 0x7fd39c13b430 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd3a279f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd3a279f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd3a279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd380005080 con 0x7fd39c13b430 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7fd394046fa0 con 0x7fd39c13b430 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.977+0000 7fd39bfff640 10 monclient: get_auth_request con 0x7fd37c038ea0 auth_method 0 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.981+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd394015fe0 con 0x7fd39c13b430 2024-09-16T04:08:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.982+0000 7fd3997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:11.982+0000 7fd3997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:12.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.144+0000 7fd3a279f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.144+0000 7fd3a279f640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.144+0000 7fd3a279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.144+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd380005370 con 0x7fd39c13b430 2024-09-16T04:08:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.145+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v42) ==== 159+0+0 (unknown 784008757 0 0) 0x7fd3940158e0 con 0x7fd39c13b430 2024-09-16T04:08:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.145+0000 7fd3997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.145+0000 7fd3997fa640 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-16T04:08:12.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3a279f640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3a279f640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3a279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd380001140 con 0x7fd39c13b430 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3997fa640 1 -- 172.21.15.171:0/1061462008 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v42) ==== 159+0+0 (unknown 784008757 0 0) 0x7fd39401e490 con 0x7fd39c13b430 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3997fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.247+0000 7fd3997fa640 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-16T04:08:12.250 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'fooo' does not exist 2024-09-16T04:08:12.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 >> v1:172.21.15.171:6812/7327273 conn(0x7fd37c038ea0 legacy=0x7fd37c03b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 10 monclient: shutdown 2024-09-16T04:08:12.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:12.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 >> v1:172.21.15.171:6789/0 conn(0x7fd39c13b430 legacy=0x7fd39c1d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 shutdown_connections 2024-09-16T04:08:12.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 >> 172.21.15.171:0/1061462008 conn(0x7fd39c0a6460 msgr2=0x7fd39c13a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:12.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.249+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 shutdown_connections 2024-09-16T04:08:12.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.250+0000 7fd3a279f640 1 -- 172.21.15.171:0/1061462008 wait complete. 2024-09-16T04:08:12.260 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-16T04:08:12.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.340+0000 7f60fb1d6640 10 monclient: get_monmap_and_config 2024-09-16T04:08:12.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.340+0000 7f60fb1d6640 10 monclient: build_initial_monmap 2024-09-16T04:08:12.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.340+0000 7f60fb1d6640 10 monclient: monmap: 2024-09-16T04:08:12.342 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:12.341501+0000 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:12.341501+0000 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:12.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 1 Processor -- start 2024-09-16T04:08:12.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 1 -- start start 2024-09-16T04:08:12.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient: init 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f60f409b920 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f60f4136da0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f60f413d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f4073a20 con 0x7f60f4136da0 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f4099920 con 0x7f60f409b920 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f4099c00 con 0x7f60f413d320 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.341+0000 7f60fb1d6640 10 monclient(hunting): authenticate will time out at 867.846938s 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f8f72640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f60f409b920 0x7f60f409bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52346/0 (socket says 172.21.15.171:52346) 2024-09-16T04:08:12.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f8f72640 1 -- 172.21.15.171:0/3996270510 learned_addr learned my addr 172.21.15.171:0/3996270510 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f3fff640 1 -- 172.21.15.171:0/3996270510 >> v1:172.21.15.148:6789/0 conn(0x7f60f4136da0 legacy=0x7f60f413bc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265505105 0 0) 0x7f60f4099920 con 0x7f60f409b920 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient(hunting): my global_id is 4672 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e4003520 con 0x7f60f409b920 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3986721673 0 0) 0x7f60e4003520 con 0x7f60f409b920 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 >> v1:172.21.15.148:6790/0 conn(0x7f60f413d320 legacy=0x7f60f413f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 >> v1:172.21.15.148:6789/0 conn(0x7f60f4136da0 legacy=0x7f60f413bc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:12.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f4099ee0 con 0x7f60f409b920 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60fb1d6640 5 monclient: authenticate success, global_id 4672 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60e8001610 con 0x7f60f409b920 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60fb1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: dump: 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.353 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60fb1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60e8001880 con 0x7f60f409b920 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 1 -- 172.21.15.171:0/3996270510 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60e8001c90 con 0x7f60f409b920 2024-09-16T04:08:12.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60fb1d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60fb1d6640 4 set_mon_vals no callback set 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.342+0000 7f60f37fe640 10 monclient: dump: 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.355 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.356 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.356 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 10 monclient: shutdown 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 1 -- 172.21.15.171:0/3996270510 >> v1:172.21.15.171:6789/0 conn(0x7f60f409b920 legacy=0x7f60f409bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 1 -- 172.21.15.171:0/3996270510 shutdown_connections 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 1 -- 172.21.15.171:0/3996270510 >> 172.21.15.171:0/3996270510 conn(0x7f60f40a2150 msgr2=0x7f60f40a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 1 -- 172.21.15.171:0/3996270510 shutdown_connections 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 1 -- 172.21.15.171:0/3996270510 wait complete. 2024-09-16T04:08:12.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 10 monclient: build_initial_monmap 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.343+0000 7f60fb1d6640 10 monclient: monmap: 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 1 Processor -- start 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 1 -- start start 2024-09-16T04:08:12.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient: init 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f60f409b920 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f60f4136da0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f60f413d320 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f41343a0 con 0x7f60f4136da0 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f412b970 con 0x7f60f413d320 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f4134ba0 con 0x7f60f409b920 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60fb1d6640 10 monclient(hunting): authenticate will time out at 867.849721s 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60f3fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f60f4136da0 legacy=0x7f60f41d08f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60f9773640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f60f413d320 0x7f60f41d4020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52360/0 (socket says 172.21.15.171:52360) 2024-09-16T04:08:12.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60f9773640 1 -- 172.21.15.171:0/472008084 learned_addr learned my addr 172.21.15.171:0/472008084 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.344+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095700330 0 0) 0x7f60f412b970 con 0x7f60f413d320 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): my global_id is 4675 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60d8003520 con 0x7f60f413d320 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392792408 0 0) 0x7f60f4134ba0 con 0x7f60f409b920 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): my global_id is 4703 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f412b970 con 0x7f60f409b920 2024-09-16T04:08:12.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2821815552 0 0) 0x7f60d8003520 con 0x7f60f413d320 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60f4134ba0 con 0x7f60f413d320 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60ec001cc0 con 0x7f60f413d320 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): dump: 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.361 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1959603800 0 0) 0x7f60f412b970 con 0x7f60f409b920 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60d8003520 con 0x7f60f409b920 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60e8001510 con 0x7f60f409b920 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient(hunting): dump: 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.362 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 335648488 0 0) 0x7f60f4134ba0 con 0x7f60f413d320 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 >> v1:172.21.15.148:6790/0 conn(0x7f60f409b920 legacy=0x7f60f40982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 >> v1:172.21.15.148:6789/0 conn(0x7f60f4136da0 legacy=0x7f60f41d08f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:12.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 monclient: found mon.b 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f412b690 con 0x7f60f413d320 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 5 monclient: authenticate success, global_id 4675 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 10 monclient: _renew_subs 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60f413c810 con 0x7f60f413d320 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 10 monclient: _renew_subs 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f60f413b8e0 con 0x7f60f413d320 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60ec002070 con 0x7f60f413d320 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60ec003de0 con 0x7f60f413d320 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f2ffd640 4 set_mon_vals no callback set 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.345+0000 7f60f17fa640 10 monclient: dump: 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.365 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f60ec004000 con 0x7f60f413d320 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7f60ec047190 con 0x7f60f413d320 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60fb1d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60fb1d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60fb1d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d4005080 con 0x7f60f413d320 2024-09-16T04:08:12.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.346+0000 7f60f8f72640 10 monclient: get_auth_request con 0x7f60d8038ea0 auth_method 0 2024-09-16T04:08:12.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.351+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f60ec015ff0 con 0x7f60f413d320 2024-09-16T04:08:12.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.351+0000 7f60f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.351+0000 7f60f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:12.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60fb1d6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60fb1d6640 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60fb1d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) -- 0x7f60d4005370 con 0x7f60f413d320 2024-09-16T04:08:12.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v42) ==== 162+0+0 (unknown 1346102702 0 0) 0x7f60ec019ca0 con 0x7f60f413d320 2024-09-16T04:08:12.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.516+0000 7f60f17fa640 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-16T04:08:12.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60fb1d6640 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60fb1d6640 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60fb1d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) -- 0x7f60d40008d0 con 0x7f60f413d320 2024-09-16T04:08:12.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60f17fa640 1 -- 172.21.15.171:0/472008084 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v42) ==== 162+0+0 (unknown 1346102702 0 0) 0x7f60ec013cc0 con 0x7f60f413d320 2024-09-16T04:08:12.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60f17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:12.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.614+0000 7f60f17fa640 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-16T04:08:12.617 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'fuggg' does not exist 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.616+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 >> v1:172.21.15.171:6812/7327273 conn(0x7f60d8038ea0 legacy=0x7f60d803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.616+0000 7f60fb1d6640 10 monclient: shutdown 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.616+0000 7f60fb1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.616+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 >> v1:172.21.15.171:6789/0 conn(0x7f60f413d320 legacy=0x7f60f41d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.617+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 shutdown_connections 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.617+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 >> 172.21.15.171:0/472008084 conn(0x7f60f40a2150 msgr2=0x7f60f409c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.617+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 shutdown_connections 2024-09-16T04:08:12.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.617+0000 7f60fb1d6640 1 -- 172.21.15.171:0/472008084 wait complete. 2024-09-16T04:08:12.627 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo OK 2024-09-16T04:08:12.628 INFO:tasks.workunit.client.0.smithi171.stdout:OK 2024-09-16T04:08:12.629 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T04:08:12.629 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T04:08:12.653 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-16T04:08:12.655 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-16T04:08:12.655 DEBUG:teuthology.orchestra.run.smithi171:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-16T04:08:12.714 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush dump 2024-09-16T04:08:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.790+0000 7fb702fc4640 10 monclient: get_monmap_and_config 2024-09-16T04:08:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.790+0000 7fb702fc4640 10 monclient: build_initial_monmap 2024-09-16T04:08:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.790+0000 7fb702fc4640 10 monclient: monmap: 2024-09-16T04:08:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:12.791400+0000 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:12.791400+0000 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:12.793 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 1 Processor -- start 2024-09-16T04:08:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 1 -- start start 2024-09-16T04:08:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient: init 2024-09-16T04:08:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient(hunting): picked mon.noname-c con 0x7fb6fc130930 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient(hunting): picked mon.noname-a con 0x7fb6fc09eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient(hunting): picked mon.noname-b con 0x7fb6fc130d10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc1343a0 con 0x7fb6fc09eb60 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc09c1b0 con 0x7fb6fc130930 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc09d8e0 con 0x7fb6fc130d10 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb702fc4640 10 monclient(hunting): authenticate will time out at 868.296864s 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb700d60640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb6fc130930 0x7fb6fc133580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52368/0 (socket says 172.21.15.171:52368) 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb6f3fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb6fc09eb60 legacy=0x7fb6fc133c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.791+0000 7fb700d60640 1 -- 172.21.15.171:0/1550485395 learned_addr learned my addr 172.21.15.171:0/1550485395 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372378746 0 0) 0x7fb6fc09c1b0 con 0x7fb6fc130930 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient(hunting): my global_id is 4678 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6e4003520 con 0x7fb6fc130930 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 758991867 0 0) 0x7fb6e4003520 con 0x7fb6fc130930 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 >> v1:172.21.15.148:6790/0 conn(0x7fb6fc130d10 legacy=0x7fb6fc13c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 >> v1:172.21.15.148:6789/0 conn(0x7fb6fc09eb60 legacy=0x7fb6fc133c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc09dbc0 con 0x7fb6fc130930 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f8001990 con 0x7fb6fc130930 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 5 monclient: authenticate success, global_id 4678 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: dump: 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6f8001c80 con 0x7fb6fc130930 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 1 -- 172.21.15.171:0/1550485395 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f8002050 con 0x7fb6fc130930 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb6f37fe640 10 monclient: dump: 2024-09-16T04:08:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 4 set_mon_vals no callback set 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 10 monclient: shutdown 2024-09-16T04:08:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.792+0000 7fb702fc4640 1 -- 172.21.15.171:0/1550485395 >> v1:172.21.15.171:6789/0 conn(0x7fb6fc130930 legacy=0x7fb6fc133580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 1 -- 172.21.15.171:0/1550485395 shutdown_connections 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 1 -- 172.21.15.171:0/1550485395 >> 172.21.15.171:0/1550485395 conn(0x7fb6fc0a52a0 msgr2=0x7fb6fc0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 1 -- 172.21.15.171:0/1550485395 shutdown_connections 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 1 -- 172.21.15.171:0/1550485395 wait complete. 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 10 monclient: build_initial_monmap 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 10 monclient: monmap: 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.805 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 1 Processor -- start 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 1 -- start start 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.793+0000 7fb702fc4640 10 monclient: init 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient(hunting): picked mon.noname-a con 0x7fb6fc09eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient(hunting): picked mon.noname-c con 0x7fb6fc130930 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient(hunting): picked mon.noname-b con 0x7fb6fc130d10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc123810 con 0x7fb6fc09eb60 2024-09-16T04:08:12.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc122e30 con 0x7fb6fc130d10 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc12fec0 con 0x7fb6fc130930 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb702fc4640 10 monclient(hunting): authenticate will time out at 868.299548s 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb700d60640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb6fc09eb60 legacy=0x7fb6fc1cb860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb701561640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb6fc130d10 0x7fb6fc1d3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52380/0 (socket says 172.21.15.171:52380) 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb701561640 1 -- 172.21.15.171:0/2011027646 learned_addr learned my addr 172.21.15.171:0/2011027646 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585898675 0 0) 0x7fb6fc122e30 con 0x7fb6fc130d10 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): my global_id is 4681 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6e8003520 con 0x7fb6fc130d10 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933440396 0 0) 0x7fb6fc12fec0 con 0x7fb6fc130930 2024-09-16T04:08:12.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 10 monclient(hunting): my global_id is 4709 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.794+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6fc122e30 con 0x7fb6fc130930 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2345159927 0 0) 0x7fb6e8003520 con 0x7fb6fc130d10 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6fc12fec0 con 0x7fb6fc130d10 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f40018f0 con 0x7fb6fc130d10 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient(hunting): dump: 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.808 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 837784703 0 0) 0x7fb6fc12fec0 con 0x7fb6fc130d10 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 >> v1:172.21.15.148:6790/0 conn(0x7fb6fc130930 legacy=0x7fb6fc1d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 >> v1:172.21.15.148:6789/0 conn(0x7fb6fc09eb60 legacy=0x7fb6fc1cb860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 monclient: found mon.b 2024-09-16T04:08:12.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc0a3020 con 0x7fb6fc130d10 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 5 monclient: authenticate success, global_id 4681 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 10 monclient: _renew_subs 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc06cea0 con 0x7fb6fc130d10 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 10 monclient: _renew_subs 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6fc121420 con 0x7fb6fc130d10 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6f4001cc0 con 0x7fb6fc130d10 2024-09-16T04:08:12.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f4003a60 con 0x7fb6fc130d10 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f2ffd640 4 set_mon_vals no callback set 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.795+0000 7fb6f17fa640 10 monclient: dump: 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:12.811 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb6f4003c00 con 0x7fb6fc130d10 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb702fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb702fc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb702fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6dc005080 con 0x7fb6fc130d10 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7fb6f4046d80 con 0x7fb6fc130d10 2024-09-16T04:08:12.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.796+0000 7fb700d60640 10 monclient: get_auth_request con 0x7fb6e8038fb0 auth_method 0 2024-09-16T04:08:12.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.800+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb6f4015d00 con 0x7fb6fc130d10 2024-09-16T04:08:12.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.800+0000 7fb6f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:12.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.800+0000 7fb6f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:12.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.969+0000 7fb702fc4640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T04:08:12.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.969+0000 7fb702fc4640 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-16T04:08:12.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.969+0000 7fb702fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:12.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.969+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush dump"} v 0) -- 0x7fb6dc005370 con 0x7fb6fc130d10 2024-09-16T04:08:12.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.970+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v42) ==== 62+0+7331 (unknown 3275263029 0 4071782663) 0x7fb6f4015600 con 0x7fb6fc130d10 2024-09-16T04:08:12.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.970+0000 7fb6f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-16T04:08:12.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:12.970+0000 7fb6f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb702fc4640 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb702fc4640 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb702fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush dump"} v 0) -- 0x7fb6dc000bb0 con 0x7fb6fc130d10 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb6f17fa640 1 -- 172.21.15.171:0/2011027646 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v42) ==== 62+0+7331 (unknown 3275263029 0 4071782663) 0x7fb6f401e1b0 con 0x7fb6fc130d10 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb6f17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.070+0000 7fb6f17fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stdout:{ 2024-09-16T04:08:13.073 INFO:tasks.workunit.client.0.smithi171.stdout: "devices": [ 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 0, 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd.0", 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: "class": "ssd" 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 1, 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd.1", 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: "class": "ssd" 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.074 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 2, 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd.2", 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "class": "ssd" 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 3, 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd.3", 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "class": "ssd" 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.075 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 4, 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd.4", 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: "class": "ssd" 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 5, 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd.5", 2024-09-16T04:08:13.076 INFO:tasks.workunit.client.0.smithi171.stdout: "class": "ssd" 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: "types": [ 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 0, 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "osd" 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 1, 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "host" 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.077 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 2, 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "chassis" 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 3, 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "rack" 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 4, 2024-09-16T04:08:13.078 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "row" 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 5, 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "pdu" 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 6, 2024-09-16T04:08:13.079 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "pod" 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 7, 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "room" 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 8, 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "datacenter" 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.080 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 9, 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "zone" 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 10, 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "region" 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.081 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 11, 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "root" 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: "buckets": [ 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -1, 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "default", 2024-09-16T04:08:13.082 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 11, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "type_name": "root", 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 34560, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "alg": "straw2", 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "hash": "rjenkins1", 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "items": [ 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -3, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 0 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -5, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.083 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 1 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -2, 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "default~ssd", 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 11, 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "type_name": "root", 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 34560, 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "alg": "straw2", 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "hash": "rjenkins1", 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: "items": [ 2024-09-16T04:08:13.084 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -4, 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 0 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -6, 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 1 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.085 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -3, 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "smithi148", 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 1, 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "type_name": "host", 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "alg": "straw2", 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "hash": "rjenkins1", 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "items": [ 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 0, 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 0 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.086 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 2, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 1 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 1, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 2 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -4, 2024-09-16T04:08:13.087 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "smithi148~ssd", 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 1, 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "type_name": "host", 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "alg": "straw2", 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "hash": "rjenkins1", 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "items": [ 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 0, 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 0 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.088 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 2, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 1 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 1, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 2 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -5, 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "smithi171", 2024-09-16T04:08:13.089 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 1, 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "type_name": "host", 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "alg": "straw2", 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "hash": "rjenkins1", 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "items": [ 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 5, 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 0 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 4, 2024-09-16T04:08:13.090 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 1 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 3, 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 2 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "id": -6, 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "name": "smithi171~ssd", 2024-09-16T04:08:13.091 INFO:tasks.workunit.client.0.smithi171.stdout: "type_id": 1, 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "type_name": "host", 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 17280, 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "alg": "straw2", 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "hash": "rjenkins1", 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "items": [ 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 5, 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 0 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 4, 2024-09-16T04:08:13.092 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 1 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: "id": 3, 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: "weight": 5760, 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: "pos": 2 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: "rules": [ 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.093 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_id": 0, 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_name": "replicated_rule", 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "type": 1, 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "steps": [ 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "take", 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "item": -1, 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "item_name": "default" 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "choose_firstn", 2024-09-16T04:08:13.094 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 0, 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: "type": "osd" 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "emit" 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_id": 1, 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_name": "erasure-code", 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: "type": 3, 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: "steps": [ 2024-09-16T04:08:13.095 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "set_chooseleaf_tries", 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 5 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "set_choose_tries", 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 100 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "take", 2024-09-16T04:08:13.096 INFO:tasks.workunit.client.0.smithi171.stdout: "item": -1, 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: "item_name": "default" 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "choose_indep", 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 0, 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: "type": "osd" 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "emit" 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.097 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "tunables": { 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "choose_local_tries": 0, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "choose_local_fallback_tries": 0, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "choose_total_tries": 50, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "chooseleaf_descend_once": 1, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "chooseleaf_vary_r": 1, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "chooseleaf_stable": 1, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "msr_descents": 100, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "msr_collision_tries": 100, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "straw_calc_version": 1, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "allowed_bucket_algs": 54, 2024-09-16T04:08:13.098 INFO:tasks.workunit.client.0.smithi171.stdout: "profile": "jewel", 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "optimal_tunables": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "legacy_tunables": 0, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "minimum_required_version": "jewel", 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "require_feature_tunables": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "require_feature_tunables2": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "has_v2_rules": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "require_feature_tunables3": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "has_v3_rules": 0, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "has_v4_buckets": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "require_feature_tunables5": 1, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "has_v5_rules": 0, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "has_msr_rules": 0 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.099 INFO:tasks.workunit.client.0.smithi171.stdout: "choose_args": {} 2024-09-16T04:08:13.100 INFO:tasks.workunit.client.0.smithi171.stdout:} 2024-09-16T04:08:13.100 INFO:tasks.workunit.client.0.smithi171.stdout: 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 >> v1:172.21.15.171:6812/7327273 conn(0x7fb6e8038fb0 legacy=0x7fb6e803b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 10 monclient: shutdown 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 >> v1:172.21.15.171:6789/0 conn(0x7fb6fc130d10 legacy=0x7fb6fc1d3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 shutdown_connections 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 >> 172.21.15.171:0/2011027646 conn(0x7fb6fc0a52a0 msgr2=0x7fb6fc133900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 shutdown_connections 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.072+0000 7fb702fc4640 1 -- 172.21.15.171:0/2011027646 wait complete. 2024-09-16T04:08:13.101 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule dump 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.159+0000 7f2719ec0640 10 monclient: get_monmap_and_config 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.159+0000 7f2719ec0640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.159+0000 7f2719ec0640 10 monclient: monmap: 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:13.160214+0000 2024-09-16T04:08:13.161 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:13.160214+0000 2024-09-16T04:08:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:13.162 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 1 Processor -- start 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 1 -- start start 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient: init 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f271409b660 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f27140734b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f2714136d70 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271409c150 con 0x7f27140734b0 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27140998b0 con 0x7f2714136d70 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2714099b90 con 0x7f271409b660 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2719ec0640 10 monclient(hunting): authenticate will time out at 868.665682s 2024-09-16T04:08:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2713fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2714136d70 0x7f271413f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52384/0 (socket says 172.21.15.171:52384) 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2713fff640 1 -- 172.21.15.171:0/2849276352 learned_addr learned my addr 172.21.15.171:0/2849276352 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f2712ffd640 1 -- 172.21.15.171:0/2849276352 >> v1:172.21.15.148:6789/0 conn(0x7f27140734b0 legacy=0x7f271413bbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.160+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024057369 0 0) 0x7f27140998b0 con 0x7f2714136d70 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient(hunting): my global_id is 4684 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26fc003520 con 0x7f2714136d70 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 848959222 0 0) 0x7f26fc003520 con 0x7f2714136d70 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 >> v1:172.21.15.148:6790/0 conn(0x7f271409b660 legacy=0x7f271409ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 >> v1:172.21.15.148:6789/0 conn(0x7f27140734b0 legacy=0x7f271413bbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2714099e70 con 0x7f2714136d70 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c001990 con 0x7f2714136d70 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 5 monclient: authenticate success, global_id 4684 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: dump: 2024-09-16T04:08:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f270c001c80 con 0x7f2714136d70 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 1 -- 172.21.15.171:0/2849276352 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c002050 con 0x7f2714136d70 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 4 set_mon_vals no callback set 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f27127fc640 10 monclient: dump: 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 10 monclient: shutdown 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 1 -- 172.21.15.171:0/2849276352 >> v1:172.21.15.171:6789/0 conn(0x7f2714136d70 legacy=0x7f271413f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 1 -- 172.21.15.171:0/2849276352 shutdown_connections 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 1 -- 172.21.15.171:0/2849276352 >> 172.21.15.171:0/2849276352 conn(0x7f27140a1e90 msgr2=0x7f27140a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 1 -- 172.21.15.171:0/2849276352 shutdown_connections 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.161+0000 7f2719ec0640 1 -- 172.21.15.171:0/2849276352 wait complete. 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.162+0000 7f2719ec0640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.162+0000 7f2719ec0640 10 monclient: monmap: 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.162+0000 7f2719ec0640 1 Processor -- start 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.162+0000 7f2719ec0640 1 -- start start 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.162+0000 7f2719ec0640 10 monclient: init 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f27140734b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f271409b660 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f2714136d70 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27140931f0 con 0x7f27140734b0 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271408f380 con 0x7f2714136d70 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2714098180 con 0x7f271409b660 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient(hunting): authenticate will time out at 868.668339s 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f27137fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f27140734b0 legacy=0x7f271408e560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2713fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2714136d70 0x7f27141d39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52400/0 (socket says 172.21.15.171:52400) 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2713fff640 1 -- 172.21.15.171:0/1377759095 learned_addr learned my addr 172.21.15.171:0/1377759095 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300153584 0 0) 0x7f271408f380 con 0x7f2714136d70 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): my global_id is 4687 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2700003550 con 0x7f2714136d70 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572751318 0 0) 0x7f2714098180 con 0x7f271409b660 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): my global_id is 4715 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271408f380 con 0x7f271409b660 2024-09-16T04:08:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 342558241 0 0) 0x7f2700003550 con 0x7f2714136d70 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2714098180 con 0x7f2714136d70 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c0017b0 con 0x7f2714136d70 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): dump: 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 539271068 0 0) 0x7f2714098180 con 0x7f2714136d70 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 >> v1:172.21.15.148:6790/0 conn(0x7f271409b660 legacy=0x7f271408ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 >> v1:172.21.15.148:6789/0 conn(0x7f27140734b0 legacy=0x7f271408e560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 monclient: found mon.b 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27140901c0 con 0x7f2714136d70 2024-09-16T04:08:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient: _finish_auth 0 2024-09-16T04:08:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f26fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 5 monclient: authenticate success, global_id 4687 2024-09-16T04:08:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient: _renew_subs 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.163+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2714092f10 con 0x7f2714136d70 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f2719ec0640 10 monclient: _renew_subs 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f2719ec0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2714090e40 con 0x7f2714136d70 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f270c001c80 con 0x7f2714136d70 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c0051e0 con 0x7f2714136d70 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f2711ffb640 4 set_mon_vals no callback set 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 10 monclient: dump: 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f270c010610 con 0x7f2714136d70 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.164+0000 7f2719ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.165+0000 7f2719ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.165+0000 7f2719ec0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.165+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f4005080 con 0x7f2714136d70 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.165+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7f270c0471f0 con 0x7f2714136d70 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.165+0000 7f27137fe640 10 monclient: get_auth_request con 0x7f2700038f90 auth_method 0 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.169+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f270c016150 con 0x7f2714136d70 2024-09-16T04:08:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.169+0000 7f26fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.169+0000 7f26fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.338+0000 7f2719ec0640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.338+0000 7f2719ec0640 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.338+0000 7f2719ec0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.338+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) -- 0x7f26f4005370 con 0x7f2714136d70 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.339+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v42) ==== 67+0+1058 (unknown 2429497187 0 3068570633) 0x7f270c01e600 con 0x7f2714136d70 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.339+0000 7f26fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-16T04:08:13.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.339+0000 7f26fbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:13.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f2719ec0640 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-16T04:08:13.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f2719ec0640 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T04:08:13.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f2719ec0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule dump"} v 0) -- 0x7f26f4001090 con 0x7f2714136d70 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f26fbfff640 1 -- 172.21.15.171:0/1377759095 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v42) ==== 67+0+1058 (unknown 2429497187 0 3068570633) 0x7f270c013e20 con 0x7f2714136d70 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f26fbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.439+0000 7f26fbfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stdout:[ 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_id": 0, 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_name": "replicated_rule", 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stdout: "type": 1, 2024-09-16T04:08:13.442 INFO:tasks.workunit.client.0.smithi171.stdout: "steps": [ 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "take", 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "item": -1, 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "item_name": "default" 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "choose_firstn", 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 0, 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "type": "osd" 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "emit" 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.443 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_id": 1, 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: "rule_name": "erasure-code", 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: "type": 3, 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: "steps": [ 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "set_chooseleaf_tries", 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 5 2024-09-16T04:08:13.444 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "set_choose_tries", 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 100 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "take", 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "item": -1, 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "item_name": "default" 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "choose_indep", 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "num": 0, 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: "type": "osd" 2024-09-16T04:08:13.445 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout: "op": "emit" 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout:] 2024-09-16T04:08:13.446 INFO:tasks.workunit.client.0.smithi171.stdout: 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 >> v1:172.21.15.171:6812/7327273 conn(0x7f2700038f90 legacy=0x7f270003b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 10 monclient: shutdown 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 >> v1:172.21.15.171:6789/0 conn(0x7f2714136d70 legacy=0x7f27141d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 shutdown_connections 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 >> 172.21.15.171:0/1377759095 conn(0x7f27140a1e90 msgr2=0x7f271409fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 shutdown_connections 2024-09-16T04:08:13.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.441+0000 7f2719ec0640 1 -- 172.21.15.171:0/1377759095 wait complete. 2024-09-16T04:08:13.452 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls 2024-09-16T04:08:13.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.529+0000 7f22a3d0a640 10 monclient: get_monmap_and_config 2024-09-16T04:08:13.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.529+0000 7f22a3d0a640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.529+0000 7f22a3d0a640 10 monclient: monmap: 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:13.530542+0000 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:13.530542+0000 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:13.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 1 Processor -- start 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 1 -- start start 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient: init 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient(hunting): picked mon.noname-c con 0x7f229c0685d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient(hunting): picked mon.noname-b con 0x7f229c1368d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient(hunting): picked mon.noname-a con 0x7f229c13a3e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c133ef0 con 0x7f229c13a3e0 2024-09-16T04:08:13.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c09f430 con 0x7f229c0685d0 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c09f150 con 0x7f229c1368d0 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.530+0000 7f22a3d0a640 10 monclient(hunting): authenticate will time out at 869.035985s 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a1aa6640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f229c0685d0 0x7f229c09b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52416/0 (socket says 172.21.15.171:52416) 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a1aa6640 1 -- 172.21.15.171:0/4294416630 learned_addr learned my addr 172.21.15.171:0/4294416630 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a22a7640 1 -- 172.21.15.171:0/4294416630 >> v1:172.21.15.148:6789/0 conn(0x7f229c13a3e0 legacy=0x7f229c13c7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155359357 0 0) 0x7f229c09f430 con 0x7f229c0685d0 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient(hunting): my global_id is 4690 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2288003520 con 0x7f229c0685d0 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2780342803 0 0) 0x7f2288003520 con 0x7f229c0685d0 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 >> v1:172.21.15.148:6790/0 conn(0x7f229c1368d0 legacy=0x7f229c138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 >> v1:172.21.15.148:6789/0 conn(0x7f229c13a3e0 legacy=0x7f229c13c7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: _finish_hunting 0 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 monclient: found mon.noname-c 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c09daa0 con 0x7f229c0685d0 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: _finish_auth 0 2024-09-16T04:08:13.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: _check_auth_tickets 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a3d0a640 5 monclient: authenticate success, global_id 4690 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2298001610 con 0x7f229c0685d0 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a3d0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: dump: 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.543 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a3d0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2298001970 con 0x7f229c0685d0 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a3d0a640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 1 -- 172.21.15.171:0/4294416630 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2298001d40 con 0x7f229c0685d0 2024-09-16T04:08:13.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a3d0a640 4 set_mon_vals no callback set 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.531+0000 7f22a0aa4640 10 monclient: dump: 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 10 monclient: shutdown 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 1 -- 172.21.15.171:0/4294416630 >> v1:172.21.15.171:6789/0 conn(0x7f229c0685d0 legacy=0x7f229c09b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 1 -- 172.21.15.171:0/4294416630 shutdown_connections 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 1 -- 172.21.15.171:0/4294416630 >> 172.21.15.171:0/4294416630 conn(0x7f229c0a1e90 msgr2=0x7f229c0a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 1 -- 172.21.15.171:0/4294416630 shutdown_connections 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 1 -- 172.21.15.171:0/4294416630 wait complete. 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.532+0000 7f22a3d0a640 10 monclient: monmap: 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 1 Processor -- start 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 1 -- start start 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient: init 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient(hunting): picked mon.noname-b con 0x7f229c0685d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient(hunting): picked mon.noname-a con 0x7f229c1368d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient(hunting): picked mon.noname-c con 0x7f229c13a3e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c13d8a0 con 0x7f229c1368d0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c13b5e0 con 0x7f229c0685d0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f229c13bab0 con 0x7f229c13a3e0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a3d0a640 10 monclient(hunting): authenticate will time out at 869.038600s 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a1aa6640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f229c0685d0 0x7f229c13a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52426/0 (socket says 172.21.15.171:52426) 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a1aa6640 1 -- 172.21.15.171:0/84181901 learned_addr learned my addr 172.21.15.171:0/84181901 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22a12a5640 1 -- 172.21.15.171:0/84181901 >> v1:172.21.15.148:6789/0 conn(0x7f229c1368d0 legacy=0x7f229c13aed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:13.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720998472 0 0) 0x7f229c13b5e0 con 0x7f229c0685d0 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22927fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22927fc640 10 monclient(hunting): my global_id is 4693 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.533+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2280003520 con 0x7f229c0685d0 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077658040 0 0) 0x7f229c13bab0 con 0x7f229c13a3e0 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): my global_id is 4721 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f229c13b5e0 con 0x7f229c13a3e0 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1917883148 0 0) 0x7f2280003520 con 0x7f229c0685d0 2024-09-16T04:08:13.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f229c13bab0 con 0x7f229c0685d0 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f22980015d0 con 0x7f229c0685d0 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): dump: 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.550 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3128580858 0 0) 0x7f229c13bab0 con 0x7f229c0685d0 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 >> v1:172.21.15.148:6790/0 conn(0x7f229c13a3e0 legacy=0x7f229c1d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 >> v1:172.21.15.148:6789/0 conn(0x7f229c1368d0 legacy=0x7f229c13aed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 monclient: found mon.b 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c09c580 con 0x7f229c0685d0 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:13.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 5 monclient: authenticate success, global_id 4693 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 10 monclient: _renew_subs 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f229c13c5f0 con 0x7f229c0685d0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 10 monclient: _renew_subs 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f229c13f710 con 0x7f229c0685d0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2298001970 con 0x7f229c0685d0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2298004c30 con 0x7f229c0685d0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f2293fff640 4 set_mon_vals no callback set 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.534+0000 7f22927fc640 10 monclient: dump: 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2298004dd0 con 0x7f229c0685d0 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22a3d0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22a3d0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22a3d0a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2284005080 con 0x7f229c0685d0 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7f2298046c60 con 0x7f229c0685d0 2024-09-16T04:08:13.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.535+0000 7f22a12a5640 10 monclient: get_auth_request con 0x7f2280038fb0 auth_method 0 2024-09-16T04:08:13.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.539+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2298015c80 con 0x7f229c0685d0 2024-09-16T04:08:13.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.539+0000 7f22927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.539+0000 7f22927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:13.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22a3d0a640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:13.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22a3d0a640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:13.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22a3d0a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f2284005370 con 0x7f229c0685d0 2024-09-16T04:08:13.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v42) ==== 65+0+29 (unknown 2266644849 0 852635902) 0x7f2298015580 con 0x7f229c0685d0 2024-09-16T04:08:13.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:13.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.707+0000 7f22927fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:13.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22a3d0a640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22a3d0a640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22a3d0a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f2284000bc0 con 0x7f229c0685d0 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22927fc640 1 -- 172.21.15.171:0/84181901 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v42) ==== 65+0+29 (unknown 2266644849 0 852635902) 0x7f22980138e0 con 0x7f229c0685d0 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22927fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.806+0000 7f22927fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:13.809 INFO:tasks.workunit.client.0.smithi171.stdout:replicated_rule 2024-09-16T04:08:13.810 INFO:tasks.workunit.client.0.smithi171.stdout:erasure-code 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 >> v1:172.21.15.171:6812/7327273 conn(0x7f2280038fb0 legacy=0x7f228003b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 10 monclient: shutdown 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 >> v1:172.21.15.171:6789/0 conn(0x7f229c0685d0 legacy=0x7f229c13a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 shutdown_connections 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 >> 172.21.15.171:0/84181901 conn(0x7f229c0a1e90 msgr2=0x7f229c0a1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.808+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 shutdown_connections 2024-09-16T04:08:13.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.809+0000 7f22a3d0a640 1 -- 172.21.15.171:0/84181901 wait complete. 2024-09-16T04:08:13.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule list 2024-09-16T04:08:13.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.896+0000 7fc234749640 10 monclient: get_monmap_and_config 2024-09-16T04:08:13.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.896+0000 7fc234749640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.896+0000 7fc234749640 10 monclient: monmap: 2024-09-16T04:08:13.898 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:13.898 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.898 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:13.897004+0000 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:13.897004+0000 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.896+0000 7fc234749640 1 Processor -- start 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.896+0000 7fc234749640 1 -- start start 2024-09-16T04:08:13.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.896+0000 7fc234749640 10 monclient: init 2024-09-16T04:08:13.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:13.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient(hunting): picked mon.noname-a con 0x7fc22c09f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient(hunting): picked mon.noname-c con 0x7fc22c0a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient(hunting): picked mon.noname-b con 0x7fc22c09e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c0a1260 con 0x7fc22c09f640 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c09d0c0 con 0x7fc22c0a0130 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c09d3a0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc234749640 10 monclient(hunting): authenticate will time out at 869.402562s 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc231ce4640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc22c0a0130 0x7fc22c138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52430/0 (socket says 172.21.15.171:52430) 2024-09-16T04:08:13.907 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc231ce4640 1 -- 172.21.15.171:0/2806707968 learned_addr learned my addr 172.21.15.171:0/2806707968 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2324e5640 1 -- 172.21.15.171:0/2806707968 >> v1:172.21.15.148:6789/0 conn(0x7fc22c09f640 legacy=0x7fc22c09fa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152034270 0 0) 0x7fc22c09d0c0 con 0x7fc22c0a0130 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2314e3640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2314e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2314e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2314e3640 10 monclient(hunting): my global_id is 4696 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.897+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc218003440 con 0x7fc22c0a0130 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1863960910 0 0) 0x7fc22c09d3a0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.908 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient(hunting): my global_id is 4724 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc22c09d0c0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 710992592 0 0) 0x7fc218003440 con 0x7fc22c0a0130 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 >> v1:172.21.15.148:6790/0 conn(0x7fc22c09e1d0 legacy=0x7fc22c13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 >> v1:172.21.15.148:6789/0 conn(0x7fc22c09f640 legacy=0x7fc22c09fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: _finish_hunting 0 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 monclient: found mon.noname-c 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:13.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c09d680 con 0x7fc22c0a0130 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: _finish_auth 0 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: _check_auth_tickets 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc21c001540 con 0x7fc22c0a0130 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 5 monclient: authenticate success, global_id 4696 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: dump: 2024-09-16T04:08:13.910 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc21c0017a0 con 0x7fc22c0a0130 2024-09-16T04:08:13.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 1 -- 172.21.15.171:0/2806707968 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc21c001af0 con 0x7fc22c0a0130 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 4 set_mon_vals no callback set 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc2314e3640 10 monclient: dump: 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.912 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 10 monclient: shutdown 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 1 -- 172.21.15.171:0/2806707968 >> v1:172.21.15.171:6789/0 conn(0x7fc22c0a0130 legacy=0x7fc22c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 1 -- 172.21.15.171:0/2806707968 shutdown_connections 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 1 -- 172.21.15.171:0/2806707968 >> 172.21.15.171:0/2806707968 conn(0x7fc22c0a52a0 msgr2=0x7fc22c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 1 -- 172.21.15.171:0/2806707968 shutdown_connections 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.898+0000 7fc234749640 1 -- 172.21.15.171:0/2806707968 wait complete. 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.899+0000 7fc234749640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.899+0000 7fc234749640 10 monclient: monmap: 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:13.913 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.899+0000 7fc234749640 1 Processor -- start 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.899+0000 7fc234749640 1 -- start start 2024-09-16T04:08:13.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.899+0000 7fc234749640 10 monclient: init 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient(hunting): picked mon.noname-b con 0x7fc22c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient(hunting): picked mon.noname-a con 0x7fc22c09f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient(hunting): picked mon.noname-c con 0x7fc22c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c1c14d0 con 0x7fc22c09f640 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c1a5cb0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c1a8720 con 0x7fc22c0a0130 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient(hunting): authenticate will time out at 869.405320s 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc231ce4640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc22c09f640 legacy=0x7fc22c1cc0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc2324e5640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc22c09e1d0 0x7fc22c1c0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52440/0 (socket says 172.21.15.171:52440) 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc2324e5640 1 -- 172.21.15.171:0/217612429 learned_addr learned my addr 172.21.15.171:0/217612429 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1281559151 0 0) 0x7fc22c1a5cb0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4699 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc210003520 con 0x7fc22c09e1d0 2024-09-16T04:08:13.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 921586755 0 0) 0x7fc22c1a8720 con 0x7fc22c0a0130 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): my global_id is 4727 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc22c1a5cb0 con 0x7fc22c0a0130 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1241218292 0 0) 0x7fc210003520 con 0x7fc22c09e1d0 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc22c1a8720 con 0x7fc22c09e1d0 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2280041d0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:13.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:13.918 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.918 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.920 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 430753317 0 0) 0x7fc22c1a8720 con 0x7fc22c09e1d0 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 >> v1:172.21.15.148:6790/0 conn(0x7fc22c0a0130 legacy=0x7fc22c1cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 >> v1:172.21.15.148:6789/0 conn(0x7fc22c09f640 legacy=0x7fc22c1cc0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 monclient: found mon.b 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c0721e0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:13.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 5 monclient: authenticate success, global_id 4699 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient: _renew_subs 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.900+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c0a4140 con 0x7fc22c09e1d0 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 10 monclient: _renew_subs 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc22c1a8440 con 0x7fc22c09e1d0 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc228004430 con 0x7fc22c09e1d0 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:13.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc228004800 con 0x7fc22c09e1d0 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc230ce2640 4 set_mon_vals no callback set 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 10 monclient: dump: 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:13.923 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc228004a20 con 0x7fc22c09e1d0 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.901+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc214005080 con 0x7fc22c09e1d0 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.902+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7fc228046dd0 con 0x7fc22c09e1d0 2024-09-16T04:08:13.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.902+0000 7fc231ce4640 10 monclient: get_auth_request con 0x7fc210038fe0 auth_method 0 2024-09-16T04:08:13.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.906+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc228015d30 con 0x7fc22c09e1d0 2024-09-16T04:08:13.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.906+0000 7fc222ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:13.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:13.906+0000 7fc222ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:14.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc234749640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T04:08:14.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc234749640 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-16T04:08:14.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc234749640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:14.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) -- 0x7fc214005370 con 0x7fc22c09e1d0 2024-09-16T04:08:14.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v42) ==== 67+0+29 (unknown 527599421 0 852635902) 0x7fc228015630 con 0x7fc22c09e1d0 2024-09-16T04:08:14.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc222ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-16T04:08:14.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.074+0000 7fc222ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc234749640 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc234749640 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc234749640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule list"} v 0) -- 0x7fc214000bc0 con 0x7fc22c09e1d0 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc222ffd640 1 -- 172.21.15.171:0/217612429 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v42) ==== 67+0+29 (unknown 527599421 0 852635902) 0x7fc228013990 con 0x7fc22c09e1d0 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc222ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.175+0000 7fc222ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stdout:replicated_rule 2024-09-16T04:08:14.177 INFO:tasks.workunit.client.0.smithi171.stdout:erasure-code 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 >> v1:172.21.15.171:6812/7327273 conn(0x7fc210038fe0 legacy=0x7fc21003b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 10 monclient: shutdown 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 >> v1:172.21.15.171:6789/0 conn(0x7fc22c09e1d0 legacy=0x7fc22c1c0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 shutdown_connections 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 >> 172.21.15.171:0/217612429 conn(0x7fc22c0a52a0 msgr2=0x7fc22c0726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 shutdown_connections 2024-09-16T04:08:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.177+0000 7fc234749640 1 -- 172.21.15.171:0/217612429 wait complete. 2024-09-16T04:08:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T04:08:14.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.270+0000 7fc287d31640 10 monclient: get_monmap_and_config 2024-09-16T04:08:15.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.270+0000 7fc287d31640 10 monclient: build_initial_monmap 2024-09-16T04:08:15.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.270+0000 7fc287d31640 10 monclient: monmap: 2024-09-16T04:08:15.059 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:15.059 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:14.271001+0000 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:14.271001+0000 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:15.060 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.270+0000 7fc287d31640 1 Processor -- start 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.270+0000 7fc287d31640 1 -- start start 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.270+0000 7fc287d31640 10 monclient: init 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient(hunting): picked mon.noname-a con 0x7fc280136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient(hunting): picked mon.noname-c con 0x7fc280137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient(hunting): picked mon.noname-b con 0x7fc2801375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28009b2c0 con 0x7fc280136d60 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28009b5a0 con 0x7fc280137140 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2800a6640 con 0x7fc2801375a0 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc287d31640 10 monclient(hunting): authenticate will time out at 869.776481s 2024-09-16T04:08:15.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc2852cc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc280137140 0x7fc28013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52446/0 (socket says 172.21.15.171:52446) 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc2852cc640 1 -- 172.21.15.171:0/2896952130 learned_addr learned my addr 172.21.15.171:0/2896952130 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc285acd640 1 -- 172.21.15.171:0/2896952130 >> v1:172.21.15.148:6789/0 conn(0x7fc280136d60 legacy=0x7fc28009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640371229 0 0) 0x7fc28009b5a0 con 0x7fc280137140 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc284acb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc284acb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc284acb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc284acb640 10 monclient(hunting): my global_id is 4702 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.271+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc26c003520 con 0x7fc280137140 2024-09-16T04:08:15.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 45157764 0 0) 0x7fc26c003520 con 0x7fc280137140 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 >> v1:172.21.15.148:6790/0 conn(0x7fc2801375a0 legacy=0x7fc28013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 >> v1:172.21.15.148:6789/0 conn(0x7fc280136d60 legacy=0x7fc28009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 monclient: found mon.noname-c 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc280143b20 con 0x7fc280137140 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: _finish_auth 0 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc270001610 con 0x7fc280137140 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 5 monclient: authenticate success, global_id 4702 2024-09-16T04:08:15.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: dump: 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.072 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc270001900 con 0x7fc280137140 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 1 -- 172.21.15.171:0/2896952130 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc270001cd0 con 0x7fc280137140 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 4 set_mon_vals no callback set 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc284acb640 10 monclient: dump: 2024-09-16T04:08:15.073 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.074 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 10 monclient: shutdown 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 1 -- 172.21.15.171:0/2896952130 >> v1:172.21.15.171:6789/0 conn(0x7fc280137140 legacy=0x7fc28013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 1 -- 172.21.15.171:0/2896952130 shutdown_connections 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 1 -- 172.21.15.171:0/2896952130 >> 172.21.15.171:0/2896952130 conn(0x7fc2800a9ae0 msgr2=0x7fc2800a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 1 -- 172.21.15.171:0/2896952130 shutdown_connections 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.272+0000 7fc287d31640 1 -- 172.21.15.171:0/2896952130 wait complete. 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient: build_initial_monmap 2024-09-16T04:08:15.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient: monmap: 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 1 Processor -- start 2024-09-16T04:08:15.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 1 -- start start 2024-09-16T04:08:15.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient: init 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient(hunting): picked mon.noname-c con 0x7fc280136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient(hunting): picked mon.noname-b con 0x7fc280137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.273+0000 7fc287d31640 10 monclient(hunting): picked mon.noname-a con 0x7fc2801375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28013a510 con 0x7fc2801375a0 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc280138890 con 0x7fc280137140 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc280139340 con 0x7fc280136d60 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:15.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 10 monclient(hunting): authenticate will time out at 869.779027s 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2862ce640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc2801375a0 legacy=0x7fc2801d3a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2852cc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc280137140 0x7fc280138180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52460/0 (socket says 172.21.15.171:52460) 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2852cc640 1 -- 172.21.15.171:0/2803372015 learned_addr learned my addr 172.21.15.171:0/2803372015 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545480959 0 0) 0x7fc280138890 con 0x7fc280137140 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): my global_id is 4705 2024-09-16T04:08:15.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc264003550 con 0x7fc280137140 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687727619 0 0) 0x7fc280139340 con 0x7fc280136d60 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): my global_id is 4733 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc280138890 con 0x7fc280136d60 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1209959357 0 0) 0x7fc264003550 con 0x7fc280137140 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc280139340 con 0x7fc280137140 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2700015d0 con 0x7fc280137140 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): dump: 2024-09-16T04:08:15.080 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.081 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4146922981 0 0) 0x7fc280139340 con 0x7fc280137140 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 >> v1:172.21.15.148:6790/0 conn(0x7fc280136d60 legacy=0x7fc280137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 >> v1:172.21.15.148:6789/0 conn(0x7fc2801375a0 legacy=0x7fc2801d3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 monclient: found mon.b 2024-09-16T04:08:15.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc280138c80 con 0x7fc280137140 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 5 monclient: authenticate success, global_id 4705 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 10 monclient: _renew_subs 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc28013a7f0 con 0x7fc280137140 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 10 monclient: _renew_subs 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc28013f920 con 0x7fc280137140 2024-09-16T04:08:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc270001900 con 0x7fc280137140 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc270004e60 con 0x7fc280137140 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc277fff640 4 set_mon_vals no callback set 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.274+0000 7fc2767fc640 10 monclient: dump: 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc270005080 con 0x7fc280137140 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc287d31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc287d31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc287d31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc268005080 con 0x7fc280137140 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(42..42 src has 1..42) ==== 3598+0+0 (unknown 3551095731 0 0) 0x7fc270046fa0 con 0x7fc280137140 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.275+0000 7fc285acd640 10 monclient: get_auth_request con 0x7fc264038bc0 auth_method 0 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.280+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc270015f60 con 0x7fc280137140 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.280+0000 7fc2767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.280+0000 7fc2767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.451+0000 7fc287d31640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.451+0000 7fc287d31640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.451+0000 7fc287d31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:14.451+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7fc268005370 con 0x7fc280137140 2024-09-16T04:08:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.192+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v43) ==== 126+0+0 (unknown 2269662458 0 0) 0x7fc270015860 con 0x7fc280137140 2024-09-16T04:08:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.192+0000 7fc2767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.192+0000 7fc2767fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.304+0000 7fc287d31640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.304+0000 7fc287d31640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.304+0000 7fc287d31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.304+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7fc268001140 con 0x7fc280137140 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.305+0000 7fc2767fc640 1 -- 172.21.15.171:0/2803372015 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v43) ==== 149+0+0 (unknown 4013494279 0 0) 0x7fc270013bc0 con 0x7fc280137140 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.305+0000 7fc2767fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.305+0000 7fc2767fc640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T04:08:15.307 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo already exists 2024-09-16T04:08:15.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 >> v1:172.21.15.171:6812/7327273 conn(0x7fc264038bc0 legacy=0x7fc26403b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 10 monclient: shutdown 2024-09-16T04:08:15.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:15.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 >> v1:172.21.15.171:6789/0 conn(0x7fc280137140 legacy=0x7fc280138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 shutdown_connections 2024-09-16T04:08:15.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 >> 172.21.15.171:0/2803372015 conn(0x7fc2800a9ae0 msgr2=0x7fc2800a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:15.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 shutdown_connections 2024-09-16T04:08:15.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.307+0000 7fc287d31640 1 -- 172.21.15.171:0/2803372015 wait complete. 2024-09-16T04:08:15.318 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-16T04:08:15.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.400+0000 7fea17a8b640 10 monclient: get_monmap_and_config 2024-09-16T04:08:15.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.400+0000 7fea17a8b640 10 monclient: build_initial_monmap 2024-09-16T04:08:15.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.400+0000 7fea17a8b640 10 monclient: monmap: 2024-09-16T04:08:15.402 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:15.402 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.402 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:15.401366+0000 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:15.401366+0000 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:15.403 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 1 Processor -- start 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 1 -- start start 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient: init 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient(hunting): picked mon.noname-a con 0x7fea10136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient(hunting): picked mon.noname-b con 0x7fea10137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient(hunting): picked mon.noname-c con 0x7fea101375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1009b2c0 con 0x7fea10136d60 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1009b5a0 con 0x7fea101375a0 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea100a6640 con 0x7fea10137140 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea17a8b640 10 monclient(hunting): authenticate will time out at 870.906832s 2024-09-16T04:08:15.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea15827640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fea10136d60 legacy=0x7fea1009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea16028640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fea101375a0 0x7fea1013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52470/0 (socket says 172.21.15.171:52470) 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.401+0000 7fea16028640 1 -- 172.21.15.171:0/3921653679 learned_addr learned my addr 172.21.15.171:0/3921653679 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948808699 0 0) 0x7fea1009b5a0 con 0x7fea101375a0 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient(hunting): my global_id is 4708 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9fc003520 con 0x7fea101375a0 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 261537300 0 0) 0x7fe9fc003520 con 0x7fea101375a0 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 >> v1:172.21.15.148:6790/0 conn(0x7fea10137140 legacy=0x7fea1013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 >> v1:172.21.15.148:6789/0 conn(0x7fea10136d60 legacy=0x7fea1009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: _finish_hunting 0 2024-09-16T04:08:15.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 monclient: found mon.noname-c 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea10143b20 con 0x7fea101375a0 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: _finish_auth 0 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: _check_auth_tickets 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fea08001990 con 0x7fea101375a0 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 5 monclient: authenticate success, global_id 4708 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: dump: 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea08001c00 con 0x7fea101375a0 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 1 -- 172.21.15.171:0/3921653679 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fea08001fd0 con 0x7fea101375a0 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 4 set_mon_vals no callback set 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea14825640 10 monclient: dump: 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.416 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 10 monclient: shutdown 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.402+0000 7fea17a8b640 1 -- 172.21.15.171:0/3921653679 >> v1:172.21.15.171:6789/0 conn(0x7fea101375a0 legacy=0x7fea1013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 1 -- 172.21.15.171:0/3921653679 shutdown_connections 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 1 -- 172.21.15.171:0/3921653679 >> 172.21.15.171:0/3921653679 conn(0x7fea100a9ae0 msgr2=0x7fea100a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 1 -- 172.21.15.171:0/3921653679 shutdown_connections 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 1 -- 172.21.15.171:0/3921653679 wait complete. 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 10 monclient: build_initial_monmap 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 10 monclient: monmap: 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:15.417 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.403+0000 7fea17a8b640 1 Processor -- start 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 1 -- start start 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient: init 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient(hunting): picked mon.noname-a con 0x7fea10136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient(hunting): picked mon.noname-c con 0x7fea10137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient(hunting): picked mon.noname-b con 0x7fea101375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:15.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea10139470 con 0x7fea10136d60 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea10139750 con 0x7fea101375a0 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1009f170 con 0x7fea10137140 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea17a8b640 10 monclient(hunting): authenticate will time out at 870.909468s 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea15827640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fea10136d60 legacy=0x7fea1009e350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea16028640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fea101375a0 0x7fea101d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52486/0 (socket says 172.21.15.171:52486) 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea16028640 1 -- 172.21.15.171:0/265910167 learned_addr learned my addr 172.21.15.171:0/265910167 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532459482 0 0) 0x7fea10139750 con 0x7fea101375a0 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): my global_id is 4711 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9f4003520 con 0x7fea101375a0 2024-09-16T04:08:15.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526385545 0 0) 0x7fea1009f170 con 0x7fea10137140 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): my global_id is 4739 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea10139750 con 0x7fea10137140 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3410091448 0 0) 0x7fe9f4003520 con 0x7fea101375a0 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea1009f170 con 0x7fea101375a0 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fea08001890 con 0x7fea101375a0 2024-09-16T04:08:15.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): dump: 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.404+0000 7fea067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2150944674 0 0) 0x7fea10139750 con 0x7fea10137140 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9f4003520 con 0x7fea10137140 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fea00001830 con 0x7fea10137140 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient(hunting): dump: 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.422 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.423 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.423 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.423 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.423 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.423 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.423 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4186449060 0 0) 0x7fea1009f170 con 0x7fea101375a0 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 >> v1:172.21.15.148:6790/0 conn(0x7fea10137140 legacy=0x7fea1009ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 >> v1:172.21.15.148:6789/0 conn(0x7fea10136d60 legacy=0x7fea1009e350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 monclient: found mon.b 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea100a8160 con 0x7fea101375a0 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 5 monclient: authenticate success, global_id 4711 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 10 monclient: _renew_subs 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fea1013f6d0 con 0x7fea101375a0 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 10 monclient: _renew_subs 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fea10139190 con 0x7fea101375a0 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea08004f70 con 0x7fea101375a0 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fea080052c0 con 0x7fea101375a0 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea067fc640 10 monclient: dump: 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.405+0000 7fea07fff640 4 set_mon_vals no callback set 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fea08005460 con 0x7fea101375a0 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea17a8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea17a8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea17a8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea100008d0 con 0x7fea101375a0 2024-09-16T04:08:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(43..43 src has 1..43) ==== 3657+0+0 (unknown 2151068073 0 0) 0x7fea08047120 con 0x7fea101375a0 2024-09-16T04:08:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.406+0000 7fea15827640 10 monclient: get_auth_request con 0x7fe9f4038ef0 auth_method 0 2024-09-16T04:08:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.411+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fea08016040 con 0x7fea101375a0 2024-09-16T04:08:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.411+0000 7fea067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.411+0000 7fea067fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea17a8b640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea17a8b640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea17a8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7fea1013f910 con 0x7fea101375a0 2024-09-16T04:08:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v43) ==== 149+0+0 (unknown 4013494279 0 0) 0x7fea08015940 con 0x7fea101375a0 2024-09-16T04:08:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.581+0000 7fea067fc640 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-16T04:08:15.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea17a8b640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea17a8b640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea17a8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) -- 0x7fea101362e0 con 0x7fea101375a0 2024-09-16T04:08:15.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea067fc640 1 -- 172.21.15.171:0/265910167 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v43) ==== 149+0+0 (unknown 4013494279 0 0) 0x7fea0801e4f0 con 0x7fea101375a0 2024-09-16T04:08:15.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea067fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-16T04:08:15.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.684+0000 7fea067fc640 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-16T04:08:15.687 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo already exists 2024-09-16T04:08:15.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 >> v1:172.21.15.171:6812/7327273 conn(0x7fe9f4038ef0 legacy=0x7fe9f403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 10 monclient: shutdown 2024-09-16T04:08:15.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:15.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 >> v1:172.21.15.171:6789/0 conn(0x7fea101375a0 legacy=0x7fea101d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 shutdown_connections 2024-09-16T04:08:15.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 >> 172.21.15.171:0/265910167 conn(0x7fea100a9ae0 msgr2=0x7fea100a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:15.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 shutdown_connections 2024-09-16T04:08:15.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.686+0000 7fea17a8b640 1 -- 172.21.15.171:0/265910167 wait complete. 2024-09-16T04:08:15.697 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-16T04:08:15.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.776+0000 7fa3c522c640 10 monclient: get_monmap_and_config 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.776+0000 7fa3c522c640 10 monclient: build_initial_monmap 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.777+0000 7fa3c522c640 10 monclient: monmap: 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:15.777788+0000 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:15.777788+0000 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:15.779 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.780 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:15.780 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:15.780 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:15.780 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.777+0000 7fa3c522c640 1 Processor -- start 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.777+0000 7fa3c522c640 1 -- start start 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.777+0000 7fa3c522c640 10 monclient: init 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c00a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c009f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c013d2c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c00737b0 con 0x7fa3c009f320 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c006c6d0 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c006d980 con 0x7fa3c013d2c0 2024-09-16T04:08:15.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient(hunting): authenticate will time out at 871.283104s 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa3c00a0ae0 0x7fa3c00a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52500/0 (socket says 172.21.15.171:52500) 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bed9d640 1 -- 172.21.15.171:0/2206755896 learned_addr learned my addr 172.21.15.171:0/2206755896 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3be59c640 1 -- 172.21.15.171:0/2206755896 >> v1:172.21.15.148:6789/0 conn(0x7fa3c009f320 legacy=0x7fa3c013bba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3643735775 0 0) 0x7fa3c006c6d0 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): my global_id is 4714 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a8003520 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410864975 0 0) 0x7fa3c006d980 con 0x7fa3c013d2c0 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient(hunting): my global_id is 4742 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3c006c6d0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3914467708 0 0) 0x7fa3a8003520 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 >> v1:172.21.15.148:6790/0 conn(0x7fa3c013d2c0 legacy=0x7fa3c013f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 >> v1:172.21.15.148:6789/0 conn(0x7fa3c009f320 legacy=0x7fa3c013bba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 monclient: found mon.noname-c 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c006c950 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: _finish_auth 0 2024-09-16T04:08:15.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3b0001610 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 5 monclient: authenticate success, global_id 4714 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: dump: 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.793 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3b0001930 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 1 -- 172.21.15.171:0/2206755896 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3b0001d00 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3bdd9b640 10 monclient: dump: 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.794 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.778+0000 7fa3c522c640 4 set_mon_vals no callback set 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 10 monclient: shutdown 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 1 -- 172.21.15.171:0/2206755896 >> v1:172.21.15.171:6789/0 conn(0x7fa3c00a0ae0 legacy=0x7fa3c00a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 1 -- 172.21.15.171:0/2206755896 shutdown_connections 2024-09-16T04:08:15.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 1 -- 172.21.15.171:0/2206755896 >> 172.21.15.171:0/2206755896 conn(0x7fa3c00a77f0 msgr2=0x7fa3c00a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 1 -- 172.21.15.171:0/2206755896 shutdown_connections 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 1 -- 172.21.15.171:0/2206755896 wait complete. 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 10 monclient: build_initial_monmap 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.779+0000 7fa3c522c640 10 monclient: monmap: 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:15.796 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 1 Processor -- start 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 1 -- start start 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient: init 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c009f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c00a0ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c013d2c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c0123cf0 con 0x7fa3c009f320 2024-09-16T04:08:15.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c0122840 con 0x7fa3c013d2c0 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c01242b0 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3c522c640 10 monclient(hunting): authenticate will time out at 871.285957s 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.780+0000 7fa3bed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa3c009f320 legacy=0x7fa3c0121a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3bf59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa3c013d2c0 0x7fa3c01d3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52506/0 (socket says 172.21.15.171:52506) 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3bf59e640 1 -- 172.21.15.171:0/2725666366 learned_addr learned my addr 172.21.15.171:0/2725666366 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860091219 0 0) 0x7fa3c0122840 con 0x7fa3c013d2c0 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): my global_id is 4717 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3ac003520 con 0x7fa3c013d2c0 2024-09-16T04:08:15.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591616664 0 0) 0x7fa3c01242b0 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): my global_id is 4745 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3c0122840 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3414770449 0 0) 0x7fa3ac003520 con 0x7fa3c013d2c0 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3c01242b0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3b8001c50 con 0x7fa3c013d2c0 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): dump: 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.799 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3394276244 0 0) 0x7fa3c0122840 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3ac003520 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3b40040f0 con 0x7fa3c00a0ae0 2024-09-16T04:08:15.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient(hunting): dump: 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 145932222 0 0) 0x7fa3c01242b0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 >> v1:172.21.15.148:6790/0 conn(0x7fa3c00a0ae0 legacy=0x7fa3c0122130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 >> v1:172.21.15.148:6789/0 conn(0x7fa3c009f320 legacy=0x7fa3c0121a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 monclient: found mon.b 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c0122e00 con 0x7fa3c013d2c0 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 5 monclient: authenticate success, global_id 4717 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 10 monclient: _renew_subs 2024-09-16T04:08:15.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c013c6d0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 10 monclient: _renew_subs 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3c013bcb0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3b8002020 con 0x7fa3c013d2c0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3b8003de0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3a77fe640 10 monclient: dump: 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.803 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.781+0000 7fa3bd59a640 4 set_mon_vals no callback set 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa3b8003f80 con 0x7fa3c013d2c0 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3c522c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3c522c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3c522c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a0005080 con 0x7fa3c013d2c0 2024-09-16T04:08:15.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(43..43 src has 1..43) ==== 3657+0+0 (unknown 2151068073 0 0) 0x7fa3b80470f0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.782+0000 7fa3bed9d640 10 monclient: get_auth_request con 0x7fa3ac038ef0 auth_method 0 2024-09-16T04:08:15.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.787+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa3b8015fd0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.787+0000 7fa3a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:15.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.787+0000 7fa3a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:15.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.953+0000 7fa3c522c640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T04:08:15.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.953+0000 7fa3c522c640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T04:08:15.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.953+0000 7fa3c522c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:15.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.953+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) -- 0x7fa3a0005370 con 0x7fa3c013d2c0 2024-09-16T04:08:15.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.977+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v44) ==== 126+0+0 (unknown 3441966178 0 0) 0x7fa3b80158d0 con 0x7fa3c013d2c0 2024-09-16T04:08:15.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.977+0000 7fa3a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T04:08:15.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:15.977+0000 7fa3a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:16.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3c522c640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T04:08:16.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3c522c640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T04:08:16.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3c522c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) -- 0x7fa3a0001140 con 0x7fa3c013d2c0 2024-09-16T04:08:16.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3a77fe640 1 -- 172.21.15.171:0/2725666366 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v44) ==== 149+0+0 (unknown 145563413 0 0) 0x7fa3b801e480 con 0x7fa3c013d2c0 2024-09-16T04:08:16.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3a77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-16T04:08:16.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.080+0000 7fa3a77fe640 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-16T04:08:16.083 INFO:tasks.workunit.client.0.smithi171.stderr:rule bar already exists 2024-09-16T04:08:16.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 >> v1:172.21.15.171:6812/7327273 conn(0x7fa3ac038ef0 legacy=0x7fa3ac03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:16.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 10 monclient: shutdown 2024-09-16T04:08:16.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:16.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 >> v1:172.21.15.171:6789/0 conn(0x7fa3c013d2c0 legacy=0x7fa3c01d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:16.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 shutdown_connections 2024-09-16T04:08:16.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 >> 172.21.15.171:0/2725666366 conn(0x7fa3c00a77f0 msgr2=0x7fa3c009d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:16.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 shutdown_connections 2024-09-16T04:08:16.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.082+0000 7fa3c522c640 1 -- 172.21.15.171:0/2725666366 wait complete. 2024-09-16T04:08:16.093 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm-device-class all 2024-09-16T04:08:16.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.172+0000 7f3e675e4640 10 monclient: get_monmap_and_config 2024-09-16T04:08:16.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.172+0000 7f3e675e4640 10 monclient: build_initial_monmap 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.172+0000 7f3e675e4640 10 monclient: monmap: 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:16.173170+0000 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:16.173170+0000 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:16.176 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:16.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.172+0000 7f3e675e4640 1 Processor -- start 2024-09-16T04:08:16.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 1 -- start start 2024-09-16T04:08:16.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient: init 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient(hunting): picked mon.noname-c con 0x7f3e60136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient(hunting): picked mon.noname-b con 0x7f3e60139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient(hunting): picked mon.noname-a con 0x7f3e6013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e600a1340 con 0x7f3e6013d620 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e60137650 con 0x7f3e60136b60 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e600a0aa0 con 0x7f3e60139b10 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:16.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e675e4640 10 monclient(hunting): authenticate will time out at 871.678639s 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e65b81640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f3e6013d620 legacy=0x7f3e6013fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e65380640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3e60136b60 0x7f3e60136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52518/0 (socket says 172.21.15.171:52518) 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e65380640 1 -- 172.21.15.171:0/1283646083 learned_addr learned my addr 172.21.15.171:0/1283646083 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3082894998 0 0) 0x7f3e60137650 con 0x7f3e60136b60 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e57fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e57fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e57fff640 10 monclient(hunting): my global_id is 4720 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.173+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e4c003520 con 0x7f3e60136b60 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2540155688 0 0) 0x7f3e600a0aa0 con 0x7f3e60139b10 2024-09-16T04:08:16.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient(hunting): my global_id is 4748 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e60137650 con 0x7f3e60139b10 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3979053394 0 0) 0x7f3e4c003520 con 0x7f3e60136b60 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 >> v1:172.21.15.148:6790/0 conn(0x7f3e60139b10 legacy=0x7f3e6013bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 >> v1:172.21.15.148:6789/0 conn(0x7f3e6013d620 legacy=0x7f3e6013fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 monclient: found mon.noname-c 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e600a0550 con 0x7f3e60136b60 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3e5c001610 con 0x7f3e60136b60 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 5 monclient: authenticate success, global_id 4720 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: dump: 2024-09-16T04:08:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:16.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e5c001930 con 0x7f3e60136b60 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 4 set_mon_vals no callback set 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 1 -- 172.21.15.171:0/1283646083 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3e5c001d00 con 0x7f3e60136b60 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e57fff640 10 monclient: dump: 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:16.190 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 10 monclient: shutdown 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 1 -- 172.21.15.171:0/1283646083 >> v1:172.21.15.171:6789/0 conn(0x7f3e60136b60 legacy=0x7f3e60136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 1 -- 172.21.15.171:0/1283646083 shutdown_connections 2024-09-16T04:08:16.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 1 -- 172.21.15.171:0/1283646083 >> 172.21.15.171:0/1283646083 conn(0x7f3e600a98b0 msgr2=0x7f3e600a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 1 -- 172.21.15.171:0/1283646083 shutdown_connections 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.174+0000 7f3e675e4640 1 -- 172.21.15.171:0/1283646083 wait complete. 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.175+0000 7f3e675e4640 10 monclient: build_initial_monmap 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.175+0000 7f3e675e4640 10 monclient: monmap: 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:16.192 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.175+0000 7f3e675e4640 1 Processor -- start 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.175+0000 7f3e675e4640 1 -- start start 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.175+0000 7f3e675e4640 10 monclient: init 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient(hunting): picked mon.noname-c con 0x7f3e60136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient(hunting): picked mon.noname-b con 0x7f3e60139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:16.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient(hunting): picked mon.noname-a con 0x7f3e6013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e60120990 con 0x7f3e6013d620 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e6009f4d0 con 0x7f3e60139b10 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e600a6f50 con 0x7f3e60136b60 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient(hunting): authenticate will time out at 871.681280s 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e65b81640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f3e6013d620 legacy=0x7f3e601d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e64b7f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3e60139b10 0x7f3e601d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52522/0 (socket says 172.21.15.171:52522) 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e64b7f640 1 -- 172.21.15.171:0/3705464607 learned_addr learned my addr 172.21.15.171:0/3705464607 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917193529 0 0) 0x7f3e6009f4d0 con 0x7f3e60139b10 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:16.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): my global_id is 4723 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e44003550 con 0x7f3e60139b10 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 224862419 0 0) 0x7f3e600a6f50 con 0x7f3e60136b60 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): my global_id is 4751 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e6009f4d0 con 0x7f3e60136b60 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 733667638 0 0) 0x7f3e44003550 con 0x7f3e60139b10 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3e600a6f50 con 0x7f3e60139b10 2024-09-16T04:08:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3e50001970 con 0x7f3e60139b10 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1658406240 0 0) 0x7f3e600a6f50 con 0x7f3e60139b10 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 >> v1:172.21.15.148:6790/0 conn(0x7f3e60136b60 legacy=0x7f3e601cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 >> v1:172.21.15.148:6789/0 conn(0x7f3e6013d620 legacy=0x7f3e601d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 monclient: found mon.b 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e600a8b00 con 0x7f3e60139b10 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 5 monclient: authenticate success, global_id 4723 2024-09-16T04:08:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient: _renew_subs 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3e600a0240 con 0x7f3e60139b10 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e50001c50 con 0x7f3e60139b10 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.176+0000 7f3e55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3e50003a60 con 0x7f3e60139b10 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e55ffb640 10 monclient: dump: 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e577fe640 4 set_mon_vals no callback set 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e675e4640 10 monclient: _renew_subs 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e675e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3e601206b0 con 0x7f3e60139b10 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.177+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3e50003c80 con 0x7f3e60139b10 2024-09-16T04:08:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.178+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(44..44 src has 1..44) ==== 3716+0+0 (unknown 2578529648 0 0) 0x7f3e50046e80 con 0x7f3e60139b10 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.178+0000 7f3e65380640 10 monclient: get_auth_request con 0x7f3e44039010 auth_method 0 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.178+0000 7f3e675e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.178+0000 7f3e675e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.178+0000 7f3e675e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.178+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e60120d30 con 0x7f3e60139b10 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.182+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3e50015d00 con 0x7f3e60139b10 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.182+0000 7f3e55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.182+0000 7f3e55ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:16.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.355+0000 7f3e675e4640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T04:08:16.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.355+0000 7f3e675e4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T04:08:16.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.355+0000 7f3e675e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:16.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:16.355+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) -- 0x7f3e600a4820 con 0x7f3e60139b10 2024-09-16T04:08:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.206+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v45) ==== 131+0+0 (unknown 2832017534 0 0) 0x7f3e50015600 con 0x7f3e60139b10 2024-09-16T04:08:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.206+0000 7f3e55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T04:08:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.206+0000 7f3e55ffb640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-16T04:08:17.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.313+0000 7f3e675e4640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T04:08:17.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.313+0000 7f3e675e4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T04:08:17.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.313+0000 7f3e675e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:17.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:17.313+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) -- 0x7f3e600a4ca0 con 0x7f3e60139b10 2024-09-16T04:08:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.210+0000 7f3e55ffb640 1 -- 172.21.15.171:0/3705464607 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v46) ==== 282+0+0 (unknown 1309378230 0 0) 0x7f3e5001e1b0 con 0x7f3e60139b10 2024-09-16T04:08:18.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.210+0000 7f3e55ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-16T04:08:18.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.210+0000 7f3e55ffb640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T04:08:18.213 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-16T04:08:18.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.214+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 >> v1:172.21.15.171:6812/7327273 conn(0x7f3e44039010 legacy=0x7f3e4403b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.214+0000 7f3e675e4640 10 monclient: shutdown 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.214+0000 7f3e675e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.214+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 >> v1:172.21.15.171:6789/0 conn(0x7f3e60139b10 legacy=0x7f3e601d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.214+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 shutdown_connections 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.214+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 >> 172.21.15.171:0/3705464607 conn(0x7f3e600a98b0 msgr2=0x7f3e600a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.215+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 shutdown_connections 2024-09-16T04:08:18.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.215+0000 7f3e675e4640 1 -- 172.21.15.171:0/3705464607 wait complete. 2024-09-16T04:08:18.228 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.311+0000 7fecf7f86640 10 monclient: get_monmap_and_config 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.311+0000 7fecf7f86640 10 monclient: build_initial_monmap 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.311+0000 7fecf7f86640 10 monclient: monmap: 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:18.311922+0000 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:18.311922+0000 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:18.313 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:18.314 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:18.314 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:18.314 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:18.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.311+0000 7fecf7f86640 1 Processor -- start 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.311+0000 7fecf7f86640 1 -- start start 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.311+0000 7fecf7f86640 10 monclient: init 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient(hunting): picked mon.noname-b con 0x7fecf0136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient(hunting): picked mon.noname-c con 0x7fecf0139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient(hunting): picked mon.noname-a con 0x7fecf013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf00a1340 con 0x7fecf013d620 2024-09-16T04:08:18.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf0137650 con 0x7fecf0139b10 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf00a0aa0 con 0x7fecf0136b60 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf7f86640 10 monclient(hunting): authenticate will time out at 873.817504s 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf5521640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fecf0139b10 0x7fecf013bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:53936/0 (socket says 172.21.15.171:53936) 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf5521640 1 -- 172.21.15.171:0/135203956 learned_addr learned my addr 172.21.15.171:0/135203956 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf6523640 1 -- 172.21.15.171:0/135203956 >> v1:172.21.15.148:6789/0 conn(0x7fecf013d620 legacy=0x7fecf013fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343516021 0 0) 0x7fecf0137650 con 0x7fecf0139b10 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf4d20640 10 monclient(hunting): _negotiate 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf4d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:18.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf4d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf4d20640 10 monclient(hunting): my global_id is 4726 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.312+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecdc003520 con 0x7fecf0139b10 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2477589879 0 0) 0x7fecf00a0aa0 con 0x7fecf0136b60 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient(hunting): _negotiate 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient(hunting): my global_id is 4754 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf0137650 con 0x7fecf0136b60 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2786380808 0 0) 0x7fecdc003520 con 0x7fecf0139b10 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 >> v1:172.21.15.148:6790/0 conn(0x7fecf0136b60 legacy=0x7fecf0136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:18.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 >> v1:172.21.15.148:6789/0 conn(0x7fecf013d620 legacy=0x7fecf013fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: _finish_hunting 0 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 monclient: found mon.noname-c 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf00a0550 con 0x7fecf0139b10 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: _finish_auth 0 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: _check_auth_tickets 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fece0001610 con 0x7fecf0139b10 2024-09-16T04:08:18.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: dump: 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:18.327 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 5 monclient: authenticate success, global_id 4726 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fece0001870 con 0x7fecf0139b10 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 1 -- 172.21.15.171:0/135203956 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fece0001c40 con 0x7fecf0139b10 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 4 set_mon_vals no callback set 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:18.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf4d20640 10 monclient: dump: 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:18.329 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 10 monclient: shutdown 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 1 -- 172.21.15.171:0/135203956 >> v1:172.21.15.171:6789/0 conn(0x7fecf0139b10 legacy=0x7fecf013bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 1 -- 172.21.15.171:0/135203956 shutdown_connections 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 1 -- 172.21.15.171:0/135203956 >> 172.21.15.171:0/135203956 conn(0x7fecf00a98b0 msgr2=0x7fecf00a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 1 -- 172.21.15.171:0/135203956 shutdown_connections 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.313+0000 7fecf7f86640 1 -- 172.21.15.171:0/135203956 wait complete. 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.314+0000 7fecf7f86640 10 monclient: build_initial_monmap 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.314+0000 7fecf7f86640 10 monclient: monmap: 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.314+0000 7fecf7f86640 1 Processor -- start 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.314+0000 7fecf7f86640 1 -- start start 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.314+0000 7fecf7f86640 10 monclient: init 2024-09-16T04:08:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient(hunting): picked mon.noname-b con 0x7fecf0136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient(hunting): picked mon.noname-c con 0x7fecf0139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient(hunting): picked mon.noname-a con 0x7fecf013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf0120a30 con 0x7fecf013d620 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf01cc5c0 con 0x7fecf0136b60 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf00a0240 con 0x7fecf0139b10 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient(hunting): authenticate will time out at 873.820289s 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf5d22640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fecf0136b60 0x7fecf01cb7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:53950/0 (socket says 172.21.15.171:53950) 2024-09-16T04:08:18.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf5d22640 1 -- 172.21.15.171:0/2055993301 learned_addr learned my addr 172.21.15.171:0/2055993301 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf6523640 1 -- 172.21.15.171:0/2055993301 >> v1:172.21.15.148:6789/0 conn(0x7fecf013d620 legacy=0x7fecf01d3a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 149657969 0 0) 0x7fecf01cc5c0 con 0x7fecf0136b60 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): my global_id is 4729 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd4003550 con 0x7fecf0136b60 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989260523 0 0) 0x7fecf00a0240 con 0x7fecf0139b10 2024-09-16T04:08:18.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): my global_id is 4757 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf01cc5c0 con 0x7fecf0139b10 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2494869370 0 0) 0x7fecd4003550 con 0x7fecf0136b60 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fecf00a0240 con 0x7fecf0136b60 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fecec001a00 con 0x7fecf0136b60 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): dump: 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:18.334 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 601060223 0 0) 0x7fecf00a0240 con 0x7fecf0136b60 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 >> v1:172.21.15.148:6790/0 conn(0x7fecf0139b10 legacy=0x7fecf01cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 >> v1:172.21.15.148:6789/0 conn(0x7fecf013d620 legacy=0x7fecf01d3a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:18.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 monclient: found mon.b 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf00a6f50 con 0x7fecf0136b60 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fece67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 5 monclient: authenticate success, global_id 4729 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient: _renew_subs 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.315+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fecf00a2320 con 0x7fecf0136b60 2024-09-16T04:08:18.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fecf7f86640 10 monclient: _renew_subs 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fecf7f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fecf0120750 con 0x7fecf0136b60 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fecec0038c0 con 0x7fecf0136b60 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fecec003c90 con 0x7fecf0136b60 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece7fff640 4 set_mon_vals no callback set 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.316+0000 7fece67fc640 10 monclient: dump: 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:18.337 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fecf7f86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fecf7f86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fecf7f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fecec003e30 con 0x7fecf0136b60 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd8005080 con 0x7fecf0136b60 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(46..46 src has 1..46) ==== 3412+0+0 (unknown 2545646586 0 0) 0x7fecec046cf0 con 0x7fecf0136b60 2024-09-16T04:08:18.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.317+0000 7fecf5521640 10 monclient: get_auth_request con 0x7fecd4038c40 auth_method 0 2024-09-16T04:08:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.321+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fecec015e00 con 0x7fecf0136b60 2024-09-16T04:08:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.321+0000 7fece67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.321+0000 7fece67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:18.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.495+0000 7fecf7f86640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T04:08:18.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.495+0000 7fecf7f86640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T04:08:18.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.495+0000 7fecf7f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:18.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:18.495+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) -- 0x7fecd8005370 con 0x7fecf0136b60 2024-09-16T04:08:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.212+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v47) ==== 135+0+0 (unknown 2203644516 0 0) 0x7fecec015700 con 0x7fecf0136b60 2024-09-16T04:08:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.212+0000 7fece67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T04:08:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.212+0000 7fece67fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-16T04:08:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.319+0000 7fecf7f86640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T04:08:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.319+0000 7fecf7f86640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T04:08:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.319+0000 7fecf7f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:19.319+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) -- 0x7fecd8000ab0 con 0x7fecf0136b60 2024-09-16T04:08:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.225+0000 7fece67fc640 1 -- 172.21.15.171:0/2055993301 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v48) ==== 237+0+0 (unknown 911157329 0 0) 0x7fecec01e290 con 0x7fecf0136b60 2024-09-16T04:08:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.225+0000 7fece67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-16T04:08:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.225+0000 7fece67fc640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T04:08:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-16T04:08:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 >> v1:172.21.15.171:6812/7327273 conn(0x7fecd4038c40 legacy=0x7fecd403b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 10 monclient: shutdown 2024-09-16T04:08:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 >> v1:172.21.15.171:6789/0 conn(0x7fecf0136b60 legacy=0x7fecf01cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 shutdown_connections 2024-09-16T04:08:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 >> 172.21.15.171:0/2055993301 conn(0x7fecf00a98b0 msgr2=0x7fecf01393f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 shutdown_connections 2024-09-16T04:08:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.228+0000 7fecf7f86640 1 -- 172.21.15.171:0/2055993301 wait complete. 2024-09-16T04:08:20.242 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-16T04:08:20.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.325+0000 7fa5f3a7b640 10 monclient: get_monmap_and_config 2024-09-16T04:08:20.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.325+0000 7fa5f3a7b640 10 monclient: build_initial_monmap 2024-09-16T04:08:20.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.325+0000 7fa5f3a7b640 10 monclient: monmap: 2024-09-16T04:08:20.327 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:20.327 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:20.326325+0000 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:20.326325+0000 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:20.328 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:20.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 1 Processor -- start 2024-09-16T04:08:20.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 1 -- start start 2024-09-16T04:08:20.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient: init 2024-09-16T04:08:20.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:20.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ec09eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ec134b20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ec134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ec09d060 con 0x7fa5ec134b20 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ec09d340 con 0x7fa5ec134f30 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ec068840 con 0x7fa5ec09eaa0 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.326+0000 7fa5f3a7b640 10 monclient(hunting): authenticate will time out at 875.831909s 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f1016640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa5ec134b20 legacy=0x7fa5ec139980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f2018640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa5ec134f30 0x7fa5ec13d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:53958/0 (socket says 172.21.15.171:53958) 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f2018640 1 -- 172.21.15.171:0/4285126394 learned_addr learned my addr 172.21.15.171:0/4285126394 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:20.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586871211 0 0) 0x7fa5ec09d340 con 0x7fa5ec134f30 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient(hunting): _negotiate 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient(hunting): my global_id is 4732 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d8003520 con 0x7fa5ec134f30 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2054018719 0 0) 0x7fa5d8003520 con 0x7fa5ec134f30 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 >> v1:172.21.15.148:6790/0 conn(0x7fa5ec09eaa0 legacy=0x7fa5ec09ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 >> v1:172.21.15.148:6789/0 conn(0x7fa5ec134b20 legacy=0x7fa5ec139980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: _finish_hunting 0 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 monclient: found mon.noname-c 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:20.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ec141800 con 0x7fa5ec134f30 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: _finish_auth 0 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: _check_auth_tickets 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa5e4001990 con 0x7fa5ec134f30 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: dump: 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.338 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f3a7b640 5 monclient: authenticate success, global_id 4732 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f3a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5e4001c80 con 0x7fa5ec134f30 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f3a7b640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 1 -- 172.21.15.171:0/4285126394 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa5e4002090 con 0x7fa5ec134f30 2024-09-16T04:08:20.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f3a7b640 4 set_mon_vals no callback set 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.327+0000 7fa5f0815640 10 monclient: dump: 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:20.340 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 10 monclient: shutdown 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/4285126394 >> v1:172.21.15.171:6789/0 conn(0x7fa5ec134f30 legacy=0x7fa5ec13d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/4285126394 shutdown_connections 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/4285126394 >> 172.21.15.171:0/4285126394 conn(0x7fa5ec0a1e90 msgr2=0x7fa5ec0a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/4285126394 shutdown_connections 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/4285126394 wait complete. 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 10 monclient: build_initial_monmap 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.328+0000 7fa5f3a7b640 10 monclient: monmap: 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:20.341 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 1 Processor -- start 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 1 -- start start 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient: init 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa5ec09eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:20.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa5ec134b20 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa5ec134f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ec13e090 con 0x7fa5ec134f30 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ec13c000 con 0x7fa5ec134b20 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5ec1414d0 con 0x7fa5ec09eaa0 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f3a7b640 10 monclient(hunting): authenticate will time out at 875.834611s 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f1016640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa5ec134b20 0x7fa5ec1d0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:53962/0 (socket says 172.21.15.171:53962) 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f1016640 1 -- 172.21.15.171:0/70675780 learned_addr learned my addr 172.21.15.171:0/70675780 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5f2018640 1 -- 172.21.15.171:0/70675780 >> v1:172.21.15.148:6789/0 conn(0x7fa5ec134f30 legacy=0x7fa5ec1d42b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262457323 0 0) 0x7fa5ec13c000 con 0x7fa5ec134b20 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:20.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): my global_id is 4735 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d0003520 con 0x7fa5ec134b20 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1525619721 0 0) 0x7fa5ec1414d0 con 0x7fa5ec09eaa0 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 10 monclient(hunting): my global_id is 4763 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.329+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5ec13c000 con 0x7fa5ec09eaa0 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 236981128 0 0) 0x7fa5d0003520 con 0x7fa5ec134b20 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5ec1414d0 con 0x7fa5ec134b20 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa5dc0041d0 con 0x7fa5ec134b20 2024-09-16T04:08:20.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient(hunting): dump: 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:20.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2190190556 0 0) 0x7fa5ec1414d0 con 0x7fa5ec134b20 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 >> v1:172.21.15.148:6790/0 conn(0x7fa5ec09eaa0 legacy=0x7fa5ec0982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 >> v1:172.21.15.148:6789/0 conn(0x7fa5ec134f30 legacy=0x7fa5ec1d42b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 monclient: found mon.b 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5ec13bd20 con 0x7fa5ec134b20 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 5 monclient: authenticate success, global_id 4735 2024-09-16T04:08:20.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 10 monclient: _renew_subs 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa5ec13ba40 con 0x7fa5ec134b20 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 10 monclient: _renew_subs 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5ec13a8b0 con 0x7fa5ec134b20 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5dc004480 con 0x7fa5ec134b20 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa5dc004850 con 0x7fa5ec134b20 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:20.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e27fc640 10 monclient: dump: 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.330+0000 7fa5e3fff640 4 set_mon_vals no callback set 2024-09-16T04:08:20.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa5dc004ab0 con 0x7fa5ec134b20 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(48..48 src has 1..48) ==== 3636+0+0 (unknown 243338553 0 0) 0x7fa5dc046da0 con 0x7fa5ec134b20 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5f3a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5f3a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5f3a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ec09f860 con 0x7fa5ec134b20 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.331+0000 7fa5f1817640 10 monclient: get_auth_request con 0x7fa5d0038f60 auth_method 0 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.335+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa5dc015cf0 con 0x7fa5ec134b20 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.335+0000 7fa5e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:20.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.335+0000 7fa5e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:20.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.505+0000 7fa5f3a7b640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T04:08:20.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.505+0000 7fa5f3a7b640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T04:08:20.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.505+0000 7fa5f3a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:20.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.505+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) -- 0x7fa5ec06c550 con 0x7fa5ec134b20 2024-09-16T04:08:20.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.979+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v49) ==== 135+0+0 (unknown 3857376242 0 0) 0x7fa5dc0199a0 con 0x7fa5ec134b20 2024-09-16T04:08:20.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.979+0000 7fa5e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T04:08:20.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:20.979+0000 7fa5e27fc640 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-16T04:08:21.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.085+0000 7fa5f3a7b640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T04:08:21.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.085+0000 7fa5f3a7b640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T04:08:21.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.085+0000 7fa5f3a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:21.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.085+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) -- 0x7fa5ec13e710 con 0x7fa5ec134b20 2024-09-16T04:08:21.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.984+0000 7fa5e27fc640 1 -- 172.21.15.171:0/70675780 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v50) ==== 237+0+0 (unknown 1931645736 0 0) 0x7fa5dc01e260 con 0x7fa5ec134b20 2024-09-16T04:08:21.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.984+0000 7fa5e27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-16T04:08:21.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.984+0000 7fa5e27fc640 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T04:08:21.986 INFO:tasks.workunit.client.0.smithi171.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-16T04:08:21.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 >> v1:172.21.15.171:6812/7327273 conn(0x7fa5d0038f60 legacy=0x7fa5d003b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:21.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 10 monclient: shutdown 2024-09-16T04:08:21.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:21.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 >> v1:172.21.15.171:6789/0 conn(0x7fa5ec134b20 legacy=0x7fa5ec1d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:21.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 shutdown_connections 2024-09-16T04:08:21.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 >> 172.21.15.171:0/70675780 conn(0x7fa5ec0a1e90 msgr2=0x7fa5ec1406b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:21.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 shutdown_connections 2024-09-16T04:08:21.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:21.987+0000 7fa5f3a7b640 1 -- 172.21.15.171:0/70675780 wait complete. 2024-09-16T04:08:22.001 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-16T04:08:22.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.087+0000 7fda9d8b7640 10 monclient: get_monmap_and_config 2024-09-16T04:08:22.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.087+0000 7fda9d8b7640 10 monclient: build_initial_monmap 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.087+0000 7fda9d8b7640 10 monclient: monmap: 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:22.088641+0000 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:22.088641+0000 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:22.090 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:22.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.088+0000 7fda9d8b7640 1 Processor -- start 2024-09-16T04:08:22.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.088+0000 7fda9d8b7640 1 -- start start 2024-09-16T04:08:22.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.088+0000 7fda9d8b7640 10 monclient: init 2024-09-16T04:08:22.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:22.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:22.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient(hunting): picked mon.noname-b con 0x7fda98136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient(hunting): picked mon.noname-c con 0x7fda98139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient(hunting): picked mon.noname-a con 0x7fda9813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda980a1340 con 0x7fda9813d620 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda98137650 con 0x7fda98139b10 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda980a0aa0 con 0x7fda98136b60 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient(hunting): authenticate will time out at 877.594155s 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda967fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fda98139b10 0x7fda9813bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:53976/0 (socket says 172.21.15.171:53976) 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda967fc640 1 -- 172.21.15.171:0/233123891 learned_addr learned my addr 172.21.15.171:0/233123891 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda977fe640 1 -- 172.21.15.171:0/233123891 >> v1:172.21.15.148:6789/0 conn(0x7fda9813d620 legacy=0x7fda9813fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898683210 0 0) 0x7fda98137650 con 0x7fda98139b10 2024-09-16T04:08:22.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient(hunting): my global_id is 4738 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda80003520 con 0x7fda98139b10 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4260008677 0 0) 0x7fda80003520 con 0x7fda98139b10 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 >> v1:172.21.15.148:6790/0 conn(0x7fda98136b60 legacy=0x7fda98136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 >> v1:172.21.15.148:6789/0 conn(0x7fda9813d620 legacy=0x7fda9813fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:22.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda980a0550 con 0x7fda98139b10 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda8c001610 con 0x7fda98139b10 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 5 monclient: authenticate success, global_id 4738 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: dump: 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.101 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda8c001970 con 0x7fda98139b10 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 1 -- 172.21.15.171:0/233123891 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda8c001d40 con 0x7fda98139b10 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda9d8b7640 4 set_mon_vals no callback set 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:22.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.089+0000 7fda95ffb640 10 monclient: dump: 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:22.103 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:22.104 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:22.104 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:22.104 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:22.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 10 monclient: shutdown 2024-09-16T04:08:22.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 1 -- 172.21.15.171:0/233123891 >> v1:172.21.15.171:6789/0 conn(0x7fda98139b10 legacy=0x7fda9813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 1 -- 172.21.15.171:0/233123891 shutdown_connections 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 1 -- 172.21.15.171:0/233123891 >> 172.21.15.171:0/233123891 conn(0x7fda980a98b0 msgr2=0x7fda980a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 1 -- 172.21.15.171:0/233123891 shutdown_connections 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 1 -- 172.21.15.171:0/233123891 wait complete. 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 10 monclient: build_initial_monmap 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.090+0000 7fda9d8b7640 10 monclient: monmap: 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:22.105 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 1 Processor -- start 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 1 -- start start 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient: init 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:22.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient(hunting): picked mon.noname-b con 0x7fda98136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient(hunting): picked mon.noname-a con 0x7fda98139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient(hunting): picked mon.noname-c con 0x7fda9813d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda98120920 con 0x7fda98139b10 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda9809f430 con 0x7fda98136b60 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda98120c00 con 0x7fda9813d620 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda9d8b7640 10 monclient(hunting): authenticate will time out at 877.596887s 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda967fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fda98139b10 legacy=0x7fda9809dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda96ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fda98136b60 0x7fda9809d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:53990/0 (socket says 172.21.15.171:53990) 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.091+0000 7fda96ffd640 1 -- 172.21.15.171:0/1262592668 learned_addr learned my addr 172.21.15.171:0/1262592668 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389944766 0 0) 0x7fda9809f430 con 0x7fda98136b60 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:22.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): my global_id is 4741 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda84003550 con 0x7fda98136b60 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174576425 0 0) 0x7fda98120c00 con 0x7fda9813d620 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): my global_id is 4769 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda9809f430 con 0x7fda9813d620 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1081226439 0 0) 0x7fda84003550 con 0x7fda98136b60 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda98120c00 con 0x7fda98136b60 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda88001cf0 con 0x7fda98136b60 2024-09-16T04:08:22.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): dump: 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:22.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3856298573 0 0) 0x7fda98120c00 con 0x7fda98136b60 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 >> v1:172.21.15.148:6790/0 conn(0x7fda9813d620 legacy=0x7fda981d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 >> v1:172.21.15.148:6789/0 conn(0x7fda98139b10 legacy=0x7fda9809dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 monclient: found mon.b 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda980a8b70 con 0x7fda98136b60 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 5 monclient: authenticate success, global_id 4741 2024-09-16T04:08:22.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 10 monclient: _renew_subs 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda980a0240 con 0x7fda98136b60 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 10 monclient: _renew_subs 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda9809e380 con 0x7fda98136b60 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda88003c40 con 0x7fda98136b60 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda88003e70 con 0x7fda98136b60 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda957fa640 4 set_mon_vals no callback set 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.092+0000 7fda7ffff640 10 monclient: dump: 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:22.111 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fda88004010 con 0x7fda98136b60 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda9d8b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda9d8b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:22.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda9d8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:22.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda78005080 con 0x7fda98136b60 2024-09-16T04:08:22.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(50..50 src has 1..50) ==== 3868+0+0 (unknown 1332482532 0 0) 0x7fda880471a0 con 0x7fda98136b60 2024-09-16T04:08:22.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.093+0000 7fda967fc640 10 monclient: get_auth_request con 0x7fda84038bf0 auth_method 0 2024-09-16T04:08:22.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.098+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fda880160f0 con 0x7fda98136b60 2024-09-16T04:08:22.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.098+0000 7fda7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:22.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.098+0000 7fda7ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:22.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.267+0000 7fda9d8b7640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T04:08:22.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.267+0000 7fda9d8b7640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T04:08:22.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.267+0000 7fda9d8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:22.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:22.267+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) -- 0x7fda78005370 con 0x7fda98136b60 2024-09-16T04:08:23.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.229+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v51) ==== 150+0+0 (unknown 1418137206 0 0) 0x7fda880159f0 con 0x7fda98136b60 2024-09-16T04:08:23.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.229+0000 7fda7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T04:08:23.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.229+0000 7fda7ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda9d8b7640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda9d8b7640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda9d8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) -- 0x7fda78001090 con 0x7fda98136b60 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda7ffff640 1 -- 172.21.15.171:0/1262592668 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v51) ==== 177+0+0 (unknown 1365241453 0 0) 0x7fda8801e5a0 con 0x7fda98136b60 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda7ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.335+0000 7fda7ffff640 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-16T04:08:23.337 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo-ssd already exists 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 >> v1:172.21.15.171:6812/7327273 conn(0x7fda84038bf0 legacy=0x7fda8403b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 10 monclient: shutdown 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 >> v1:172.21.15.171:6789/0 conn(0x7fda98136b60 legacy=0x7fda9809d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 shutdown_connections 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 >> 172.21.15.171:0/1262592668 conn(0x7fda980a98b0 msgr2=0x7fda98139380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:23.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 shutdown_connections 2024-09-16T04:08:23.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.337+0000 7fda9d8b7640 1 -- 172.21.15.171:0/1262592668 wait complete. 2024-09-16T04:08:23.348 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-16T04:08:23.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.429+0000 7f43704c9640 10 monclient: get_monmap_and_config 2024-09-16T04:08:23.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.429+0000 7f43704c9640 10 monclient: build_initial_monmap 2024-09-16T04:08:23.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.429+0000 7f43704c9640 10 monclient: monmap: 2024-09-16T04:08:23.431 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:23.430593+0000 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:23.430593+0000 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:23.432 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:23.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.430+0000 7f43704c9640 1 Processor -- start 2024-09-16T04:08:23.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.430+0000 7f43704c9640 1 -- start start 2024-09-16T04:08:23.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.430+0000 7f43704c9640 10 monclient: init 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.430+0000 7f43704c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.430+0000 7f43704c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.430+0000 7f43704c9640 10 monclient(hunting): picked mon.noname-a con 0x7f436809f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 10 monclient(hunting): picked mon.noname-b con 0x7f43680a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 10 monclient(hunting): picked mon.noname-c con 0x7f436809e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43680a1260 con 0x7f436809f640 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436809d0c0 con 0x7f436809e1d0 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436809d3a0 con 0x7f43680a0130 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 10 monclient(hunting): authenticate will time out at 878.936053s 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436ea66640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f436809e1d0 0x7f436813bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54004/0 (socket says 172.21.15.171:54004) 2024-09-16T04:08:23.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436ea66640 1 -- 172.21.15.171:0/3837264766 learned_addr learned my addr 172.21.15.171:0/3837264766 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436e265640 1 -- 172.21.15.171:0/3837264766 >> v1:172.21.15.148:6789/0 conn(0x7f436809f640 legacy=0x7f436809fa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417989838 0 0) 0x7f436809d0c0 con 0x7f436809e1d0 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient(hunting): _negotiate 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient(hunting): my global_id is 4744 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4350003520 con 0x7f436809e1d0 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4217057525 0 0) 0x7f4350003520 con 0x7f436809e1d0 2024-09-16T04:08:23.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 >> v1:172.21.15.148:6790/0 conn(0x7f43680a0130 legacy=0x7f4368138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 >> v1:172.21.15.148:6789/0 conn(0x7f436809f640 legacy=0x7f436809fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: _finish_hunting 0 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 monclient: found mon.noname-c 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436809d680 con 0x7f436809e1d0 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: _finish_auth 0 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: _check_auth_tickets 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4358001990 con 0x7f436809e1d0 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: dump: 2024-09-16T04:08:23.442 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 5 monclient: authenticate success, global_id 4744 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4358001c80 con 0x7f436809e1d0 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:23.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 1 -- 172.21.15.171:0/3837264766 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4358002090 con 0x7f436809e1d0 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f43704c9640 4 set_mon_vals no callback set 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.431+0000 7f436d263640 10 monclient: dump: 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:23.444 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 10 monclient: shutdown 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 1 -- 172.21.15.171:0/3837264766 >> v1:172.21.15.171:6789/0 conn(0x7f436809e1d0 legacy=0x7f436813bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 1 -- 172.21.15.171:0/3837264766 shutdown_connections 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 1 -- 172.21.15.171:0/3837264766 >> 172.21.15.171:0/3837264766 conn(0x7f43680a52a0 msgr2=0x7f43680a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 1 -- 172.21.15.171:0/3837264766 shutdown_connections 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 1 -- 172.21.15.171:0/3837264766 wait complete. 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 10 monclient: build_initial_monmap 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.432+0000 7f43704c9640 10 monclient: monmap: 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:23.445 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 1 Processor -- start 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 1 -- start start 2024-09-16T04:08:23.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient: init 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient(hunting): picked mon.noname-b con 0x7f436809e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient(hunting): picked mon.noname-a con 0x7f436809f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient(hunting): picked mon.noname-c con 0x7f43680a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f436806ceb0 con 0x7f436809f640 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4368137e20 con 0x7f436809e1d0 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f436e265640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f436809e1d0 0x7f43681c8370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54020/0 (socket says 172.21.15.171:54020) 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f436e265640 1 -- 172.21.15.171:0/3279205281 learned_addr learned my addr 172.21.15.171:0/3279205281 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43681386c0 con 0x7f43680a0130 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:23.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f43704c9640 10 monclient(hunting): authenticate will time out at 878.938873s 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.433+0000 7f436da64640 1 -- 172.21.15.171:0/3279205281 >> v1:172.21.15.148:6789/0 conn(0x7f436809f640 legacy=0x7f43681cd340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209837339 0 0) 0x7f4368137e20 con 0x7f436809e1d0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): my global_id is 4747 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4354003520 con 0x7f436809e1d0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4045696760 0 0) 0x7f4354003520 con 0x7f436809e1d0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4368137e20 con 0x7f436809e1d0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4360004180 con 0x7f436809e1d0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:23.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3743499748 0 0) 0x7f43681386c0 con 0x7f43680a0130 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:23.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient(hunting): my global_id is 4775 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4354003520 con 0x7f43680a0130 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 330725565 0 0) 0x7f4368137e20 con 0x7f436809e1d0 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 >> v1:172.21.15.148:6790/0 conn(0x7f43680a0130 legacy=0x7f43681d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 >> v1:172.21.15.148:6789/0 conn(0x7f436809f640 legacy=0x7f43681cd340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 monclient: found mon.b 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4368138b60 con 0x7f436809e1d0 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:23.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 5 monclient: authenticate success, global_id 4747 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 10 monclient: _renew_subs 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4368137290 con 0x7f436809e1d0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 10 monclient: _renew_subs 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f43681c8a80 con 0x7f436809e1d0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4360004470 con 0x7f436809e1d0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4360004840 con 0x7f436809e1d0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:23.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f436ca62640 4 set_mon_vals no callback set 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.434+0000 7f4366ffd640 10 monclient: dump: 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4360004aa0 con 0x7f436809e1d0 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f43704c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:23.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f43704c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f43704c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4348005080 con 0x7f436809e1d0 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(51..51 src has 1..51) ==== 3931+0+0 (unknown 1259710449 0 0) 0x7f4360047030 con 0x7f436809e1d0 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.435+0000 7f436da64640 10 monclient: get_auth_request con 0x7f4354038f90 auth_method 0 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.439+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4360015d80 con 0x7f436809e1d0 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.440+0000 7f4366ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.440+0000 7f4366ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:23.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.607+0000 7f43704c9640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T04:08:23.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.607+0000 7f43704c9640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T04:08:23.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.607+0000 7f43704c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:23.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:23.607+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) -- 0x7f4348005370 con 0x7f436809e1d0 2024-09-16T04:08:24.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.233+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v52) ==== 150+0+0 (unknown 2876392965 0 0) 0x7f4360015680 con 0x7f436809e1d0 2024-09-16T04:08:24.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.233+0000 7f4366ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T04:08:24.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.233+0000 7f4366ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:24.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f43704c9640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T04:08:24.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f43704c9640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T04:08:24.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f43704c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) -- 0x7f4348005610 con 0x7f436809e1d0 2024-09-16T04:08:24.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f4366ffd640 1 -- 172.21.15.171:0/3279205281 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v52) ==== 177+0+0 (unknown 1392463029 0 0) 0x7f43600199d0 con 0x7f436809e1d0 2024-09-16T04:08:24.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f4366ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-16T04:08:24.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.338+0000 7f4366ffd640 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-16T04:08:24.341 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo-hdd already exists 2024-09-16T04:08:24.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 >> v1:172.21.15.171:6812/7327273 conn(0x7f4354038f90 legacy=0x7f435403b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 10 monclient: shutdown 2024-09-16T04:08:24.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:24.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 >> v1:172.21.15.171:6789/0 conn(0x7f436809e1d0 legacy=0x7f43681c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 shutdown_connections 2024-09-16T04:08:24.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 >> 172.21.15.171:0/3279205281 conn(0x7f43680a52a0 msgr2=0x7f43680a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:24.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 shutdown_connections 2024-09-16T04:08:24.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.340+0000 7f43704c9640 1 -- 172.21.15.171:0/3279205281 wait complete. 2024-09-16T04:08:24.352 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T04:08:24.352 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo-ssd 2024-09-16T04:08:24.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.430+0000 7fa1a1531640 10 monclient: get_monmap_and_config 2024-09-16T04:08:24.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.430+0000 7fa1a1531640 10 monclient: build_initial_monmap 2024-09-16T04:08:24.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.430+0000 7fa1a1531640 10 monclient: monmap: 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:24.431529+0000 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:24.431529+0000 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:24.433 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 1 Processor -- start 2024-09-16T04:08:24.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 1 -- start start 2024-09-16T04:08:24.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient: init 2024-09-16T04:08:24.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:24.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:24.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c13d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c0a1340 con 0x7fa19c136b60 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c137650 con 0x7fa19c139b10 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c0a0aa0 con 0x7fa19c13d620 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1a1531640 10 monclient(hunting): authenticate will time out at 879.936884s 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa19affd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa19c136b60 legacy=0x7fa19c136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.431+0000 7fa1927fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa19c139b10 0x7fa19c13bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54028/0 (socket says 172.21.15.171:54028) 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa1927fc640 1 -- 172.21.15.171:0/2382876816 learned_addr learned my addr 172.21.15.171:0/2382876816 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4240454632 0 0) 0x7fa19c137650 con 0x7fa19c139b10 2024-09-16T04:08:24.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient(hunting): my global_id is 4750 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa188003520 con 0x7fa19c139b10 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 106471377 0 0) 0x7fa188003520 con 0x7fa19c139b10 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 >> v1:172.21.15.148:6790/0 conn(0x7fa19c13d620 legacy=0x7fa19c13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 >> v1:172.21.15.148:6789/0 conn(0x7fa19c136b60 legacy=0x7fa19c136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19c0a0550 con 0x7fa19c139b10 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa1a1531640 5 monclient: authenticate success, global_id 4750 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa184001610 con 0x7fa19c139b10 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa1a1531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: dump: 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.444 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa1a1531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa184001970 con 0x7fa19c139b10 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa1a1531640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 1 -- 172.21.15.171:0/2382876816 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa1840048a0 con 0x7fa19c139b10 2024-09-16T04:08:24.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa1a1531640 4 set_mon_vals no callback set 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.432+0000 7fa19a7fc640 10 monclient: dump: 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.446 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 10 monclient: shutdown 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 1 -- 172.21.15.171:0/2382876816 >> v1:172.21.15.171:6789/0 conn(0x7fa19c139b10 legacy=0x7fa19c13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 1 -- 172.21.15.171:0/2382876816 shutdown_connections 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 1 -- 172.21.15.171:0/2382876816 >> 172.21.15.171:0/2382876816 conn(0x7fa19c0a98b0 msgr2=0x7fa19c0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:24.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 1 -- 172.21.15.171:0/2382876816 shutdown_connections 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 1 -- 172.21.15.171:0/2382876816 wait complete. 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 10 monclient: build_initial_monmap 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 10 monclient: monmap: 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:24.448 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.433+0000 7fa1a1531640 1 Processor -- start 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 1 -- start start 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient: init 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient(hunting): picked mon.noname-b con 0x7fa19c136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient(hunting): picked mon.noname-a con 0x7fa19c139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient(hunting): picked mon.noname-c con 0x7fa19c13d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c09eb40 con 0x7fa19c139b10 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c09ac00 con 0x7fa19c136b60 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa19c13ad10 con 0x7fa19c13d620 2024-09-16T04:08:24.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1a1531640 10 monclient(hunting): authenticate will time out at 879.939614s 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa19affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa19c136b60 0x7fa19c139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54036/0 (socket says 172.21.15.171:54036) 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa19affd640 1 -- 172.21.15.171:0/1704123338 learned_addr learned my addr 172.21.15.171:0/1704123338 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa1927fc640 1 -- 172.21.15.171:0/1704123338 >> v1:172.21.15.148:6789/0 conn(0x7fa19c139b10 legacy=0x7fa19c13a600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50173565 0 0) 0x7fa19c09ac00 con 0x7fa19c136b60 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa193fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa193fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa193fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa193fff640 10 monclient(hunting): my global_id is 4753 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.434+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa17c003520 con 0x7fa19c136b60 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1211712634 0 0) 0x7fa17c003520 con 0x7fa19c136b60 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa19c09ac00 con 0x7fa19c136b60 2024-09-16T04:08:24.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa18c004180 con 0x7fa19c136b60 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): dump: 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.451 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259772228 0 0) 0x7fa19c13ad10 con 0x7fa19c13d620 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient(hunting): my global_id is 4781 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa17c003520 con 0x7fa19c13d620 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1274871440 0 0) 0x7fa19c09ac00 con 0x7fa19c136b60 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 >> v1:172.21.15.148:6790/0 conn(0x7fa19c13d620 legacy=0x7fa19c1d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 >> v1:172.21.15.148:6789/0 conn(0x7fa19c139b10 legacy=0x7fa19c13a600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 monclient: found mon.b 2024-09-16T04:08:24.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19c09aee0 con 0x7fa19c136b60 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 5 monclient: authenticate success, global_id 4753 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 10 monclient: _renew_subs 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa19c0a0240 con 0x7fa19c136b60 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 10 monclient: _renew_subs 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa19c13ced0 con 0x7fa19c136b60 2024-09-16T04:08:24.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa18c004470 con 0x7fa19c136b60 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa18c004840 con 0x7fa19c136b60 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa199ffb640 4 set_mon_vals no callback set 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.435+0000 7fa193fff640 10 monclient: dump: 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.454 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa18c004aa0 con 0x7fa19c136b60 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa1a1531640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa1a1531640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa1a1531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa180005080 con 0x7fa19c136b60 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(52..52 src has 1..52) ==== 3994+0+0 (unknown 1064049271 0 0) 0x7fa18c047070 con 0x7fa19c136b60 2024-09-16T04:08:24.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.436+0000 7fa1927fc640 10 monclient: get_auth_request con 0x7fa17c038f90 auth_method 0 2024-09-16T04:08:24.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.440+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa18c015d80 con 0x7fa19c136b60 2024-09-16T04:08:24.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.440+0000 7fa193fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:24.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.440+0000 7fa193fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa1a1531640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa1a1531640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa1a1531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7fa180005b80 con 0x7fa19c136b60 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7fa18c015680 con 0x7fa19c136b60 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa193fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.609+0000 7fa193fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa1a1531640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa1a1531640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa1a1531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7fa180001090 con 0x7fa19c136b60 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa193fff640 1 -- 172.21.15.171:0/1704123338 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7fa18c0199d0 con 0x7fa19c136b60 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa193fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.710+0000 7fa193fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 >> v1:172.21.15.171:6812/7327273 conn(0x7fa17c038f90 legacy=0x7fa17c03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 10 monclient: shutdown 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 >> v1:172.21.15.171:6789/0 conn(0x7fa19c136b60 legacy=0x7fa19c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 shutdown_connections 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 >> 172.21.15.171:0/1704123338 conn(0x7fa19c0a98b0 msgr2=0x7fa19c0a8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 shutdown_connections 2024-09-16T04:08:24.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.712+0000 7fa1a1531640 1 -- 172.21.15.171:0/1704123338 wait complete. 2024-09-16T04:08:24.723 INFO:tasks.workunit.client.0.smithi171.stdout:foo-ssd 2024-09-16T04:08:24.724 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-16T04:08:24.724 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false grep foo-hdd 2024-09-16T04:08:24.724 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:24.724 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo-hdd 2024-09-16T04:08:24.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.802+0000 7fabbac1e640 10 monclient: get_monmap_and_config 2024-09-16T04:08:24.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.802+0000 7fabbac1e640 10 monclient: build_initial_monmap 2024-09-16T04:08:24.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.802+0000 7fabbac1e640 10 monclient: monmap: 2024-09-16T04:08:24.804 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:24.804 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.804 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:24.803275+0000 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:24.803275+0000 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:24.805 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 1 Processor -- start 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 1 -- start start 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient: init 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient(hunting): picked mon.noname-a con 0x7fabb409b660 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient(hunting): picked mon.noname-b con 0x7fabb40734b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient(hunting): picked mon.noname-c con 0x7fabb4136d70 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:24.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb409c150 con 0x7fabb409b660 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb40998b0 con 0x7fabb4136d70 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb4099b90 con 0x7fabb40734b0 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabbac1e640 10 monclient(hunting): authenticate will time out at 880.308869s 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabb91bb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fabb4136d70 0x7fabb413f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54050/0 (socket says 172.21.15.171:54050) 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabb91bb640 1 -- 172.21.15.171:0/1210740863 learned_addr learned my addr 172.21.15.171:0/1210740863 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.803+0000 7fabb89ba640 1 -- 172.21.15.171:0/1210740863 >> v1:172.21.15.148:6789/0 conn(0x7fabb409b660 legacy=0x7fabb409ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059995712 0 0) 0x7fabb40998b0 con 0x7fabb4136d70 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:24.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient(hunting): my global_id is 4756 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba4003520 con 0x7fabb4136d70 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2548671855 0 0) 0x7faba4003520 con 0x7fabb4136d70 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 >> v1:172.21.15.148:6790/0 conn(0x7fabb40734b0 legacy=0x7fabb413bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 >> v1:172.21.15.148:6789/0 conn(0x7fabb409b660 legacy=0x7fabb409ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb4099e70 con 0x7fabb4136d70 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fabac001990 con 0x7fabb4136d70 2024-09-16T04:08:24.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: dump: 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.815 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 5 monclient: authenticate success, global_id 4756 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabac001c80 con 0x7fabb4136d70 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 1 -- 172.21.15.171:0/1210740863 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fabac002090 con 0x7fabb4136d70 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabb37fe640 10 monclient: dump: 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.816 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 4 set_mon_vals no callback set 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 10 monclient: shutdown 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.804+0000 7fabbac1e640 1 -- 172.21.15.171:0/1210740863 >> v1:172.21.15.171:6789/0 conn(0x7fabb4136d70 legacy=0x7fabb413f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 1 -- 172.21.15.171:0/1210740863 shutdown_connections 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 1 -- 172.21.15.171:0/1210740863 >> 172.21.15.171:0/1210740863 conn(0x7fabb40a1e90 msgr2=0x7fabb40a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 1 -- 172.21.15.171:0/1210740863 shutdown_connections 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 1 -- 172.21.15.171:0/1210740863 wait complete. 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 10 monclient: build_initial_monmap 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 10 monclient: monmap: 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.818 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 1 Processor -- start 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 1 -- start start 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.805+0000 7fabbac1e640 10 monclient: init 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient(hunting): picked mon.noname-b con 0x7fabb40734b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient(hunting): picked mon.noname-c con 0x7fabb409b660 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient(hunting): picked mon.noname-a con 0x7fabb4136d70 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb408f790 con 0x7fabb4136d70 2024-09-16T04:08:24.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb4134b50 con 0x7fabb40734b0 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb4092720 con 0x7fabb409b660 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabbac1e640 10 monclient(hunting): authenticate will time out at 880.311562s 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb91bb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fabb4136d70 legacy=0x7fabb41d39d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb89ba640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fabb40734b0 0x7fabb408e970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54062/0 (socket says 172.21.15.171:54062) 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb89ba640 1 -- 172.21.15.171:0/699552176 learned_addr learned my addr 172.21.15.171:0/699552176 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928790495 0 0) 0x7fabb4134b50 con 0x7fabb40734b0 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): my global_id is 4759 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab98003520 con 0x7fabb40734b0 2024-09-16T04:08:24.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50736904 0 0) 0x7fabb4092720 con 0x7fabb409b660 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): my global_id is 4787 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb4134b50 con 0x7fabb409b660 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3081005689 0 0) 0x7fab98003520 con 0x7fabb40734b0 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabb4092720 con 0x7fabb40734b0 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faba80041d0 con 0x7fabb40734b0 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.806+0000 7fabb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient(hunting): dump: 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.821 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 514990232 0 0) 0x7fabb4092720 con 0x7fabb40734b0 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 >> v1:172.21.15.148:6790/0 conn(0x7fabb409b660 legacy=0x7fabb408f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 >> v1:172.21.15.148:6789/0 conn(0x7fabb4136d70 legacy=0x7fabb41d39d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:24.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 monclient: found mon.b 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb4091e60 con 0x7fabb40734b0 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 5 monclient: authenticate success, global_id 4759 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 10 monclient: _renew_subs 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fabb4090f00 con 0x7fabb40734b0 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 10 monclient: _renew_subs 2024-09-16T04:08:24.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fabb4097f30 con 0x7fabb40734b0 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faba8004480 con 0x7fabb40734b0 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faba8004850 con 0x7fabb40734b0 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb2ffd640 4 set_mon_vals no callback set 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 10 monclient: dump: 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:24.824 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.807+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7faba8004ab0 con 0x7fabb40734b0 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.808+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(52..52 src has 1..52) ==== 3994+0+0 (unknown 1064049271 0 0) 0x7faba8046f10 con 0x7fabb40734b0 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.808+0000 7fabbac1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.808+0000 7fabbac1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.808+0000 7fabbac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.808+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab94005080 con 0x7fabb40734b0 2024-09-16T04:08:24.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.808+0000 7fabb3fff640 10 monclient: get_auth_request con 0x7fab98038f60 auth_method 0 2024-09-16T04:08:24.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.812+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7faba8015d60 con 0x7fabb40734b0 2024-09-16T04:08:24.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.812+0000 7fabb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:24.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.812+0000 7fabb17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabbac1e640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabbac1e640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabbac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7fab94005b80 con 0x7fabb40734b0 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7faba8015660 con 0x7fabb40734b0 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:24.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:24.983+0000 7fabb17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:25.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabbac1e640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:25.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabbac1e640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:25.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabbac1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) -- 0x7fab94005760 con 0x7fabb40734b0 2024-09-16T04:08:25.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabb17fa640 1 -- 172.21.15.171:0/699552176 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v52) ==== 90+0+8 (unknown 2553503695 0 905510949) 0x7faba80199b0 con 0x7fabb40734b0 2024-09-16T04:08:25.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabb17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-16T04:08:25.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.084+0000 7fabb17fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:25.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 >> v1:172.21.15.171:6812/7327273 conn(0x7fab98038f60 legacy=0x7fab9803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 10 monclient: shutdown 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 >> v1:172.21.15.171:6789/0 conn(0x7fabb40734b0 legacy=0x7fabb408e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 shutdown_connections 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 >> 172.21.15.171:0/699552176 conn(0x7fabb40a1e90 msgr2=0x7fabb40985b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 shutdown_connections 2024-09-16T04:08:25.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.086+0000 7fabbac1e640 1 -- 172.21.15.171:0/699552176 wait complete. 2024-09-16T04:08:25.098 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:25.098 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T04:08:25.098 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo-hdd 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.176+0000 7fdc8e0dd640 10 monclient: get_monmap_and_config 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.176+0000 7fdc8e0dd640 10 monclient: build_initial_monmap 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.176+0000 7fdc8e0dd640 10 monclient: monmap: 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:25.176946+0000 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:25.176946+0000 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:25.178 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.179 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:25.179 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:25.179 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:25.179 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 1 Processor -- start 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 1 -- start start 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient: init 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdc88136dd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:25.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdc881371b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdc881375b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8809af40 con 0x7fdc881371b0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8809b220 con 0x7fdc88136dd0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8809aa20 con 0x7fdc881375b0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc8e0dd640 10 monclient(hunting): authenticate will time out at 880.682609s 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc877fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdc88136dd0 0x7fdc8809bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54068/0 (socket says 172.21.15.171:54068) 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc877fe640 1 -- 172.21.15.171:0/3385825561 learned_addr learned my addr 172.21.15.171:0/3385825561 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc7ffff640 1 -- 172.21.15.171:0/3385825561 >> v1:172.21.15.148:6789/0 conn(0x7fdc881371b0 legacy=0x7fdc8813bbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045392162 0 0) 0x7fdc8809b220 con 0x7fdc88136dd0 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc86ffd640 10 monclient(hunting): my global_id is 4762 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.177+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc74003520 con 0x7fdc88136dd0 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 621360144 0 0) 0x7fdc8809aa20 con 0x7fdc881375b0 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient(hunting): my global_id is 4790 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc8809b220 con 0x7fdc881375b0 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 360472204 0 0) 0x7fdc74003520 con 0x7fdc88136dd0 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 >> v1:172.21.15.148:6790/0 conn(0x7fdc881375b0 legacy=0x7fdc8813f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 >> v1:172.21.15.148:6789/0 conn(0x7fdc881371b0 legacy=0x7fdc8813bbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:25.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 monclient: found mon.noname-c 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc880a4d30 con 0x7fdc88136dd0 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdc78001610 con 0x7fdc88136dd0 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 5 monclient: authenticate success, global_id 4762 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:25.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: dump: 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc78001930 con 0x7fdc88136dd0 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 1 -- 172.21.15.171:0/3385825561 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdc78001d00 con 0x7fdc88136dd0 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 4 set_mon_vals no callback set 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:25.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc86ffd640 10 monclient: dump: 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 10 monclient: shutdown 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/3385825561 >> v1:172.21.15.171:6789/0 conn(0x7fdc88136dd0 legacy=0x7fdc8809bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/3385825561 shutdown_connections 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/3385825561 >> 172.21.15.171:0/3385825561 conn(0x7fdc880a9860 msgr2=0x7fdc880a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/3385825561 shutdown_connections 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.178+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/3385825561 wait complete. 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.179+0000 7fdc8e0dd640 10 monclient: build_initial_monmap 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.179+0000 7fdc8e0dd640 10 monclient: monmap: 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.196 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.179+0000 7fdc8e0dd640 1 Processor -- start 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.179+0000 7fdc8e0dd640 1 -- start start 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.179+0000 7fdc8e0dd640 10 monclient: init 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdc88136dd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdc881371b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdc881375b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:25.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8813c150 con 0x7fdc88136dd0 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc8813f880 con 0x7fdc881375b0 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880a0340 con 0x7fdc881371b0 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient(hunting): authenticate will time out at 880.685245s 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc877fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdc88136dd0 legacy=0x7fdc88138430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc87fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdc881375b0 0x7fdc881d3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54070/0 (socket says 172.21.15.171:54070) 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc87fff640 1 -- 172.21.15.171:0/723937395 learned_addr learned my addr 172.21.15.171:0/723937395 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672446696 0 0) 0x7fdc8813f880 con 0x7fdc881375b0 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): my global_id is 4765 2024-09-16T04:08:25.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc68003520 con 0x7fdc881375b0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2074780700 0 0) 0x7fdc880a0340 con 0x7fdc881371b0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): my global_id is 4793 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc8813f880 con 0x7fdc881371b0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1574236784 0 0) 0x7fdc68003520 con 0x7fdc881375b0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc880a0340 con 0x7fdc881375b0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdc80001c50 con 0x7fdc881375b0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): dump: 2024-09-16T04:08:25.199 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2524158646 0 0) 0x7fdc880a0340 con 0x7fdc881375b0 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 >> v1:172.21.15.148:6790/0 conn(0x7fdc881371b0 legacy=0x7fdc88138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 >> v1:172.21.15.148:6789/0 conn(0x7fdc88136dd0 legacy=0x7fdc88138430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 monclient: found mon.b 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc88139e60 con 0x7fdc881375b0 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 5 monclient: authenticate success, global_id 4765 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient: _renew_subs 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8813be70 con 0x7fdc881375b0 2024-09-16T04:08:25.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient: _renew_subs 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc8813a9c0 con 0x7fdc881375b0 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc80002020 con 0x7fdc881375b0 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdc80003de0 con 0x7fdc881375b0 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc867fc640 4 set_mon_vals no callback set 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.180+0000 7fdc84ff9640 10 monclient: dump: 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.202 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.181+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdc80003f80 con 0x7fdc881375b0 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.182+0000 7fdc8e0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.182+0000 7fdc8e0dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.182+0000 7fdc8e0dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.182+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(52..52 src has 1..52) ==== 3994+0+0 (unknown 1064049271 0 0) 0x7fdc80047260 con 0x7fdc881375b0 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.182+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc6c005080 con 0x7fdc881375b0 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.182+0000 7fdc877fe640 10 monclient: get_auth_request con 0x7fdc68038be0 auth_method 0 2024-09-16T04:08:25.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.186+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdc80015fd0 con 0x7fdc881375b0 2024-09-16T04:08:25.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.186+0000 7fdc84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.186+0000 7fdc84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc8e0dd640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc8e0dd640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc8e0dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7fdc6c005b80 con 0x7fdc881375b0 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7fdc800158d0 con 0x7fdc881375b0 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.359+0000 7fdc84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc8e0dd640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc8e0dd640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc8e0dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7fdc6c005760 con 0x7fdc881375b0 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc84ff9640 1 -- 172.21.15.171:0/723937395 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7fdc80019c20 con 0x7fdc881375b0 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc84ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.460+0000 7fdc84ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:25.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 >> v1:172.21.15.171:6812/7327273 conn(0x7fdc68038be0 legacy=0x7fdc6803b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 10 monclient: shutdown 2024-09-16T04:08:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 >> v1:172.21.15.171:6789/0 conn(0x7fdc881375b0 legacy=0x7fdc881d3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 shutdown_connections 2024-09-16T04:08:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 >> 172.21.15.171:0/723937395 conn(0x7fdc880a9860 msgr2=0x7fdc880a8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 shutdown_connections 2024-09-16T04:08:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.462+0000 7fdc8e0dd640 1 -- 172.21.15.171:0/723937395 wait complete. 2024-09-16T04:08:25.473 INFO:tasks.workunit.client.0.smithi171.stdout:foo-hdd 2024-09-16T04:08:25.474 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-16T04:08:25.474 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false grep foo-ssd 2024-09-16T04:08:25.474 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:25.474 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo-ssd 2024-09-16T04:08:25.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.551+0000 7f8d0b3a2640 10 monclient: get_monmap_and_config 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.551+0000 7f8d0b3a2640 10 monclient: build_initial_monmap 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.551+0000 7f8d0b3a2640 10 monclient: monmap: 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:25.552713+0000 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:25.552713+0000 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:25.554 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.552+0000 7f8d0b3a2640 1 Processor -- start 2024-09-16T04:08:25.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.552+0000 7f8d0b3a2640 1 -- start start 2024-09-16T04:08:25.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.552+0000 7f8d0b3a2640 10 monclient: init 2024-09-16T04:08:25.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient(hunting): picked mon.noname-b con 0x7f8d04136af0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient(hunting): picked mon.noname-a con 0x7f8d040a1bd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient(hunting): picked mon.noname-c con 0x7f8d0413d170 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d040a0470 con 0x7f8d040a1bd0 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0409fe10 con 0x7f8d0413d170 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d040a4eb0 con 0x7f8d04136af0 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient(hunting): authenticate will time out at 881.058279s 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d09b9f640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8d040a1bd0 legacy=0x7f8d0413ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0aba1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8d0413d170 0x7f8d0413f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54080/0 (socket says 172.21.15.171:54080) 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0aba1640 1 -- 172.21.15.171:0/3182386969 learned_addr learned my addr 172.21.15.171:0/3182386969 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:25.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4044581886 0 0) 0x7f8d0409fe10 con 0x7f8d0413d170 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): my global_id is 4768 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf0003520 con 0x7f8d0413d170 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328702874 0 0) 0x7f8d040a4eb0 con 0x7f8d04136af0 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient(hunting): my global_id is 4796 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d0409fe10 con 0x7f8d04136af0 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3104279503 0 0) 0x7f8cf0003520 con 0x7f8d0413d170 2024-09-16T04:08:25.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 >> v1:172.21.15.148:6790/0 conn(0x7f8d04136af0 legacy=0x7f8d04136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 >> v1:172.21.15.148:6789/0 conn(0x7f8d040a1bd0 legacy=0x7f8d0413ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: _finish_hunting 0 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 monclient: found mon.noname-c 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d040a11f0 con 0x7f8d0413d170 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: _finish_auth 0 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: _check_auth_tickets 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8cfc001990 con 0x7f8d0413d170 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 5 monclient: authenticate success, global_id 4768 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: dump: 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8cfc001c40 con 0x7f8d0413d170 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0b3a2640 4 set_mon_vals no callback set 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 1 -- 172.21.15.171:0/3182386969 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8cfc002010 con 0x7f8d0413d170 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.553+0000 7f8d0939e640 10 monclient: dump: 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 10 monclient: shutdown 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182386969 >> v1:172.21.15.171:6789/0 conn(0x7f8d0413d170 legacy=0x7f8d0413f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182386969 shutdown_connections 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182386969 >> 172.21.15.171:0/3182386969 conn(0x7f8d040a96b0 msgr2=0x7f8d040a5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182386969 shutdown_connections 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.554+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182386969 wait complete. 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient: build_initial_monmap 2024-09-16T04:08:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient: monmap: 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 1 Processor -- start 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 1 -- start start 2024-09-16T04:08:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient: init 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient(hunting): picked mon.noname-a con 0x7f8d040a1bd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient(hunting): picked mon.noname-b con 0x7f8d04136af0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient(hunting): picked mon.noname-c con 0x7f8d0413d170 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0413ca20 con 0x7f8d040a1bd0 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d040a0160 con 0x7f8d04136af0 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0409f1c0 con 0x7f8d0413d170 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0b3a2640 10 monclient(hunting): authenticate will time out at 881.060800s 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d09b9f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8d04136af0 0x7f8d041d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54082/0 (socket says 172.21.15.171:54082) 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d09b9f640 1 -- 172.21.15.171:0/3182668937 learned_addr learned my addr 172.21.15.171:0/3182668937 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.555+0000 7f8d0a3a0640 1 -- 172.21.15.171:0/3182668937 >> v1:172.21.15.148:6789/0 conn(0x7f8d040a1bd0 legacy=0x7f8d04135260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 503004727 0 0) 0x7f8d040a0160 con 0x7f8d04136af0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): my global_id is 4771 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ce8003520 con 0x7f8d04136af0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 537093846 0 0) 0x7f8ce8003520 con 0x7f8d04136af0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d040a0160 con 0x7f8d04136af0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8cf40041d0 con 0x7f8d04136af0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): dump: 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870163070 0 0) 0x7f8d0409f1c0 con 0x7f8d0413d170 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient(hunting): my global_id is 4799 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ce8003520 con 0x7f8d0413d170 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1804847483 0 0) 0x7f8d040a0160 con 0x7f8d04136af0 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 >> v1:172.21.15.148:6790/0 conn(0x7f8d0413d170 legacy=0x7f8d041d3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 >> v1:172.21.15.148:6789/0 conn(0x7f8d040a1bd0 legacy=0x7f8d04135260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 monclient: found mon.b 2024-09-16T04:08:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d04120d20 con 0x7f8d04136af0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 5 monclient: authenticate success, global_id 4771 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 10 monclient: _renew_subs 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d04121000 con 0x7f8d04136af0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 10 monclient: _renew_subs 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d0409e570 con 0x7f8d04136af0 2024-09-16T04:08:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8cf40044c0 con 0x7f8d04136af0 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8cf4004890 con 0x7f8d04136af0 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8d08b9d640 4 set_mon_vals no callback set 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.556+0000 7f8cfaffd640 10 monclient: dump: 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8cf4004a30 con 0x7f8d04136af0 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(52..52 src has 1..52) ==== 3994+0+0 (unknown 1064049271 0 0) 0x7f8cf4046ff0 con 0x7f8d04136af0 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8d0b3a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8d0b3a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8d0b3a2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cec005080 con 0x7f8d04136af0 2024-09-16T04:08:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.557+0000 7f8d0a3a0640 10 monclient: get_auth_request con 0x7f8ce8038fc0 auth_method 0 2024-09-16T04:08:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.562+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8cf4019990 con 0x7f8d04136af0 2024-09-16T04:08:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.562+0000 7f8cfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.562+0000 7f8cfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8d0b3a2640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8d0b3a2640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8d0b3a2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7f8cec005b80 con 0x7f8d04136af0 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7f8cf4045cc0 con 0x7f8d04136af0 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8cfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.729+0000 7f8cfaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:25.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8d0b3a2640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8d0b3a2640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8d0b3a2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) -- 0x7f8cec005760 con 0x7f8d04136af0 2024-09-16T04:08:25.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8cfaffd640 1 -- 172.21.15.171:0/3182668937 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v52) ==== 90+0+8 (unknown 1780583566 0 1286160932) 0x7f8cf4011d50 con 0x7f8d04136af0 2024-09-16T04:08:25.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8cfaffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-16T04:08:25.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.830+0000 7f8cfaffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:25.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.832+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 >> v1:172.21.15.171:6812/7327273 conn(0x7f8ce8038fc0 legacy=0x7f8ce803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.832+0000 7f8d0b3a2640 10 monclient: shutdown 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.832+0000 7f8d0b3a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.832+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 >> v1:172.21.15.171:6789/0 conn(0x7f8d04136af0 legacy=0x7f8d041d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.832+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 shutdown_connections 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.832+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 >> 172.21.15.171:0/3182668937 conn(0x7f8d040a96b0 msgr2=0x7f8d040a2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.833+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 shutdown_connections 2024-09-16T04:08:25.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.833+0000 7f8d0b3a2640 1 -- 172.21.15.171:0/3182668937 wait complete. 2024-09-16T04:08:25.843 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:25.844 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-16T04:08:25.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.923+0000 7f32d9a53640 10 monclient: get_monmap_and_config 2024-09-16T04:08:25.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.923+0000 7f32d9a53640 10 monclient: build_initial_monmap 2024-09-16T04:08:25.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.923+0000 7f32d9a53640 10 monclient: monmap: 2024-09-16T04:08:25.925 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:25.924071+0000 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:25.924071+0000 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:25.926 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.923+0000 7f32d9a53640 1 Processor -- start 2024-09-16T04:08:25.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.923+0000 7f32d9a53640 1 -- start start 2024-09-16T04:08:25.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.923+0000 7f32d9a53640 10 monclient: init 2024-09-16T04:08:25.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:25.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:25.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient(hunting): picked mon.noname-c con 0x7f32d409eb10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient(hunting): picked mon.noname-b con 0x7f32d409eef0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient(hunting): picked mon.noname-a con 0x7f32d413b180 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d409bb10 con 0x7f32d413b180 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d409f340 con 0x7f32d409eb10 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d409cb10 con 0x7f32d409eef0 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d9a53640 10 monclient(hunting): authenticate will time out at 881.429503s 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f32d409eb10 0x7f32d4134ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54090/0 (socket says 172.21.15.171:54090) 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d2ffd640 1 -- 172.21.15.171:0/1857189609 learned_addr learned my addr 172.21.15.171:0/1857189609 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d37fe640 1 -- 172.21.15.171:0/1857189609 >> v1:172.21.15.148:6789/0 conn(0x7f32d413b180 legacy=0x7f32d413d570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164659462 0 0) 0x7f32d409f340 con 0x7f32d409eb10 2024-09-16T04:08:25.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient(hunting): my global_id is 4774 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32bc003520 con 0x7f32d409eb10 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2237415314 0 0) 0x7f32bc003520 con 0x7f32d409eb10 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 >> v1:172.21.15.148:6790/0 conn(0x7f32d409eef0 legacy=0x7f32d4139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 >> v1:172.21.15.148:6789/0 conn(0x7f32d413b180 legacy=0x7f32d413d570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d409cdf0 con 0x7f32d409eb10 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f32c4001540 con 0x7f32d409eb10 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.924+0000 7f32d1ffb640 10 monclient: dump: 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.937 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 5 monclient: authenticate success, global_id 4774 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32c40018a0 con 0x7f32d409eb10 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d1ffb640 1 -- 172.21.15.171:0/1857189609 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f32c4001c70 con 0x7f32d409eb10 2024-09-16T04:08:25.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d1ffb640 10 monclient: dump: 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 4 set_mon_vals no callback set 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 10 monclient: shutdown 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 1 -- 172.21.15.171:0/1857189609 >> v1:172.21.15.171:6789/0 conn(0x7f32d409eb10 legacy=0x7f32d4134ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 1 -- 172.21.15.171:0/1857189609 shutdown_connections 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 1 -- 172.21.15.171:0/1857189609 >> 172.21.15.171:0/1857189609 conn(0x7f32d40a2150 msgr2=0x7f32d40a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 1 -- 172.21.15.171:0/1857189609 shutdown_connections 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.925+0000 7f32d9a53640 1 -- 172.21.15.171:0/1857189609 wait complete. 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.926+0000 7f32d9a53640 10 monclient: build_initial_monmap 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.926+0000 7f32d9a53640 10 monclient: monmap: 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:25.941 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.926+0000 7f32d9a53640 1 Processor -- start 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.926+0000 7f32d9a53640 1 -- start start 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.926+0000 7f32d9a53640 10 monclient: init 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient(hunting): picked mon.noname-b con 0x7f32d409eb10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient(hunting): picked mon.noname-a con 0x7f32d409eef0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient(hunting): picked mon.noname-c con 0x7f32d413b180 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d408e0a0 con 0x7f32d409eef0 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d409a290 con 0x7f32d409eb10 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d408e380 con 0x7f32d413b180 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:25.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient(hunting): authenticate will time out at 881.432122s 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f32d409eb10 0x7f32d4099470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54100/0 (socket says 172.21.15.171:54100) 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d27fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f32d409eef0 legacy=0x7f32d4099b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d2ffd640 1 -- 172.21.15.171:0/2951720876 learned_addr learned my addr 172.21.15.171:0/2951720876 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 43791911 0 0) 0x7f32d409a290 con 0x7f32d409eb10 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): my global_id is 4777 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32c0003520 con 0x7f32d409eb10 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147563050 0 0) 0x7f32d408e380 con 0x7f32d413b180 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:25.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): my global_id is 4805 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32d409a290 con 0x7f32d413b180 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3131844999 0 0) 0x7f32c0003520 con 0x7f32d409eb10 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32d408e380 con 0x7f32d409eb10 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f32c4001bf0 con 0x7f32d409eb10 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): dump: 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.944 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2369704419 0 0) 0x7f32d409a290 con 0x7f32d413b180 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f32c0003520 con 0x7f32d413b180 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f32cc0043b0 con 0x7f32d413b180 2024-09-16T04:08:25.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient(hunting): dump: 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2860244288 0 0) 0x7f32d408e380 con 0x7f32d409eb10 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 >> v1:172.21.15.148:6790/0 conn(0x7f32d413b180 legacy=0x7f32d41d3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 >> v1:172.21.15.148:6789/0 conn(0x7f32d409eef0 legacy=0x7f32d4099b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 monclient: found mon.b 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d413aa80 con 0x7f32d409eb10 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 5 monclient: authenticate success, global_id 4777 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient: _renew_subs 2024-09-16T04:08:25.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f32d413e940 con 0x7f32d409eb10 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient: _renew_subs 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f32d409a6c0 con 0x7f32d409eb10 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f32c4004bf0 con 0x7f32d409eb10 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f32c4004f40 con 0x7f32d409eb10 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32d17fa640 4 set_mon_vals no callback set 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.927+0000 7f32bbfff640 10 monclient: dump: 2024-09-16T04:08:25.948 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.928+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f32c4010370 con 0x7f32d409eb10 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.928+0000 7f32d9a53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.928+0000 7f32d9a53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.928+0000 7f32d9a53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.928+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b4005080 con 0x7f32d409eb10 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.929+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(52..52 src has 1..52) ==== 3994+0+0 (unknown 1064049271 0 0) 0x7f32c4046e20 con 0x7f32d409eb10 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.929+0000 7f32d27fc640 10 monclient: get_auth_request con 0x7f32c0038ef0 auth_method 0 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.933+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f32c4015d10 con 0x7f32d409eb10 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.933+0000 7f32bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:25.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:25.933+0000 7f32bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:26.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.109+0000 7f32d9a53640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T04:08:26.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.109+0000 7f32d9a53640 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T04:08:26.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.109+0000 7f32d9a53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:26.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.109+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) -- 0x7f32b4005370 con 0x7f32d409eb10 2024-09-16T04:08:26.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.982+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v54) ==== 151+0+0 (unknown 161330536 0 0) 0x7f32c4015610 con 0x7f32d409eb10 2024-09-16T04:08:26.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.982+0000 7f32bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T04:08:26.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:26.982+0000 7f32bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32d9a53640 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32d9a53640 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32d9a53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) -- 0x7f32b4001150 con 0x7f32d409eb10 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32bbfff640 1 -- 172.21.15.171:0/2951720876 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v54) ==== 151+0+0 (unknown 161330536 0 0) 0x7f32c4044ad0 con 0x7f32d409eb10 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32bbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-16T04:08:27.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.095+0000 7f32bbfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:27.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 >> v1:172.21.15.171:6812/7327273 conn(0x7f32c0038ef0 legacy=0x7f32c003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:27.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 10 monclient: shutdown 2024-09-16T04:08:27.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:27.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 >> v1:172.21.15.171:6789/0 conn(0x7f32d409eb10 legacy=0x7f32d4099470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:27.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 shutdown_connections 2024-09-16T04:08:27.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 >> 172.21.15.171:0/2951720876 conn(0x7f32d40a2150 msgr2=0x7f32d40a3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:27.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 shutdown_connections 2024-09-16T04:08:27.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.097+0000 7f32d9a53640 1 -- 172.21.15.171:0/2951720876 wait complete. 2024-09-16T04:08:27.109 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-16T04:08:27.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.190+0000 7ff0b7386640 10 monclient: get_monmap_and_config 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.190+0000 7ff0b7386640 10 monclient: build_initial_monmap 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.191+0000 7ff0b7386640 10 monclient: monmap: 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:27.191784+0000 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:27.191784+0000 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:27.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.191+0000 7ff0b7386640 1 Processor -- start 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.191+0000 7ff0b7386640 1 -- start start 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.191+0000 7ff0b7386640 10 monclient: init 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b0072aa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b00702c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b013b0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b00a0590 con 0x7ff0b00702c0 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b009f000 con 0x7ff0b013b0e0 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b00a02b0 con 0x7ff0b0072aa0 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b7386640 10 monclient(hunting): authenticate will time out at 882.697490s 2024-09-16T04:08:27.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b4921640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff0b00702c0 legacy=0x7ff0b01399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b5923640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff0b013b0e0 0x7ff0b013d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54114/0 (socket says 172.21.15.171:54114) 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0b5923640 1 -- 172.21.15.171:0/2795289795 learned_addr learned my addr 172.21.15.171:0/2795289795 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788569639 0 0) 0x7ff0b009f000 con 0x7ff0b013b0e0 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): my global_id is 4780 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff09c003520 con 0x7ff0b013b0e0 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4239251930 0 0) 0x7ff0b00a02b0 con 0x7ff0b0072aa0 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:27.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient(hunting): my global_id is 4808 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b009f000 con 0x7ff0b0072aa0 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1551646121 0 0) 0x7ff09c003520 con 0x7ff0b013b0e0 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 >> v1:172.21.15.148:6790/0 conn(0x7ff0b0072aa0 legacy=0x7ff0b0072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 >> v1:172.21.15.148:6789/0 conn(0x7ff0b00702c0 legacy=0x7ff0b01399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 monclient: found mon.noname-c 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.192+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b009f280 con 0x7ff0b013b0e0 2024-09-16T04:08:27.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 5 monclient: authenticate success, global_id 4780 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0a8001990 con 0x7ff0b013b0e0 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: dump: 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:27.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0a8001cb0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 1 -- 172.21.15.171:0/2795289795 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0a8002080 con 0x7ff0b013b0e0 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 4 set_mon_vals no callback set 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:27.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0a7fff640 10 monclient: dump: 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 10 monclient: shutdown 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 1 -- 172.21.15.171:0/2795289795 >> v1:172.21.15.171:6789/0 conn(0x7ff0b013b0e0 legacy=0x7ff0b013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:27.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 1 -- 172.21.15.171:0/2795289795 shutdown_connections 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 1 -- 172.21.15.171:0/2795289795 >> 172.21.15.171:0/2795289795 conn(0x7ff0b00a77f0 msgr2=0x7ff0b00a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 1 -- 172.21.15.171:0/2795289795 shutdown_connections 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.193+0000 7ff0b7386640 1 -- 172.21.15.171:0/2795289795 wait complete. 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.194+0000 7ff0b7386640 10 monclient: build_initial_monmap 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.194+0000 7ff0b7386640 10 monclient: monmap: 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:27.207 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.194+0000 7ff0b7386640 1 Processor -- start 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.194+0000 7ff0b7386640 1 -- start start 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.194+0000 7ff0b7386640 10 monclient: init 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b00702c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b0072aa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b013b0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b0121da0 con 0x7ff0b00702c0 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b0122080 con 0x7ff0b013b0e0 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b5122640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff0b00702c0 legacy=0x7ff0b01362c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:27.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0b0071eb0 con 0x7ff0b0072aa0 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient(hunting): authenticate will time out at 882.700199s 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b5923640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff0b013b0e0 0x7ff0b01d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54124/0 (socket says 172.21.15.171:54124) 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b5923640 1 -- 172.21.15.171:0/2484728979 learned_addr learned my addr 172.21.15.171:0/2484728979 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413148949 0 0) 0x7ff0b0122080 con 0x7ff0b013b0e0 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): my global_id is 4783 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff094003520 con 0x7ff0b013b0e0 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2054906211 0 0) 0x7ff0b0071eb0 con 0x7ff0b0072aa0 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:27.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): my global_id is 4811 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0b0122080 con 0x7ff0b0072aa0 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2919146878 0 0) 0x7ff094003520 con 0x7ff0b013b0e0 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0b0071eb0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0a8001950 con 0x7ff0b013b0e0 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.210 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2613763429 0 0) 0x7ff0b0122080 con 0x7ff0b0072aa0 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff094003520 con 0x7ff0b0072aa0 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0a00040f0 con 0x7ff0b0072aa0 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:27.211 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2899833157 0 0) 0x7ff0b0071eb0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 >> v1:172.21.15.148:6790/0 conn(0x7ff0b0072aa0 legacy=0x7ff0b01ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 >> v1:172.21.15.148:6789/0 conn(0x7ff0b00702c0 legacy=0x7ff0b01362c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:27.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 monclient: found mon.b 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b00719a0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 5 monclient: authenticate success, global_id 4783 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient: _renew_subs 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.195+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b009e1b0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0a8005040 con 0x7ff0b013b0e0 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:27.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0a8005390 con 0x7ff0b013b0e0 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 10 monclient: dump: 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:27.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0b7386640 10 monclient: _renew_subs 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a77fe640 4 set_mon_vals no callback set 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0b7386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0b0120670 con 0x7ff0b013b0e0 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.196+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff0a80107c0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.197+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(54..54 src has 1..54) ==== 4291+0+0 (unknown 156026946 0 0) 0x7ff0a80474a0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.197+0000 7ff0b7386640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.197+0000 7ff0b7386640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.197+0000 7ff0b7386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.197+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b00008d0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.197+0000 7ff0b5122640 10 monclient: get_auth_request con 0x7ff094038ef0 auth_method 0 2024-09-16T04:08:27.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.201+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff0a80161a0 con 0x7ff0b013b0e0 2024-09-16T04:08:27.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.201+0000 7ff0a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:27.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.201+0000 7ff0a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:27.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.368+0000 7ff0b7386640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T04:08:27.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.368+0000 7ff0b7386640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T04:08:27.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.368+0000 7ff0b7386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:27.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:27.368+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) -- 0x7ff0b011f100 con 0x7ff0b013b0e0 2024-09-16T04:08:28.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:28.989+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v56) ==== 177+0+0 (unknown 337347282 0 0) 0x7ff0a8015aa0 con 0x7ff0b013b0e0 2024-09-16T04:08:28.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:28.989+0000 7ff0a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T04:08:28.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:28.989+0000 7ff0a5ffb640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-16T04:08:29.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0b7386640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T04:08:29.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0b7386640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T04:08:29.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0b7386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) -- 0x7ff0b011f360 con 0x7ff0b013b0e0 2024-09-16T04:08:29.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0a5ffb640 1 -- 172.21.15.171:0/2484728979 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v56) ==== 184+0+0 (unknown 2586109229 0 0) 0x7ff0a801e5f0 con 0x7ff0b013b0e0 2024-09-16T04:08:29.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0a5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-16T04:08:29.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.093+0000 7ff0a5ffb640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-16T04:08:29.096 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'ec-foo' already exists 2024-09-16T04:08:29.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 >> v1:172.21.15.171:6812/7327273 conn(0x7ff094038ef0 legacy=0x7ff09403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:29.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 10 monclient: shutdown 2024-09-16T04:08:29.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:29.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 >> v1:172.21.15.171:6789/0 conn(0x7ff0b013b0e0 legacy=0x7ff0b01d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:29.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 shutdown_connections 2024-09-16T04:08:29.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 >> 172.21.15.171:0/2484728979 conn(0x7ff0b00a77f0 msgr2=0x7ff0b00a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:29.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 shutdown_connections 2024-09-16T04:08:29.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.095+0000 7ff0b7386640 1 -- 172.21.15.171:0/2484728979 wait complete. 2024-09-16T04:08:29.107 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-16T04:08:29.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.188+0000 7fb97f5cf640 10 monclient: get_monmap_and_config 2024-09-16T04:08:29.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.188+0000 7fb97f5cf640 10 monclient: build_initial_monmap 2024-09-16T04:08:29.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.188+0000 7fb97f5cf640 10 monclient: monmap: 2024-09-16T04:08:29.190 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:29.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:29.190 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:29.189400+0000 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:29.189400+0000 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:29.191 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:29.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 1 Processor -- start 2024-09-16T04:08:29.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 1 -- start start 2024-09-16T04:08:29.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient: init 2024-09-16T04:08:29.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:29.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:29.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb97809ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb9780a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb9780a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9780a0f20 con 0x7fb9780a0370 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9780a1200 con 0x7fb9780a0800 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809bca0 con 0x7fb97809ff90 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.189+0000 7fb97f5cf640 10 monclient(hunting): authenticate will time out at 884.694900s 2024-09-16T04:08:29.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97db6c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb9780a0800 0x7fb97813d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33698/0 (socket says 172.21.15.171:33698) 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97cb6a640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb9780a0370 legacy=0x7fb978139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97db6c640 1 -- 172.21.15.171:0/320685240 learned_addr learned my addr 172.21.15.171:0/320685240 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263768274 0 0) 0x7fb9780a1200 con 0x7fb9780a0800 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient(hunting): my global_id is 4786 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb964003520 con 0x7fb9780a0800 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3310321713 0 0) 0x7fb964003520 con 0x7fb9780a0800 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 >> v1:172.21.15.148:6790/0 conn(0x7fb97809ff90 legacy=0x7fb97809f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 >> v1:172.21.15.148:6789/0 conn(0x7fb9780a0370 legacy=0x7fb978139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:29.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 monclient: found mon.noname-c 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97809e830 con 0x7fb9780a0800 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb970001990 con 0x7fb9780a0800 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: dump: 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:29.202 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97f5cf640 5 monclient: authenticate success, global_id 4786 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97f5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb970001c70 con 0x7fb9780a0800 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97f5cf640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:29.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 1 -- 172.21.15.171:0/320685240 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb970002080 con 0x7fb9780a0800 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb97f5cf640 4 set_mon_vals no callback set 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.190+0000 7fb96ffff640 10 monclient: dump: 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:29.204 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 10 monclient: shutdown 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 1 -- 172.21.15.171:0/320685240 >> v1:172.21.15.171:6789/0 conn(0x7fb9780a0800 legacy=0x7fb97813d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 1 -- 172.21.15.171:0/320685240 shutdown_connections 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 1 -- 172.21.15.171:0/320685240 >> 172.21.15.171:0/320685240 conn(0x7fb9780a5450 msgr2=0x7fb9780a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 1 -- 172.21.15.171:0/320685240 shutdown_connections 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 1 -- 172.21.15.171:0/320685240 wait complete. 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 10 monclient: build_initial_monmap 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 10 monclient: monmap: 2024-09-16T04:08:29.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.191+0000 7fb97f5cf640 1 Processor -- start 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 1 -- start start 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient: init 2024-09-16T04:08:29.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb97809ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb9780a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb9780a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97813cbf0 con 0x7fb9780a0370 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809c0b0 con 0x7fb97809ff90 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97809d110 con 0x7fb9780a0800 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97f5cf640 10 monclient(hunting): authenticate will time out at 884.697467s 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97d36b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb97809ff90 0x7fb97809c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33700/0 (socket says 172.21.15.171:33700) 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97d36b640 1 -- 172.21.15.171:0/580942368 learned_addr learned my addr 172.21.15.171:0/580942368 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb97cb6a640 1 -- 172.21.15.171:0/580942368 >> v1:172.21.15.148:6789/0 conn(0x7fb9780a0370 legacy=0x7fb97809ca00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:29.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468741894 0 0) 0x7fb97809c0b0 con 0x7fb97809ff90 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): my global_id is 4789 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb95c003520 con 0x7fb97809ff90 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407492424 0 0) 0x7fb97809d110 con 0x7fb9780a0800 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): my global_id is 4817 2024-09-16T04:08:29.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb97809c0b0 con 0x7fb9780a0800 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2555117001 0 0) 0x7fb95c003520 con 0x7fb97809ff90 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb97809d110 con 0x7fb97809ff90 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb9740019c0 con 0x7fb97809ff90 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): dump: 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.192+0000 7fb96dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1371175381 0 0) 0x7fb97809d110 con 0x7fb97809ff90 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 >> v1:172.21.15.148:6790/0 conn(0x7fb9780a0800 legacy=0x7fb9781d3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 >> v1:172.21.15.148:6789/0 conn(0x7fb9780a0370 legacy=0x7fb97809ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 monclient: found mon.b 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:29.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97806e760 con 0x7fb97809ff90 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 5 monclient: authenticate success, global_id 4789 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 10 monclient: _renew_subs 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb97813adc0 con 0x7fb97809ff90 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9740038c0 con 0x7fb97809ff90 2024-09-16T04:08:29.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb974003c90 con 0x7fb97809ff90 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96dffb640 10 monclient: dump: 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:29.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 10 monclient: _renew_subs 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb96f7fe640 4 set_mon_vals no callback set 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.193+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb97813bc00 con 0x7fb97809ff90 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb974003e30 con 0x7fb97809ff90 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb97f5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb97f5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:29.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb97f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(56..56 src has 1..56) ==== 4732+0+0 (unknown 714910539 0 0) 0x7fb9740471c0 con 0x7fb97809ff90 2024-09-16T04:08:29.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb978136ef0 con 0x7fb97809ff90 2024-09-16T04:08:29.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.194+0000 7fb97cb6a640 10 monclient: get_auth_request con 0x7fb95c038f20 auth_method 0 2024-09-16T04:08:29.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.199+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb974015d30 con 0x7fb97809ff90 2024-09-16T04:08:29.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.199+0000 7fb96dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:29.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.199+0000 7fb96dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:29.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.364+0000 7fb97f5cf640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:29.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.364+0000 7fb97f5cf640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:29.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.364+0000 7fb97f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:29.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.364+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb97806bd60 con 0x7fb97809ff90 2024-09-16T04:08:30.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.998+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v57) ==== 154+0+0 (unknown 1549633872 0 0) 0x7fb974015630 con 0x7fb97809ff90 2024-09-16T04:08:30.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.998+0000 7fb96dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:30.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:29.998+0000 7fb96dffb640 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-16T04:08:30.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.098+0000 7fb97f5cf640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.098+0000 7fb97f5cf640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.098+0000 7fb97f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.098+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb97813e880 con 0x7fb97809ff90 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.099+0000 7fb96dffb640 1 -- 172.21.15.171:0/580942368 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v57) ==== 161+0+0 (unknown 3408549089 0 0) 0x7fb974019980 con 0x7fb97809ff90 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.099+0000 7fb96dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.099+0000 7fb96dffb640 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-16T04:08:30.101 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'ec-foo' does not exist 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 >> v1:172.21.15.171:6812/7327273 conn(0x7fb95c038f20 legacy=0x7fb95c03b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 10 monclient: shutdown 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 >> v1:172.21.15.171:6789/0 conn(0x7fb97809ff90 legacy=0x7fb97809c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 shutdown_connections 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 >> 172.21.15.171:0/580942368 conn(0x7fb9780a5450 msgr2=0x7fb9780a4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 shutdown_connections 2024-09-16T04:08:30.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.101+0000 7fb97f5cf640 1 -- 172.21.15.171:0/580942368 wait complete. 2024-09-16T04:08:30.112 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls 2024-09-16T04:08:30.112 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo 2024-09-16T04:08:30.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.189+0000 7fc74539f640 10 monclient: get_monmap_and_config 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.189+0000 7fc74539f640 10 monclient: build_initial_monmap 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.190+0000 7fc74539f640 10 monclient: monmap: 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:30.190778+0000 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:30.190778+0000 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:30.192 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.190+0000 7fc74539f640 1 Processor -- start 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.190+0000 7fc74539f640 1 -- start start 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.190+0000 7fc74539f640 10 monclient: init 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7401338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient(hunting): picked mon.noname-b con 0x7fc74009aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:30.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient(hunting): picked mon.noname-a con 0x7fc74013d240 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7401343d0 con 0x7fc74013d240 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc740075db0 con 0x7fc7401338e0 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74009b1e0 con 0x7fc74009aa20 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient(hunting): authenticate will time out at 885.696257s 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc7401338e0 0x7fc740133cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33712/0 (socket says 172.21.15.171:33712) 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73effd640 1 -- 172.21.15.171:0/1344909918 learned_addr learned my addr 172.21.15.171:0/1344909918 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73f7fe640 1 -- 172.21.15.171:0/1344909918 >> v1:172.21.15.148:6789/0 conn(0x7fc74013d240 legacy=0x7fc74013f630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535220483 0 0) 0x7fc740075db0 con 0x7fc7401338e0 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:30.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient(hunting): my global_id is 4792 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc728003520 con 0x7fc7401338e0 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3107526688 0 0) 0x7fc728003520 con 0x7fc7401338e0 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 >> v1:172.21.15.148:6790/0 conn(0x7fc74009aa20 legacy=0x7fc74013bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 >> v1:172.21.15.148:6789/0 conn(0x7fc74013d240 legacy=0x7fc74013f630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74009b4c0 con 0x7fc7401338e0 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:30.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 5 monclient: authenticate success, global_id 4792 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc730001610 con 0x7fc7401338e0 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: dump: 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.202 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc730001900 con 0x7fc7401338e0 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 1 -- 172.21.15.171:0/1344909918 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc730001d10 con 0x7fc7401338e0 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc74539f640 4 set_mon_vals no callback set 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:30.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.191+0000 7fc73dffb640 10 monclient: dump: 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 10 monclient: shutdown 2024-09-16T04:08:30.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 1 -- 172.21.15.171:0/1344909918 >> v1:172.21.15.171:6789/0 conn(0x7fc7401338e0 legacy=0x7fc740133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 1 -- 172.21.15.171:0/1344909918 shutdown_connections 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 1 -- 172.21.15.171:0/1344909918 >> 172.21.15.171:0/1344909918 conn(0x7fc7400a28d0 msgr2=0x7fc7400a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 1 -- 172.21.15.171:0/1344909918 shutdown_connections 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 1 -- 172.21.15.171:0/1344909918 wait complete. 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 10 monclient: build_initial_monmap 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.192+0000 7fc74539f640 10 monclient: monmap: 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.205 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 1 Processor -- start 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 1 -- start start 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient: init 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient(hunting): picked mon.noname-b con 0x7fc74009aa20 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7401338e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient(hunting): picked mon.noname-c con 0x7fc74013d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74009d5b0 con 0x7fc7401338e0 2024-09-16T04:08:30.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74009d0c0 con 0x7fc74009aa20 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74011dc30 con 0x7fc74013d240 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc74539f640 10 monclient(hunting): authenticate will time out at 885.698943s 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc73e7fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc7401338e0 legacy=0x7fc7401d0880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc73effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc74009aa20 0x7fc7401321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33716/0 (socket says 172.21.15.171:33716) 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.193+0000 7fc73effd640 1 -- 172.21.15.171:0/339475069 learned_addr learned my addr 172.21.15.171:0/339475069 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2354293658 0 0) 0x7fc74009b4c0 con 0x7fc74009aa20 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): my global_id is 4795 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc72c003520 con 0x7fc74009aa20 2024-09-16T04:08:30.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2752453687 0 0) 0x7fc74011dc30 con 0x7fc74013d240 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): my global_id is 4820 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc74009b4c0 con 0x7fc74013d240 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1626255030 0 0) 0x7fc72c003520 con 0x7fc74009aa20 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc74011dc30 con 0x7fc74009aa20 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc7300015d0 con 0x7fc74009aa20 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): dump: 2024-09-16T04:08:30.208 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1847167462 0 0) 0x7fc74011dc30 con 0x7fc74009aa20 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 >> v1:172.21.15.148:6790/0 conn(0x7fc74013d240 legacy=0x7fc7401d3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 >> v1:172.21.15.148:6789/0 conn(0x7fc7401338e0 legacy=0x7fc7401d0880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:30.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 monclient: found mon.b 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7400a0180 con 0x7fc74009aa20 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 5 monclient: authenticate success, global_id 4795 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 10 monclient: _renew_subs 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc74011d950 con 0x7fc74009aa20 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 10 monclient: _renew_subs 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc74011c960 con 0x7fc74009aa20 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc730001900 con 0x7fc74009aa20 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc730004c70 con 0x7fc74009aa20 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc73d7fa640 4 set_mon_vals no callback set 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.194+0000 7fc7277fe640 10 monclient: dump: 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.211 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc7300100a0 con 0x7fc74009aa20 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc74539f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc74539f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc74539f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc720005080 con 0x7fc74009aa20 2024-09-16T04:08:30.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(57..57 src has 1..57) ==== 4377+0+0 (unknown 1358300759 0 0) 0x7fc730047040 con 0x7fc74009aa20 2024-09-16T04:08:30.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.195+0000 7fc73e7fc640 10 monclient: get_auth_request con 0x7fc72c038fe0 auth_method 0 2024-09-16T04:08:30.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.200+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc74009d0c0 con 0x7fc74009aa20 2024-09-16T04:08:30.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.200+0000 7fc7277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:30.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.200+0000 7fc7277fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc74539f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc74539f640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc74539f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fc720005370 con 0x7fc74009aa20 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) ==== 65+0+60 (unknown 2262488454 0 1596542489) 0x7fc730015cf0 con 0x7fc74009aa20 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc7277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:30.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.369+0000 7fc7277fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc74539f640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc74539f640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc74539f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fc720000bc0 con 0x7fc74009aa20 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc7277fe640 1 -- 172.21.15.171:0/339475069 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v57) ==== 65+0+60 (unknown 2262488454 0 1596542489) 0x7fc7300155f0 con 0x7fc74009aa20 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc7277fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:30.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.469+0000 7fc7277fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 >> v1:172.21.15.171:6812/7327273 conn(0x7fc72c038fe0 legacy=0x7fc72c03b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 10 monclient: shutdown 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 >> v1:172.21.15.171:6789/0 conn(0x7fc74009aa20 legacy=0x7fc7401321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 shutdown_connections 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 >> 172.21.15.171:0/339475069 conn(0x7fc7400a28d0 msgr2=0x7fc7400a46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 shutdown_connections 2024-09-16T04:08:30.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.471+0000 7fc74539f640 1 -- 172.21.15.171:0/339475069 wait complete. 2024-09-16T04:08:30.482 INFO:tasks.workunit.client.0.smithi171.stdout:foo 2024-09-16T04:08:30.482 INFO:tasks.workunit.client.0.smithi171.stdout:foo-ssd 2024-09-16T04:08:30.482 INFO:tasks.workunit.client.0.smithi171.stdout:foo-hdd 2024-09-16T04:08:30.482 INFO:tasks.workunit.client.0.smithi171.stdout:ec-foo 2024-09-16T04:08:30.482 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T04:08:30.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.561+0000 7f7154a65640 10 monclient: get_monmap_and_config 2024-09-16T04:08:30.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.561+0000 7f7154a65640 10 monclient: build_initial_monmap 2024-09-16T04:08:30.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.561+0000 7f7154a65640 10 monclient: monmap: 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:30.562556+0000 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:30.562556+0000 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:30.564 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.562+0000 7f7154a65640 1 Processor -- start 2024-09-16T04:08:30.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.562+0000 7f7154a65640 1 -- start start 2024-09-16T04:08:30.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.562+0000 7f7154a65640 10 monclient: init 2024-09-16T04:08:30.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient(hunting): picked mon.noname-c con 0x7f715009f7c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient(hunting): picked mon.noname-a con 0x7f715009fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient(hunting): picked mon.noname-b con 0x7f71501395d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71500a1490 con 0x7f715009fba0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715009d1b0 con 0x7f715009f7c0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715009d490 con 0x7f71501395d0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient(hunting): authenticate will time out at 886.068233s 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714e59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f715009f7c0 0x7f715009de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33718/0 (socket says 172.21.15.171:33718) 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714dd9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f715009fba0 legacy=0x7f71500a0030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714e59c640 1 -- 172.21.15.171:0/3671610384 learned_addr learned my addr 172.21.15.171:0/3671610384 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:30.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849709270 0 0) 0x7f715009d1b0 con 0x7f715009f7c0 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient(hunting): my global_id is 4798 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7138004630 con 0x7f715009f7c0 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118857650 0 0) 0x7f7138004630 con 0x7f715009f7c0 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 >> v1:172.21.15.148:6790/0 conn(0x7f71501395d0 legacy=0x7f715013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 >> v1:172.21.15.148:6789/0 conn(0x7f715009fba0 legacy=0x7f71500a0030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 monclient: found mon.noname-c 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715009bca0 con 0x7f715009f7c0 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: _finish_auth 0 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 5 monclient: authenticate success, global_id 4798 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7140001610 con 0x7f715009f7c0 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: dump: 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.574 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7140001970 con 0x7f715009f7c0 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f7154a65640 4 set_mon_vals no callback set 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 1 -- 172.21.15.171:0/3671610384 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f71400048a0 con 0x7f715009f7c0 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:30.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.563+0000 7f714d59a640 10 monclient: dump: 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 10 monclient: shutdown 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:30.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 1 -- 172.21.15.171:0/3671610384 >> v1:172.21.15.171:6789/0 conn(0x7f715009f7c0 legacy=0x7f715009de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 1 -- 172.21.15.171:0/3671610384 shutdown_connections 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 1 -- 172.21.15.171:0/3671610384 >> 172.21.15.171:0/3671610384 conn(0x7f71500a5450 msgr2=0x7f71500a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 1 -- 172.21.15.171:0/3671610384 shutdown_connections 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.564+0000 7f7154a65640 1 -- 172.21.15.171:0/3671610384 wait complete. 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient: build_initial_monmap 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient: monmap: 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:30.577 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 1 Processor -- start 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 1 -- start start 2024-09-16T04:08:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient: init 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient(hunting): picked mon.noname-c con 0x7f715009f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient(hunting): picked mon.noname-a con 0x7f715009fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient(hunting): picked mon.noname-b con 0x7f71501395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7150136010 con 0x7f715009fba0 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7150124a90 con 0x7f71501395d0 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71500a3df0 con 0x7f715009f7c0 2024-09-16T04:08:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.565+0000 7f7154a65640 10 monclient(hunting): authenticate will time out at 886.070988s 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f714dd9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f715009fba0 legacy=0x7f71501d09d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f714ed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f71501395d0 0x7f71501d4100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33726/0 (socket says 172.21.15.171:33726) 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f714ed9d640 1 -- 172.21.15.171:0/1954497251 learned_addr learned my addr 172.21.15.171:0/1954497251 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226822244 0 0) 0x7f7150124a90 con 0x7f71501395d0 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): my global_id is 4801 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7130003520 con 0x7f71501395d0 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1858753873 0 0) 0x7f71500a3df0 con 0x7f715009f7c0 2024-09-16T04:08:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): my global_id is 4826 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7150124a90 con 0x7f715009f7c0 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 221274975 0 0) 0x7f7130003520 con 0x7f71501395d0 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71500a3df0 con 0x7f71501395d0 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7148001cc0 con 0x7f71501395d0 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): dump: 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1647077082 0 0) 0x7f71500a3df0 con 0x7f71501395d0 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 >> v1:172.21.15.148:6790/0 conn(0x7f715009f7c0 legacy=0x7f71501cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 >> v1:172.21.15.148:6789/0 conn(0x7f715009fba0 legacy=0x7f71501d09d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 monclient: found mon.b 2024-09-16T04:08:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71501238c0 con 0x7f71501395d0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: _finish_auth 0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 5 monclient: authenticate success, global_id 4801 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 10 monclient: _renew_subs 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7150124d70 con 0x7f71501395d0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 10 monclient: _renew_subs 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7150123640 con 0x7f71501395d0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7148002070 con 0x7f71501395d0 2024-09-16T04:08:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7148003e60 con 0x7f71501395d0 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f714cd99640 4 set_mon_vals no callback set 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.566+0000 7f713effd640 10 monclient: dump: 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7148004000 con 0x7f71501395d0 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f7154a65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f7154a65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f7154a65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7134005080 con 0x7f71501395d0 2024-09-16T04:08:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(57..57 src has 1..57) ==== 4377+0+0 (unknown 1358300759 0 0) 0x7f7148047330 con 0x7f71501395d0 2024-09-16T04:08:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.567+0000 7f714e59c640 10 monclient: get_auth_request con 0x7f7130038be0 auth_method 0 2024-09-16T04:08:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.572+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7148016080 con 0x7f71501395d0 2024-09-16T04:08:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.572+0000 7f713effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.572+0000 7f713effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:30.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.740+0000 7f7154a65640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:30.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.740+0000 7f7154a65640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:30.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.740+0000 7f7154a65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:30.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:30.740+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7f7134005370 con 0x7f71501395d0 2024-09-16T04:08:31.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.003+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v58) ==== 110+0+0 (unknown 444374933 0 0) 0x7f7148015980 con 0x7f71501395d0 2024-09-16T04:08:31.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.003+0000 7f713effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.003+0000 7f713effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:31.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.105+0000 7f7154a65640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.105+0000 7f7154a65640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.105+0000 7f7154a65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.105+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7f7134005bd0 con 0x7f71501395d0 2024-09-16T04:08:31.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.106+0000 7f713effd640 1 -- 172.21.15.171:0/1954497251 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v58) ==== 139+0+0 (unknown 2836895259 0 0) 0x7f714801e510 con 0x7f71501395d0 2024-09-16T04:08:31.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.106+0000 7f713effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.106+0000 7f713effd640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T04:08:31.108 INFO:tasks.workunit.client.0.smithi171.stderr:already renamed to 'foo-asdf' 2024-09-16T04:08:31.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.107+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 >> v1:172.21.15.171:6812/7327273 conn(0x7f7130038be0 legacy=0x7f713003b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 10 monclient: shutdown 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 >> v1:172.21.15.171:6789/0 conn(0x7f71501395d0 legacy=0x7f71501d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 shutdown_connections 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 >> 172.21.15.171:0/1954497251 conn(0x7f71500a5450 msgr2=0x7f7150138190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 shutdown_connections 2024-09-16T04:08:31.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.108+0000 7f7154a65640 1 -- 172.21.15.171:0/1954497251 wait complete. 2024-09-16T04:08:31.119 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-16T04:08:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.198+0000 7f5a82427640 10 monclient: get_monmap_and_config 2024-09-16T04:08:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.198+0000 7f5a82427640 10 monclient: build_initial_monmap 2024-09-16T04:08:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.198+0000 7f5a82427640 10 monclient: monmap: 2024-09-16T04:08:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:31.199462+0000 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:31.199462+0000 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:31.201 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 1 Processor -- start 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 1 -- start start 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient: init 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7c1338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient(hunting): picked mon.noname-b con 0x7f5a7c133cc0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7c1340d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:31.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c09b420 con 0x7f5a7c1340d0 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c0753d0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c09b140 con 0x7f5a7c133cc0 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.199+0000 7f5a82427640 10 monclient(hunting): authenticate will time out at 886.704990s 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5a7c1338e0 0x7f5a7c09be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33730/0 (socket says 172.21.15.171:33730) 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7bfff640 1 -- 172.21.15.171:0/3840920722 learned_addr learned my addr 172.21.15.171:0/3840920722 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a809c4640 1 -- 172.21.15.171:0/3840920722 >> v1:172.21.15.148:6789/0 conn(0x7f5a7c1340d0 legacy=0x7f5a7c13f220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377456190 0 0) 0x7f5a7c0753d0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient(hunting): my global_id is 4804 2024-09-16T04:08:31.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a64003520 con 0x7f5a7c1338e0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2450493331 0 0) 0x7f5a64003520 con 0x7f5a7c1338e0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 >> v1:172.21.15.148:6790/0 conn(0x7f5a7c133cc0 legacy=0x7f5a7c13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 >> v1:172.21.15.148:6789/0 conn(0x7f5a7c1340d0 legacy=0x7f5a7c13f220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 monclient: found mon.noname-c 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7c09b700 con 0x7f5a7c1338e0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: _finish_auth 0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a6c001610 con 0x7f5a7c1338e0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:31.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: dump: 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a82427640 5 monclient: authenticate success, global_id 4804 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a82427640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:31.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a6c001900 con 0x7f5a7c1338e0 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a82427640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a82427640 4 set_mon_vals no callback set 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 1 -- 172.21.15.171:0/3840920722 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a6c001d10 con 0x7f5a7c1338e0 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.200+0000 7f5a7affd640 10 monclient: dump: 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 10 monclient: shutdown 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 1 -- 172.21.15.171:0/3840920722 >> v1:172.21.15.171:6789/0 conn(0x7f5a7c1338e0 legacy=0x7f5a7c09be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 1 -- 172.21.15.171:0/3840920722 shutdown_connections 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 1 -- 172.21.15.171:0/3840920722 >> 172.21.15.171:0/3840920722 conn(0x7f5a7c0a6460 msgr2=0x7f5a7c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 1 -- 172.21.15.171:0/3840920722 shutdown_connections 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 1 -- 172.21.15.171:0/3840920722 wait complete. 2024-09-16T04:08:31.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 10 monclient: build_initial_monmap 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 10 monclient: monmap: 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:31.214 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.201+0000 7f5a82427640 1 Processor -- start 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 1 -- start start 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient: init 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient(hunting): picked mon.noname-b con 0x7f5a7c1338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient(hunting): picked mon.noname-a con 0x7f5a7c133cc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7c1340d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c133480 con 0x7f5a7c133cc0 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c1c8470 con 0x7f5a7c1338e0 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7c1329b0 con 0x7f5a7c1340d0 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a82427640 10 monclient(hunting): authenticate will time out at 886.707681s 2024-09-16T04:08:31.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a7bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5a7c1338e0 0x7f5a7c13fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33736/0 (socket says 172.21.15.171:33736) 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a7bfff640 1 -- 172.21.15.171:0/1347647078 learned_addr learned my addr 172.21.15.171:0/1347647078 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a7b7fe640 1 -- 172.21.15.171:0/1347647078 >> v1:172.21.15.148:6789/0 conn(0x7f5a7c133cc0 legacy=0x7f5a7c1d02b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2287131328 0 0) 0x7f5a7c1c8470 con 0x7f5a7c1338e0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a78ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a78ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a78ff9640 10 monclient(hunting): my global_id is 4807 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.202+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a68003520 con 0x7f5a7c1338e0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1435909547 0 0) 0x7f5a68003520 con 0x7f5a7c1338e0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a7c1c8470 con 0x7f5a7c1338e0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a6c0015d0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:31.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): dump: 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.217 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889292779 0 0) 0x7f5a7c1329b0 con 0x7f5a7c1340d0 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient(hunting): my global_id is 4832 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a68003520 con 0x7f5a7c1340d0 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1271603253 0 0) 0x7f5a7c1c8470 con 0x7f5a7c1338e0 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 >> v1:172.21.15.148:6790/0 conn(0x7f5a7c1340d0 legacy=0x7f5a7c1d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 >> v1:172.21.15.148:6789/0 conn(0x7f5a7c133cc0 legacy=0x7f5a7c1d02b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 monclient: found mon.b 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:31.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7c133110 con 0x7f5a7c1338e0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 5 monclient: authenticate success, global_id 4807 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 10 monclient: _renew_subs 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7c131650 con 0x7f5a7c1338e0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 10 monclient: _renew_subs 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5a7c1321b0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a6c001900 con 0x7f5a7c1338e0 2024-09-16T04:08:31.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:31.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5a6c004ee0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:31.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a7a7fc640 4 set_mon_vals no callback set 2024-09-16T04:08:31.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:31.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.203+0000 7f5a78ff9640 10 monclient: dump: 2024-09-16T04:08:31.221 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.221 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.221 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.221 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.221 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.221 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5a6c005080 con 0x7f5a7c1338e0 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a82427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a82427640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a82427640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a5c005080 con 0x7f5a7c1338e0 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4382+0+0 (unknown 3441618914 0 0) 0x7f5a6c047240 con 0x7f5a7c1338e0 2024-09-16T04:08:31.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.204+0000 7f5a7b7fe640 10 monclient: get_auth_request con 0x7f5a68038f70 auth_method 0 2024-09-16T04:08:31.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.208+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5a6c015f50 con 0x7f5a7c1338e0 2024-09-16T04:08:31.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.208+0000 7f5a78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:31.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.208+0000 7f5a78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.382+0000 7f5a82427640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.382+0000 7f5a82427640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.382+0000 7f5a82427640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.382+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7f5a5c005370 con 0x7f5a7c1338e0 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.383+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v58) ==== 139+0+0 (unknown 2836895259 0 0) 0x7f5a6c015850 con 0x7f5a7c1338e0 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.383+0000 7f5a78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.383+0000 7f5a78ff9640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-16T04:08:31.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.487+0000 7f5a82427640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.487+0000 7f5a82427640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.487+0000 7f5a82427640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.487+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) -- 0x7f5a5c000bf0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.488+0000 7f5a78ff9640 1 -- 172.21.15.171:0/1347647078 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v58) ==== 139+0+0 (unknown 2836895259 0 0) 0x7f5a6c01e3a0 con 0x7f5a7c1338e0 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.488+0000 7f5a78ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.488+0000 7f5a78ff9640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-16T04:08:31.490 INFO:tasks.workunit.client.0.smithi171.stderr:already renamed to 'foo-asdf' 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 >> v1:172.21.15.171:6812/7327273 conn(0x7f5a68038f70 legacy=0x7f5a6803b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 10 monclient: shutdown 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 >> v1:172.21.15.171:6789/0 conn(0x7f5a7c1338e0 legacy=0x7f5a7c13fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 shutdown_connections 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 >> 172.21.15.171:0/1347647078 conn(0x7f5a7c0a6460 msgr2=0x7f5a7c129700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 shutdown_connections 2024-09-16T04:08:31.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.490+0000 7f5a82427640 1 -- 172.21.15.171:0/1347647078 wait complete. 2024-09-16T04:08:31.501 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-16T04:08:31.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.579+0000 7f556fdf1640 10 monclient: get_monmap_and_config 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.579+0000 7f556fdf1640 10 monclient: build_initial_monmap 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.580+0000 7f556fdf1640 10 monclient: monmap: 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:31.580772+0000 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:31.580772+0000 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:31.582 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.580+0000 7f556fdf1640 1 Processor -- start 2024-09-16T04:08:31.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.580+0000 7f556fdf1640 1 -- start start 2024-09-16T04:08:31.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.580+0000 7f556fdf1640 10 monclient: init 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient(hunting): picked mon.noname-a con 0x7f55680a0ae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient(hunting): picked mon.noname-b con 0x7f556809f320 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient(hunting): picked mon.noname-c con 0x7f556813d2c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55680737b0 con 0x7f55680a0ae0 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556806c6d0 con 0x7f556813d2c0 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556806d980 con 0x7f556809f320 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 10 monclient(hunting): authenticate will time out at 887.086316s 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556e38e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f556813d2c0 0x7f556813f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33752/0 (socket says 172.21.15.171:33752) 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556db8d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f55680a0ae0 legacy=0x7f55680a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:31.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556e38e640 1 -- 172.21.15.171:0/2082485982 learned_addr learned my addr 172.21.15.171:0/2082485982 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2999900747 0 0) 0x7f556806c6d0 con 0x7f556813d2c0 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): my global_id is 4810 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5554003520 con 0x7f556813d2c0 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501317990 0 0) 0x7f556806d980 con 0x7f556809f320 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient(hunting): my global_id is 4835 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f556806c6d0 con 0x7f556809f320 2024-09-16T04:08:31.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3698925518 0 0) 0x7f5554003520 con 0x7f556813d2c0 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 >> v1:172.21.15.148:6790/0 conn(0x7f556809f320 legacy=0x7f556813bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 >> v1:172.21.15.148:6789/0 conn(0x7f55680a0ae0 legacy=0x7f55680a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 monclient: found mon.noname-c 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556806c950 con 0x7f556813d2c0 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: _finish_auth 0 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: _check_auth_tickets 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5560001990 con 0x7f556813d2c0 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 5 monclient: authenticate success, global_id 4810 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:31.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556cb8b640 10 monclient: dump: 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.594 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.581+0000 7f556fdf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5560001cb0 con 0x7f556813d2c0 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556cb8b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556cb8b640 1 -- 172.21.15.171:0/2082485982 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5560002080 con 0x7f556813d2c0 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 4 set_mon_vals no callback set 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556cb8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556cb8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556cb8b640 10 monclient: dump: 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.595 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 10 monclient: shutdown 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 1 -- 172.21.15.171:0/2082485982 >> v1:172.21.15.171:6789/0 conn(0x7f556813d2c0 legacy=0x7f556813f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 1 -- 172.21.15.171:0/2082485982 shutdown_connections 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 1 -- 172.21.15.171:0/2082485982 >> 172.21.15.171:0/2082485982 conn(0x7f55680a77f0 msgr2=0x7f55680a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 1 -- 172.21.15.171:0/2082485982 shutdown_connections 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.582+0000 7f556fdf1640 1 -- 172.21.15.171:0/2082485982 wait complete. 2024-09-16T04:08:31.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient: build_initial_monmap 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient: monmap: 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 1 Processor -- start 2024-09-16T04:08:31.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 1 -- start start 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient: init 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient(hunting): picked mon.noname-c con 0x7f556809f320 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient(hunting): picked mon.noname-b con 0x7f55680a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.583+0000 7f556fdf1640 10 monclient(hunting): picked mon.noname-a con 0x7f556813d2c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55680a22f0 con 0x7f556813d2c0 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5568127290 con 0x7f55680a0ae0 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5568127570 con 0x7f556809f320 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 10 monclient(hunting): authenticate will time out at 887.089045s 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556e38e640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f556813d2c0 legacy=0x7f55681d3e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:31.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556d38c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f55680a0ae0 0x7f55681d06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33768/0 (socket says 172.21.15.171:33768) 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556d38c640 1 -- 172.21.15.171:0/2328069035 learned_addr learned my addr 172.21.15.171:0/2328069035 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1226901150 0 0) 0x7f5568127290 con 0x7f55680a0ae0 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): my global_id is 4813 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f554c003520 con 0x7f55680a0ae0 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476099777 0 0) 0x7f5568127570 con 0x7f556809f320 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): my global_id is 4838 2024-09-16T04:08:31.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5568127290 con 0x7f556809f320 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1637992368 0 0) 0x7f554c003520 con 0x7f55680a0ae0 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5568127570 con 0x7f55680a0ae0 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5558001910 con 0x7f55680a0ae0 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): dump: 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 901699814 0 0) 0x7f5568127570 con 0x7f55680a0ae0 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 >> v1:172.21.15.148:6790/0 conn(0x7f556809f320 legacy=0x7f556813be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 >> v1:172.21.15.148:6789/0 conn(0x7f556813d2c0 legacy=0x7f55681d3e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 monclient: found mon.b 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:31.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5568127850 con 0x7f55680a0ae0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 5 monclient: authenticate success, global_id 4813 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 10 monclient: _renew_subs 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5568126fb0 con 0x7f55680a0ae0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 10 monclient: _renew_subs 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f556813c580 con 0x7f55680a0ae0 2024-09-16T04:08:31.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5558001ce0 con 0x7f55680a0ae0 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5558003a60 con 0x7f55680a0ae0 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555e7fc640 10 monclient: dump: 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:31.603 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.584+0000 7f555ffff640 4 set_mon_vals no callback set 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5558003c00 con 0x7f55680a0ae0 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f556fdf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f556fdf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f556fdf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5550005080 con 0x7f55680a0ae0 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4382+0+0 (unknown 3441618914 0 0) 0x7f5558047020 con 0x7f55680a0ae0 2024-09-16T04:08:31.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.585+0000 7f556db8d640 10 monclient: get_auth_request con 0x7f554c038b90 auth_method 0 2024-09-16T04:08:31.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.590+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5558015d10 con 0x7f55680a0ae0 2024-09-16T04:08:31.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.590+0000 7f555e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:31.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.590+0000 7f555e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:31.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.756+0000 7f556fdf1640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T04:08:31.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.756+0000 7f556fdf1640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T04:08:31.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.756+0000 7f556fdf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:31.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:31.756+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) -- 0x7f5550005370 con 0x7f55680a0ae0 2024-09-16T04:08:32.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.007+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v59) ==== 110+0+0 (unknown 3512777995 0 0) 0x7f5558015610 con 0x7f55680a0ae0 2024-09-16T04:08:32.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.007+0000 7f555e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T04:08:32.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.007+0000 7f555e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:32.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.110+0000 7f556fdf1640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T04:08:32.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.110+0000 7f556fdf1640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T04:08:32.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.110+0000 7f556fdf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.110+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) -- 0x7f5550005bd0 con 0x7f55680a0ae0 2024-09-16T04:08:32.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.111+0000 7f555e7fc640 1 -- 172.21.15.171:0/2328069035 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v59) ==== 139+0+0 (unknown 2421382388 0 0) 0x7f5558019940 con 0x7f55680a0ae0 2024-09-16T04:08:32.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.111+0000 7f555e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-16T04:08:32.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.111+0000 7f555e7fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-16T04:08:32.113 INFO:tasks.workunit.client.0.smithi171.stderr:already renamed to 'bar-asdf' 2024-09-16T04:08:32.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.112+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 >> v1:172.21.15.171:6812/7327273 conn(0x7f554c038b90 legacy=0x7f554c03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.112+0000 7f556fdf1640 10 monclient: shutdown 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.112+0000 7f556fdf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.112+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 >> v1:172.21.15.171:6789/0 conn(0x7f55680a0ae0 legacy=0x7f55681d06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.112+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 shutdown_connections 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.112+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 >> 172.21.15.171:0/2328069035 conn(0x7f55680a77f0 msgr2=0x7f556809ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.113+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 shutdown_connections 2024-09-16T04:08:32.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.113+0000 7f556fdf1640 1 -- 172.21.15.171:0/2328069035 wait complete. 2024-09-16T04:08:32.124 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls 2024-09-16T04:08:32.124 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo-asdf 2024-09-16T04:08:32.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.201+0000 7f418e2ba640 10 monclient: get_monmap_and_config 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.201+0000 7f418e2ba640 10 monclient: build_initial_monmap 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.201+0000 7f418e2ba640 10 monclient: monmap: 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:32.201823+0000 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:32.201823+0000 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:32.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.201+0000 7f418e2ba640 1 Processor -- start 2024-09-16T04:08:32.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.201+0000 7f418e2ba640 1 -- start start 2024-09-16T04:08:32.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.201+0000 7f418e2ba640 10 monclient: init 2024-09-16T04:08:32.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4188136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4188139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient(hunting): picked mon.noname-c con 0x7f418813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41880a1340 con 0x7f4188139b10 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4188137650 con 0x7f418813d620 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41880a0aa0 con 0x7f4188136b60 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 10 monclient(hunting): authenticate will time out at 887.707359s 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f41877fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4188139b10 legacy=0x7f418813bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418c857640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f418813d620 0x7f418813fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33778/0 (socket says 172.21.15.171:33778) 2024-09-16T04:08:32.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418c857640 1 -- 172.21.15.171:0/2431577086 learned_addr learned my addr 172.21.15.171:0/2431577086 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161175744 0 0) 0x7f4188137650 con 0x7f418813d620 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): my global_id is 4816 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4170003520 con 0x7f418813d620 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324417175 0 0) 0x7f41880a0aa0 con 0x7f4188136b60 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient(hunting): my global_id is 4841 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4188137650 con 0x7f4188136b60 2024-09-16T04:08:32.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2316031049 0 0) 0x7f4170003520 con 0x7f418813d620 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 >> v1:172.21.15.148:6790/0 conn(0x7f4188136b60 legacy=0x7f4188136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 >> v1:172.21.15.148:6789/0 conn(0x7f4188139b10 legacy=0x7f418813bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 monclient: found mon.noname-c 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41880a0550 con 0x7f418813d620 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4180001990 con 0x7f418813d620 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 5 monclient: authenticate success, global_id 4816 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:32.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f4186ffd640 10 monclient: dump: 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.202+0000 7f418e2ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:32.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4180001cb0 con 0x7f418813d620 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f4186ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f4186ffd640 1 -- 172.21.15.171:0/2431577086 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4180002080 con 0x7f418813d620 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f4186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 4 set_mon_vals no callback set 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f4186ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f4186ffd640 10 monclient: dump: 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.216 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 10 monclient: shutdown 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 1 -- 172.21.15.171:0/2431577086 >> v1:172.21.15.171:6789/0 conn(0x7f418813d620 legacy=0x7f418813fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 1 -- 172.21.15.171:0/2431577086 shutdown_connections 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 1 -- 172.21.15.171:0/2431577086 >> 172.21.15.171:0/2431577086 conn(0x7f41880a98b0 msgr2=0x7f41880a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 1 -- 172.21.15.171:0/2431577086 shutdown_connections 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.203+0000 7f418e2ba640 1 -- 172.21.15.171:0/2431577086 wait complete. 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.204+0000 7f418e2ba640 10 monclient: build_initial_monmap 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.204+0000 7f418e2ba640 10 monclient: monmap: 2024-09-16T04:08:32.217 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:32.218 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.204+0000 7f418e2ba640 1 Processor -- start 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.204+0000 7f418e2ba640 1 -- start start 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.204+0000 7f418e2ba640 10 monclient: init 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4188136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4188139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient(hunting): picked mon.noname-a con 0x7f418813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:32.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41881204a0 con 0x7f418813d620 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4188120780 con 0x7f4188139b10 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4188136700 con 0x7f4188136b60 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient(hunting): authenticate will time out at 887.710223s 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418c857640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f418813d620 legacy=0x7f41881d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f41877fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4188139b10 0x7f41881d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33784/0 (socket says 172.21.15.171:33784) 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f41877fe640 1 -- 172.21.15.171:0/2619613866 learned_addr learned my addr 172.21.15.171:0/2619613866 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182932206 0 0) 0x7f4188120780 con 0x7f4188139b10 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): my global_id is 4819 2024-09-16T04:08:32.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4174003520 con 0x7f4188139b10 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588473308 0 0) 0x7f4188136700 con 0x7f4188136b60 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): my global_id is 4844 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4188120780 con 0x7f4188136b60 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3195540264 0 0) 0x7f4174003520 con 0x7f4188139b10 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4188136700 con 0x7f4188139b10 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f417c004180 con 0x7f4188139b10 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): dump: 2024-09-16T04:08:32.221 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1256943559 0 0) 0x7f4188120780 con 0x7f4188136b60 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4174003520 con 0x7f4188136b60 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41780018b0 con 0x7f4188136b60 2024-09-16T04:08:32.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient(hunting): dump: 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1224755104 0 0) 0x7f4188136700 con 0x7f4188139b10 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 >> v1:172.21.15.148:6790/0 conn(0x7f4188136b60 legacy=0x7f41881cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 >> v1:172.21.15.148:6789/0 conn(0x7f418813d620 legacy=0x7f41881d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 monclient: found mon.b 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418809ee90 con 0x7f4188139b10 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f4184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 5 monclient: authenticate success, global_id 4819 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient: _renew_subs 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.205+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f418809f2e0 con 0x7f4188139b10 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 10 monclient: _renew_subs 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f418809e200 con 0x7f4188139b10 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f417c0043e0 con 0x7f4188139b10 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f417c004730 con 0x7f4188139b10 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f41867fc640 4 set_mon_vals no callback set 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 10 monclient: dump: 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.225 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f417c004950 con 0x7f4188139b10 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4168005080 con 0x7f4188139b10 2024-09-16T04:08:32.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.206+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7f417c047070 con 0x7f4188139b10 2024-09-16T04:08:32.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.207+0000 7f4187fff640 10 monclient: get_auth_request con 0x7f4174038f10 auth_method 0 2024-09-16T04:08:32.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.211+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f417c015bf0 con 0x7f4188139b10 2024-09-16T04:08:32.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.211+0000 7f4184ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:32.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.211+0000 7f4184ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:32.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f418e2ba640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f418e2ba640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f418e2ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f4168005370 con 0x7f4188139b10 2024-09-16T04:08:32.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) ==== 65+0+70 (unknown 1101553479 0 3909614116) 0x7f417c0154f0 con 0x7f4188139b10 2024-09-16T04:08:32.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f4184ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.377+0000 7f4184ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:32.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f418e2ba640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f418e2ba640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f418e2ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f4168000bf0 con 0x7f4188139b10 2024-09-16T04:08:32.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f4184ff9640 1 -- 172.21.15.171:0/2619613866 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) ==== 65+0+70 (unknown 1101553479 0 3909614116) 0x7f417c015750 con 0x7f4188139b10 2024-09-16T04:08:32.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f4184ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.477+0000 7f4184ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:32.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.479+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 >> v1:172.21.15.171:6812/7327273 conn(0x7f4174038f10 legacy=0x7f417403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.479+0000 7f418e2ba640 10 monclient: shutdown 2024-09-16T04:08:32.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.479+0000 7f418e2ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:32.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.479+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 >> v1:172.21.15.171:6789/0 conn(0x7f4188139b10 legacy=0x7f41881d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.479+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 shutdown_connections 2024-09-16T04:08:32.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.479+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 >> 172.21.15.171:0/2619613866 conn(0x7f41880a98b0 msgr2=0x7f4188139200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:32.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.480+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 shutdown_connections 2024-09-16T04:08:32.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.480+0000 7f418e2ba640 1 -- 172.21.15.171:0/2619613866 wait complete. 2024-09-16T04:08:32.491 INFO:tasks.workunit.client.0.smithi171.stdout:foo-asdf 2024-09-16T04:08:32.491 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls 2024-09-16T04:08:32.491 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep bar-asdf 2024-09-16T04:08:32.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.568+0000 7f28d5f6d640 10 monclient: get_monmap_and_config 2024-09-16T04:08:32.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.568+0000 7f28d5f6d640 10 monclient: build_initial_monmap 2024-09-16T04:08:32.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.568+0000 7f28d5f6d640 10 monclient: monmap: 2024-09-16T04:08:32.570 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:32.569462+0000 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:32.569462+0000 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:32.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 1 Processor -- start 2024-09-16T04:08:32.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 1 -- start start 2024-09-16T04:08:32.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient: init 2024-09-16T04:08:32.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:32.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f28d0132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f28d009ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f28d0132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d01332b0 con 0x7f28d0132e60 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d0134310 con 0x7f28d0132a80 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d009d980 con 0x7f28d009ec20 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.569+0000 7f28d5f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28d5f6d640 10 monclient(hunting): authenticate will time out at 888.075004s 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28cffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f28d0132e60 legacy=0x7f28d013c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28cf7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f28d0132a80 0x7f28d01334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33786/0 (socket says 172.21.15.171:33786) 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28cf7fe640 1 -- 172.21.15.171:0/4225378288 learned_addr learned my addr 172.21.15.171:0/4225378288 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2440331749 0 0) 0x7f28d0134310 con 0x7f28d0132a80 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient(hunting): my global_id is 4822 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28b8003520 con 0x7f28d0132a80 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2932539800 0 0) 0x7f28b8003520 con 0x7f28d0132a80 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 >> v1:172.21.15.148:6790/0 conn(0x7f28d009ec20 legacy=0x7f28d0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 >> v1:172.21.15.148:6789/0 conn(0x7f28d0132e60 legacy=0x7f28d013c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d009dc60 con 0x7f28d0132a80 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:32.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28c0001610 con 0x7f28d0132a80 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28d5f6d640 5 monclient: authenticate success, global_id 4822 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: dump: 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.581 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28d5f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28c0001900 con 0x7f28d0132a80 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 1 -- 172.21.15.171:0/4225378288 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28c0001cd0 con 0x7f28d0132a80 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28d5f6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28d5f6d640 4 set_mon_vals no callback set 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.570+0000 7f28ce7fc640 10 monclient: dump: 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.582 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 10 monclient: shutdown 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 1 -- 172.21.15.171:0/4225378288 >> v1:172.21.15.171:6789/0 conn(0x7f28d0132a80 legacy=0x7f28d01334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 1 -- 172.21.15.171:0/4225378288 shutdown_connections 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 1 -- 172.21.15.171:0/4225378288 >> 172.21.15.171:0/4225378288 conn(0x7f28d00a52a0 msgr2=0x7f28d00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 1 -- 172.21.15.171:0/4225378288 shutdown_connections 2024-09-16T04:08:32.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 1 -- 172.21.15.171:0/4225378288 wait complete. 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 10 monclient: build_initial_monmap 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.571+0000 7f28d5f6d640 10 monclient: monmap: 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 1 Processor -- start 2024-09-16T04:08:32.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 1 -- start start 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient: init 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f28d009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f28d0132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f28d0132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d0124c80 con 0x7f28d009ec20 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d0124f60 con 0x7f28d0132e60 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28d00a3dd0 con 0x7f28d0132a80 2024-09-16T04:08:32.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28cf7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f28d009ec20 legacy=0x7f28d01cb680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28d5f6d640 10 monclient(hunting): authenticate will time out at 888.077715s 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28cffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f28d0132e60 0x7f28d01d3f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33790/0 (socket says 172.21.15.171:33790) 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28cffff640 1 -- 172.21.15.171:0/3602868989 learned_addr learned my addr 172.21.15.171:0/3602868989 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365180990 0 0) 0x7f28d0124f60 con 0x7f28d0132e60 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28b7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28b7fff640 10 monclient(hunting): my global_id is 4825 2024-09-16T04:08:32.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.572+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28bc003520 con 0x7f28d0132e60 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434759263 0 0) 0x7f28d00a3dd0 con 0x7f28d0132a80 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): my global_id is 4850 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28d0124f60 con 0x7f28d0132a80 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3060419297 0 0) 0x7f28bc003520 con 0x7f28d0132e60 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28d00a3dd0 con 0x7f28d0132e60 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28c8004440 con 0x7f28d0132e60 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): dump: 2024-09-16T04:08:32.587 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1462607203 0 0) 0x7f28d00a3dd0 con 0x7f28d0132e60 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 >> v1:172.21.15.148:6790/0 conn(0x7f28d0132a80 legacy=0x7f28d01d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 >> v1:172.21.15.148:6789/0 conn(0x7f28d009ec20 legacy=0x7f28d01cb680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:32.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 monclient: found mon.b 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d009c980 con 0x7f28d0132e60 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 5 monclient: authenticate success, global_id 4825 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 10 monclient: _renew_subs 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28d0123ab0 con 0x7f28d0132e60 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 10 monclient: _renew_subs 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f28d0131640 con 0x7f28d0132e60 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28c8004760 con 0x7f28d0132e60 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28c8004b30 con 0x7f28d0132e60 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28cdffb640 4 set_mon_vals no callback set 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.573+0000 7f28b7fff640 10 monclient: dump: 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:32.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f28c8004d70 con 0x7f28d0132e60 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28d5f6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28d5f6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28d5f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b0005080 con 0x7f28d0132e60 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7f28c8047470 con 0x7f28d0132e60 2024-09-16T04:08:32.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.574+0000 7f28cf7fe640 10 monclient: get_auth_request con 0x7f28bc038f60 auth_method 0 2024-09-16T04:08:32.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.578+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f28c8016050 con 0x7f28d0132e60 2024-09-16T04:08:32.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.578+0000 7f28b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:32.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.578+0000 7f28b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28d5f6d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28d5f6d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28d5f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f28b0005370 con 0x7f28d0132e60 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) ==== 65+0+70 (unknown 1101553479 0 3909614116) 0x7f28c8015950 con 0x7f28d0132e60 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.750+0000 7f28b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.855+0000 7f28d5f6d640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.855+0000 7f28d5f6d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.855+0000 7f28d5f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.855+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f28b0000bc0 con 0x7f28d0132e60 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.856+0000 7f28b7fff640 1 -- 172.21.15.171:0/3602868989 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v59) ==== 65+0+70 (unknown 1101553479 0 3909614116) 0x7f28c801e500 con 0x7f28d0132e60 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.856+0000 7f28b7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:32.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.856+0000 7f28b7fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 >> v1:172.21.15.171:6812/7327273 conn(0x7f28bc038f60 legacy=0x7f28bc03b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 10 monclient: shutdown 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 >> v1:172.21.15.171:6789/0 conn(0x7f28d0132e60 legacy=0x7f28d01d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 shutdown_connections 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 >> 172.21.15.171:0/3602868989 conn(0x7f28d00a52a0 msgr2=0x7f28d0131f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:32.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 shutdown_connections 2024-09-16T04:08:32.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:32.858+0000 7f28d5f6d640 1 -- 172.21.15.171:0/3602868989 wait complete. 2024-09-16T04:08:32.869 INFO:tasks.workunit.client.0.smithi171.stdout:bar-asdf 2024-09-16T04:08:32.869 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm foo 2024-09-16T04:08:32.869 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'does not exist' 2024-09-16T04:08:32.938 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:08:32.938 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:08:32.938 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:08:32.938 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:08:32.941 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:08:32.942 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:08:32.942 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:33.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.049+0000 7f888a682640 10 monclient: get_monmap_and_config 2024-09-16T04:08:33.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.049+0000 7f888a682640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.049+0000 7f888a682640 10 monclient: monmap: 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:33.050700+0000 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:33.050700+0000 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:33.052 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.050+0000 7f888a682640 1 Processor -- start 2024-09-16T04:08:33.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.050+0000 7f888a682640 1 -- start start 2024-09-16T04:08:33.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.050+0000 7f888a682640 10 monclient: init 2024-09-16T04:08:33.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:33.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient(hunting): picked mon.noname-b con 0x7f88840a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient(hunting): picked mon.noname-c con 0x7f88840a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient(hunting): picked mon.noname-a con 0x7f888413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88840a0e00 con 0x7f888413d060 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88840a10e0 con 0x7f88840a1b10 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88840a4970 con 0x7f88840a2410 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f888a682640 10 monclient(hunting): authenticate will time out at 933.676224s 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8883fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88840a2410 0x7f88840a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59266/0 (socket says 172.21.15.148:59266) 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8883fff640 1 -- 172.21.15.148:0/855060484 learned_addr learned my addr 172.21.15.148:0/855060484 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584561213 0 0) 0x7f88840a10e0 con 0x7f88840a1b10 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): my global_id is 4834 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f886c003520 con 0x7f88840a1b10 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3970923032 0 0) 0x7f88840a4970 con 0x7f88840a2410 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient(hunting): my global_id is 4859 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88840a10e0 con 0x7f88840a2410 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 289435094 0 0) 0x7f886c003520 con 0x7f88840a1b10 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 >> v1:172.21.15.148:6790/0 conn(0x7f88840a2410 legacy=0x7f88840a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 >> v1:172.21.15.148:6789/0 conn(0x7f888413d060 legacy=0x7f888413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 monclient: found mon.noname-c 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888406dfd0 con 0x7f88840a1b10 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8878001540 con 0x7f88840a1b10 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.051+0000 7f8882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 10 monclient: dump: 2024-09-16T04:08:33.058 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 5 monclient: authenticate success, global_id 4834 2024-09-16T04:08:33.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8878001860 con 0x7f88840a1b10 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 1 -- 172.21.15.148:0/855060484 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8878001c70 con 0x7f88840a1b10 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 4 set_mon_vals no callback set 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f8882ffd640 10 monclient: dump: 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.060 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 10 monclient: shutdown 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 1 -- 172.21.15.148:0/855060484 >> v1:172.21.15.171:6789/0 conn(0x7f88840a1b10 legacy=0x7f888413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.052+0000 7f888a682640 1 -- 172.21.15.148:0/855060484 shutdown_connections 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.053+0000 7f888a682640 1 -- 172.21.15.148:0/855060484 >> 172.21.15.148:0/855060484 conn(0x7f88840a97b0 msgr2=0x7f88840a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.053+0000 7f888a682640 1 -- 172.21.15.148:0/855060484 shutdown_connections 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.053+0000 7f888a682640 1 -- 172.21.15.148:0/855060484 wait complete. 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.053+0000 7f888a682640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.053+0000 7f888a682640 10 monclient: monmap: 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:33.062 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 1 Processor -- start 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 1 -- start start 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 10 monclient: init 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 10 monclient(hunting): picked mon.noname-c con 0x7f88840a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 10 monclient(hunting): picked mon.noname-b con 0x7f88840a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 10 monclient(hunting): picked mon.noname-a con 0x7f888413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888413b7b0 con 0x7f888413d060 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8884137b50 con 0x7f88840a2410 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.054+0000 7f888a682640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888409d670 con 0x7f88840a1b10 2024-09-16T04:08:33.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.055+0000 7f888a682640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.055+0000 7f8883fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88840a1b10 0x7f888409c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59272/0 (socket says 172.21.15.148:59272) 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.055+0000 7f8883fff640 1 -- 172.21.15.148:0/184870834 learned_addr learned my addr 172.21.15.148:0/184870834 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.055+0000 7f888a682640 10 monclient(hunting): authenticate will time out at 933.680032s 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227847429 0 0) 0x7f888409d670 con 0x7f88840a1b10 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): my global_id is 4862 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8870003520 con 0x7f88840a1b10 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176583158 0 0) 0x7f8884137b50 con 0x7f88840a2410 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): my global_id is 4837 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888409d670 con 0x7f88840a2410 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3938364450 0 0) 0x7f8870003520 con 0x7f88840a1b10 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8884137b50 con 0x7f88840a1b10 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8874001970 con 0x7f88840a1b10 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): dump: 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.073 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2755768409 0 0) 0x7f888409d670 con 0x7f88840a2410 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8870003520 con 0x7f88840a2410 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8878001bb0 con 0x7f88840a2410 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient(hunting): dump: 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3449653138 0 0) 0x7f8884137b50 con 0x7f88840a1b10 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 >> v1:172.21.15.171:6789/0 conn(0x7f88840a2410 legacy=0x7f888409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 >> v1:172.21.15.148:6789/0 conn(0x7f888413d060 legacy=0x7f88841d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 monclient: found mon.c 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888413ccb0 con 0x7f88840a1b10 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 5 monclient: authenticate success, global_id 4862 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 10 monclient: _renew_subs 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88840a0130 con 0x7f88840a1b10 2024-09-16T04:08:33.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 10 monclient: _renew_subs 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f888409edd0 con 0x7f88840a1b10 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.058+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88740047e0 con 0x7f88840a1b10 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.059+0000 7f8880ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.059+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8874004b30 con 0x7f88840a1b10 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.059+0000 7f8880ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.059+0000 7f8880ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.059+0000 7f8880ff9640 10 monclient: dump: 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.077 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.059+0000 7f88827fc640 4 set_mon_vals no callback set 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8874004cd0 con 0x7f88840a1b10 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f888a682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f888a682640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f888a682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88840008d0 con 0x7f88840a1b10 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7f8874046fa0 con 0x7f88840a1b10 2024-09-16T04:08:33.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.060+0000 7f88837fe640 10 monclient: get_auth_request con 0x7f8870038f20 auth_method 0 2024-09-16T04:08:33.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.065+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8874015b20 con 0x7f88840a1b10 2024-09-16T04:08:33.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.065+0000 7f8880ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.065+0000 7f8880ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:33.258 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.140+0000 7f77977fe640 1 -- 172.21.15.171:0/874520442 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v59) ==== 103+0+0 (unknown 3092499332 0 0) 0x7f77a40198f0 con 0x7f77b009b660 2024-09-16T04:08:33.258 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.140+0000 7f77977fe640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T04:08:33.258 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.245+0000 7f77977fe640 1 -- 172.21.15.171:0/874520442 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v59) ==== 103+0+0 (unknown 3092499332 0 0) 0x7f77a401e1b0 con 0x7f77b009b660 2024-09-16T04:08:33.258 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.245+0000 7f77977fe640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T04:08:33.258 INFO:tasks.workunit.client.0.smithi171.stdout:rule foo does not exist 2024-09-16T04:08:33.259 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm bar 2024-09-16T04:08:33.259 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'does not exist' 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f888a682640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f888a682640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f888a682640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f888413c5e0 con 0x7f88840a1b10 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f8880ff9640 1 -- 172.21.15.148:0/184870834 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2396920843) 0x7f8874015420 con 0x7f88840a1b10 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f8880ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.298+0000 7f8880ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:33.301 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.300+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 >> v1:172.21.15.171:6812/7327273 conn(0x7f8870038f20 legacy=0x7f887003b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.300+0000 7f888a682640 10 monclient: shutdown 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.300+0000 7f888a682640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.300+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 >> v1:172.21.15.148:6790/0 conn(0x7f88840a1b10 legacy=0x7f888409c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.300+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 shutdown_connections 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.300+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 >> 172.21.15.148:0/184870834 conn(0x7f88840a97b0 msgr2=0x7f8884135ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.301+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 shutdown_connections 2024-09-16T04:08:33.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.301+0000 7f888a682640 1 -- 172.21.15.148:0/184870834 wait complete. 2024-09-16T04:08:33.313 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:08:33.313 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:08:33.313 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:08:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.410+0000 7fef81133640 10 monclient: get_monmap_and_config 2024-09-16T04:08:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.410+0000 7fef81133640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.410+0000 7fef81133640 10 monclient: monmap: 2024-09-16T04:08:33.433 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.433 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.433 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:33.411532+0000 2024-09-16T04:08:33.433 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:33.411532+0000 2024-09-16T04:08:33.433 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.434 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.434 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.434 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:33.434 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:33.434 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 1 Processor -- start 2024-09-16T04:08:33.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 1 -- start start 2024-09-16T04:08:33.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient: init 2024-09-16T04:08:33.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:33.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient(hunting): picked mon.noname-a con 0x7fef7c1369a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient(hunting): picked mon.noname-c con 0x7fef7c0a0e30 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient(hunting): picked mon.noname-b con 0x7fef7c13d0b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7c073800 con 0x7fef7c1369a0 2024-09-16T04:08:33.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7c073a40 con 0x7fef7c0a0e30 2024-09-16T04:08:33.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7c09b140 con 0x7fef7c13d0b0 2024-09-16T04:08:33.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef81133640 10 monclient(hunting): authenticate will time out at 934.036729s 2024-09-16T04:08:33.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef7ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fef7c1369a0 0x7fef7c136d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54710/0 (socket says 172.21.15.148:54710) 2024-09-16T04:08:33.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef7b59e640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fef7c13d0b0 0x7fef7c13f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59288/0 (socket says 172.21.15.148:59288) 2024-09-16T04:08:33.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.411+0000 7fef7ad9d640 1 -- 172.21.15.148:0/542786705 learned_addr learned my addr 172.21.15.148:0/542786705 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3256047085 0 0) 0x7fef7c09b140 con 0x7fef7c13d0b0 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): my global_id is 4871 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef64003520 con 0x7fef7c13d0b0 2024-09-16T04:08:33.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2611176778 0 0) 0x7fef7c073800 con 0x7fef7c1369a0 2024-09-16T04:08:33.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): my global_id is 54105 2024-09-16T04:08:33.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7c09b140 con 0x7fef7c1369a0 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963514526 0 0) 0x7fef7c073a40 con 0x7fef7c0a0e30 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient(hunting): my global_id is 4846 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7c073800 con 0x7fef7c0a0e30 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 815325235 0 0) 0x7fef64003520 con 0x7fef7c13d0b0 2024-09-16T04:08:33.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 >> v1:172.21.15.171:6789/0 conn(0x7fef7c0a0e30 legacy=0x7fef7c13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 >> v1:172.21.15.148:6789/0 conn(0x7fef7c1369a0 legacy=0x7fef7c136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 monclient: found mon.noname-b 2024-09-16T04:08:33.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7c09b420 con 0x7fef7c13d0b0 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef740018c0 con 0x7fef7c13d0b0 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 5 monclient: authenticate success, global_id 4871 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: dump: 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.442 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef74001af0 con 0x7fef7c13d0b0 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 1 -- 172.21.15.148:0/542786705 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef74001f00 con 0x7fef7c13d0b0 2024-09-16T04:08:33.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 4 set_mon_vals no callback set 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef79d9b640 10 monclient: dump: 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.444 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 10 monclient: shutdown 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 1 -- 172.21.15.148:0/542786705 >> v1:172.21.15.148:6790/0 conn(0x7fef7c13d0b0 legacy=0x7fef7c13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.412+0000 7fef81133640 1 -- 172.21.15.148:0/542786705 shutdown_connections 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.413+0000 7fef81133640 1 -- 172.21.15.148:0/542786705 >> 172.21.15.148:0/542786705 conn(0x7fef7c0a5640 msgr2=0x7fef7c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.413+0000 7fef81133640 1 -- 172.21.15.148:0/542786705 shutdown_connections 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.413+0000 7fef81133640 1 -- 172.21.15.148:0/542786705 wait complete. 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.413+0000 7fef81133640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.413+0000 7fef81133640 10 monclient: monmap: 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:33.446 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 1 Processor -- start 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 1 -- start start 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient: init 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient(hunting): picked mon.noname-b con 0x7fef7c0a0e30 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient(hunting): picked mon.noname-a con 0x7fef7c1369a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient(hunting): picked mon.noname-c con 0x7fef7c13d0b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7c11d9e0 con 0x7fef7c1369a0 2024-09-16T04:08:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7c11e8d0 con 0x7fef7c0a0e30 2024-09-16T04:08:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7c11ee90 con 0x7fef7c13d0b0 2024-09-16T04:08:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef81133640 10 monclient(hunting): authenticate will time out at 934.039761s 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef7b59e640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fef7c13d0b0 0x7fef7c1d3c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59292/0 (socket says 172.21.15.148:59292) 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef7b59e640 1 -- 172.21.15.148:0/1151547185 learned_addr learned my addr 172.21.15.148:0/1151547185 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.414+0000 7fef7a59c640 1 --1- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6789/0 conn(0x7fef7c1369a0 0x7fef7c1d04e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54712/0 (socket says 172.21.15.148:54712) 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652705511 0 0) 0x7fef7c11ee90 con 0x7fef7c13d0b0 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): my global_id is 4874 2024-09-16T04:08:33.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef68003520 con 0x7fef7c13d0b0 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243155447 0 0) 0x7fef7c11e8d0 con 0x7fef7c0a0e30 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): my global_id is 4849 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7c11ee90 con 0x7fef7c0a0e30 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571221456 0 0) 0x7fef7c11d9e0 con 0x7fef7c1369a0 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): my global_id is 54108 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7c11e8d0 con 0x7fef7c1369a0 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3865138877 0 0) 0x7fef7c11ee90 con 0x7fef7c0a0e30 2024-09-16T04:08:33.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef7c11d9e0 con 0x7fef7c0a0e30 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef6c001a10 con 0x7fef7c0a0e30 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): dump: 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.451 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1309163298 0 0) 0x7fef68003520 con 0x7fef7c13d0b0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef7c11ee90 con 0x7fef7c13d0b0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1969048709 0 0) 0x7fef7c11e8d0 con 0x7fef7c1369a0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef68003520 con 0x7fef7c1369a0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef740017c0 con 0x7fef7c13d0b0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): dump: 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.452 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef70001830 con 0x7fef7c1369a0 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient(hunting): dump: 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.453 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1025433332 0 0) 0x7fef7c11d9e0 con 0x7fef7c0a0e30 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6790/0 conn(0x7fef7c13d0b0 legacy=0x7fef7c1d3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6789/0 conn(0x7fef7c1369a0 legacy=0x7fef7c1d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 monclient: found mon.b 2024-09-16T04:08:33.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7c11d2e0 con 0x7fef7c0a0e30 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 5 monclient: authenticate success, global_id 4849 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 10 monclient: _renew_subs 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef7c11d760 con 0x7fef7c0a0e30 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 10 monclient: _renew_subs 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef7c11ce80 con 0x7fef7c0a0e30 2024-09-16T04:08:33.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef6c001d00 con 0x7fef7c0a0e30 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef6c004c30 con 0x7fef7c0a0e30 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef7959a640 4 set_mon_vals no callback set 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.415+0000 7fef637fe640 10 monclient: dump: 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.456 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef81133640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fef5c0065c0 con 0x7fef5c002860 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fef6c00fec0 con 0x7fef7c0a0e30 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7fef6c047050 con 0x7fef7c0a0e30 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418058840 0 0) 0x7fef5c0065c0 con 0x7fef5c002860 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.416+0000 7fef637fe640 10 monclient(hunting): my global_id is 54111 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7c11d9e0 con 0x7fef5c002860 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef7b59e640 10 monclient: get_auth_request con 0x7fef680391e0 auth_method 0 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 256297023 0 0) 0x7fef7c11d9e0 con 0x7fef5c002860 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef5c0065c0 con 0x7fef5c002860 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef700035f0 con 0x7fef5c002860 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 10 monclient: dump: 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.458 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3730141329 0 0) 0x7fef5c0065c0 con 0x7fef5c002860 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef5c00a150 con 0x7fef5c002860 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fef700035f0 con 0x7fef5c002860 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6789/0 conn(0x7fef5c002860 legacy=0x7fef5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.417+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6789/0 conn(0x7fef5c002860 legacy=0x7fef5c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef81133640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fef5c009e20 con 0x7fef5c0090e0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954100682 0 0) 0x7fef5c009e20 con 0x7fef5c0090e0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef637fe640 10 monclient(hunting): my global_id is 54114 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.423+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef5c0065c0 con 0x7fef5c0090e0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1251103612 0 0) 0x7fef5c0065c0 con 0x7fef5c0090e0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef5c009e20 con 0x7fef5c0090e0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef70003dc0 con 0x7fef5c0090e0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:33.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 10 monclient: dump: 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 418789877 0 0) 0x7fef5c009e20 con 0x7fef5c0090e0 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fef5c00c390 con 0x7fef5c0090e0 2024-09-16T04:08:33.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 720658548) 0x7fef70003dc0 con 0x7fef5c0090e0 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6789/0 conn(0x7fef5c0090e0 legacy=0x7fef5c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef637fe640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.148:6789/0 conn(0x7fef5c0090e0 legacy=0x7fef5c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:33.462 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 16, 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 394, 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:33.463 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:33.464 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.465 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:33.466 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.467 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.468 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:33.469 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.470 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.471 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.472 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:33.473 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.171:6812/7327273 conn(0x7fef680391e0 legacy=0x7fef6803b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 10 monclient: shutdown 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 >> v1:172.21.15.171:6789/0 conn(0x7fef7c0a0e30 legacy=0x7fef7c133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef7b59e640 1 -- 172.21.15.148:0/1151547185 reap_dead start 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef7b59e640 1 -- 172.21.15.148:0/1151547185 reap_dead start 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 shutdown_connections 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 >> 172.21.15.148:0/1151547185 conn(0x7fef7c0a5640 msgr2=0x7fef7c0a4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 shutdown_connections 2024-09-16T04:08:33.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.424+0000 7fef81133640 1 -- 172.21.15.148:0/1151547185 wait complete. 2024-09-16T04:08:33.476 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:33.476 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:08:33.476 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:08:33.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.573+0000 7ffa684a5640 10 monclient: get_monmap_and_config 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.573+0000 7ffa684a5640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.573+0000 7ffa684a5640 10 monclient: monmap: 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:33.574832+0000 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:33.574832+0000 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:33.576 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.574+0000 7ffa684a5640 1 Processor -- start 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.574+0000 7ffa684a5640 1 -- start start 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.574+0000 7ffa684a5640 10 monclient: init 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient(hunting): picked mon.noname-b con 0x7ffa6009ebd0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa6009efb0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa6009f420 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa60137290 con 0x7ffa6009efb0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa600a1910 con 0x7ffa6009f420 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6009cbd0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient(hunting): authenticate will time out at 934.200171s 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa66241640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ffa6009ebd0 0x7ffa60136b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59296/0 (socket says 172.21.15.148:59296) 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa65a40640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ffa6009efb0 0x7ffa6013bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54744/0 (socket says 172.21.15.148:54744) 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa66241640 1 -- 172.21.15.148:0/550670757 learned_addr learned my addr 172.21.15.148:0/550670757 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2845933734 0 0) 0x7ffa60137290 con 0x7ffa6009efb0 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): my global_id is 54117 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa48003520 con 0x7ffa6009efb0 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332414647 0 0) 0x7ffa6009cbd0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): my global_id is 4877 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa60137290 con 0x7ffa6009ebd0 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4275126785 0 0) 0x7ffa600a1910 con 0x7ffa6009f420 2024-09-16T04:08:33.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient(hunting): my global_id is 4852 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa6009cbd0 con 0x7ffa6009f420 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2473104895 0 0) 0x7ffa60137290 con 0x7ffa6009ebd0 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 >> v1:172.21.15.171:6789/0 conn(0x7ffa6009f420 legacy=0x7ffa6013f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 >> v1:172.21.15.148:6789/0 conn(0x7ffa6009efb0 legacy=0x7ffa6013bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 monclient: found mon.noname-b 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6009ceb0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa500018f0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 5 monclient: authenticate success, global_id 4877 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: dump: 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.589 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffa50001b60 con 0x7ffa6009ebd0 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 1 -- 172.21.15.148:0/550670757 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa50001f70 con 0x7ffa6009ebd0 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa684a5640 4 set_mon_vals no callback set 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.575+0000 7ffa6523f640 10 monclient: dump: 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 10 monclient: shutdown 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 1 -- 172.21.15.148:0/550670757 >> v1:172.21.15.148:6790/0 conn(0x7ffa6009ebd0 legacy=0x7ffa60136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 1 -- 172.21.15.148:0/550670757 shutdown_connections 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 1 -- 172.21.15.148:0/550670757 >> 172.21.15.148:0/550670757 conn(0x7ffa600a5860 msgr2=0x7ffa600a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 1 -- 172.21.15.148:0/550670757 shutdown_connections 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.576+0000 7ffa684a5640 1 -- 172.21.15.148:0/550670757 wait complete. 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient: monmap: 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.592 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 1 Processor -- start 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 1 -- start start 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient: init 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa6009ebd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa6009efb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient(hunting): picked mon.noname-b con 0x7ffa6009f420 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6009b720 con 0x7ffa6009ebd0 2024-09-16T04:08:33.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6012a6a0 con 0x7ffa6009f420 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6009c630 con 0x7ffa6009efb0 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa684a5640 10 monclient(hunting): authenticate will time out at 934.202836s 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa66241640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ffa6009ebd0 0x7ffa6013bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54758/0 (socket says 172.21.15.148:54758) 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.577+0000 7ffa66241640 1 -- 172.21.15.148:0/1024630149 learned_addr learned my addr 172.21.15.148:0/1024630149 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018621353 0 0) 0x7ffa6009c630 con 0x7ffa6009efb0 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): my global_id is 4880 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa4c003520 con 0x7ffa6009efb0 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 438285929 0 0) 0x7ffa6009ceb0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): my global_id is 54120 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa6009c630 con 0x7ffa6009ebd0 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216813579 0 0) 0x7ffa6012a6a0 con 0x7ffa6009f420 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): my global_id is 4855 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa6009ceb0 con 0x7ffa6009f420 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3173903972 0 0) 0x7ffa4c003520 con 0x7ffa6009efb0 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa6012a6a0 con 0x7ffa6009efb0 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa54001040 con 0x7ffa6009efb0 2024-09-16T04:08:33.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): dump: 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2285044772 0 0) 0x7ffa6009ceb0 con 0x7ffa6009f420 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa4c003520 con 0x7ffa6009f420 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 894690621 0 0) 0x7ffa6009c630 con 0x7ffa6009ebd0 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa6009ceb0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa58001830 con 0x7ffa6009f420 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): dump: 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.597 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa50001eb0 con 0x7ffa6009ebd0 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient(hunting): dump: 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.598 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1300602870 0 0) 0x7ffa6012a6a0 con 0x7ffa6009efb0 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.171:6789/0 conn(0x7ffa6009f420 legacy=0x7ffa6013ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.148:6789/0 conn(0x7ffa6009ebd0 legacy=0x7ffa6013bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 monclient: found mon.c 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6012a940 con 0x7ffa6009efb0 2024-09-16T04:08:33.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 5 monclient: authenticate success, global_id 4880 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 10 monclient: _renew_subs 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6013f790 con 0x7ffa6009efb0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 10 monclient: _renew_subs 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ffa601d4ef0 con 0x7ffa6009efb0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffa540015d0 con 0x7ffa6009efb0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa54004980 con 0x7ffa6009efb0 2024-09-16T04:08:33.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa64a3e640 4 set_mon_vals no callback set 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.578+0000 7ffa5effd640 10 monclient: dump: 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa684a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffa400065c0 con 0x7ffa40002860 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ffa5400fe30 con 0x7ffa6009efb0 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7ffa54046d80 con 0x7ffa6009efb0 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613077741 0 0) 0x7ffa400065c0 con 0x7ffa40002860 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 10 monclient(hunting): my global_id is 4883 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa6012a6a0 con 0x7ffa40002860 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.579+0000 7ffa66a42640 10 monclient: get_auth_request con 0x7ffa4c0392a0 auth_method 0 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 805063844 0 0) 0x7ffa6012a6a0 con 0x7ffa40002860 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa400065c0 con 0x7ffa40002860 2024-09-16T04:08:33.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa50004eb0 con 0x7ffa40002860 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 10 monclient: dump: 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.603 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1847171221 0 0) 0x7ffa400065c0 con 0x7ffa40002860 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa4000a150 con 0x7ffa40002860 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ffa50007090 con 0x7ffa40002860 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.148:6790/0 conn(0x7ffa40002860 legacy=0x7ffa40004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.580+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.148:6790/0 conn(0x7ffa40002860 legacy=0x7ffa40004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa684a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffa40009e40 con 0x7ffa40009100 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234619135 0 0) 0x7ffa40009e40 con 0x7ffa40009100 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa5effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa5effd640 10 monclient(hunting): my global_id is 4886 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.586+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa400065c0 con 0x7ffa40009100 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 951085883 0 0) 0x7ffa400065c0 con 0x7ffa40009100 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa40009e40 con 0x7ffa40009100 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ffa50005500 con 0x7ffa40009100 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 10 monclient: dump: 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3448618705 0 0) 0x7ffa40009e40 con 0x7ffa40009100 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa4000c3d0 con 0x7ffa40009100 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 2055098905) 0x7ffa50006810 con 0x7ffa40009100 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:33.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.148:6790/0 conn(0x7ffa40009100 legacy=0x7ffa40009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa5effd640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.148:6790/0 conn(0x7ffa40009100 legacy=0x7ffa40009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 16, 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 137665, 2024-09-16T04:08:33.608 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.609 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.610 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:33.611 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:33.612 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.613 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.614 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.615 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.617 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.618 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.619 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:33.620 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.171:6812/7327273 conn(0x7ffa4c0392a0 legacy=0x7ffa4c03b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 10 monclient: shutdown 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.621 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 >> v1:172.21.15.148:6790/0 conn(0x7ffa6009efb0 legacy=0x7ffa6013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa66a42640 1 -- 172.21.15.148:0/1024630149 reap_dead start 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa66a42640 1 -- 172.21.15.148:0/1024630149 reap_dead start 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 shutdown_connections 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 >> 172.21.15.148:0/1024630149 conn(0x7ffa600a5860 msgr2=0x7ffa600a3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 shutdown_connections 2024-09-16T04:08:33.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.587+0000 7ffa684a5640 1 -- 172.21.15.148:0/1024630149 wait complete. 2024-09-16T04:08:33.622 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:33.622 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:08:33.622 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:08:33.636 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.520+0000 7fd1bcff9640 1 -- 172.21.15.171:0/1076449122 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v59) ==== 103+0+0 (unknown 3643766223 0 0) 0x7fd1b0019760 con 0x7fd1c009aa20 2024-09-16T04:08:33.636 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.520+0000 7fd1bcff9640 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-16T04:08:33.636 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.623+0000 7fd1bcff9640 1 -- 172.21.15.171:0/1076449122 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v59) ==== 103+0+0 (unknown 3643766223 0 0) 0x7fd1b001e020 con 0x7fd1c009aa20 2024-09-16T04:08:33.636 INFO:tasks.workunit.client.0.smithi171.stdout:2024-09-16T04:08:33.623+0000 7fd1bcff9640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T04:08:33.636 INFO:tasks.workunit.client.0.smithi171.stdout:rule bar does not exist 2024-09-16T04:08:33.636 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T04:08:33.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.716+0000 7f72bf586640 10 monclient: get_monmap_and_config 2024-09-16T04:08:33.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.716+0000 7f72bf586640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.716+0000 7f72bf586640 10 monclient: monmap: 2024-09-16T04:08:33.718 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:33.718 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:33.717449+0000 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:33.717449+0000 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:33.719 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:33.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 1 Processor -- start 2024-09-16T04:08:33.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 1 -- start start 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient: init 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient(hunting): picked mon.noname-b con 0x7f72b8133700 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient(hunting): picked mon.noname-a con 0x7f72b81341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient(hunting): picked mon.noname-c con 0x7f72b813a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b809da90 con 0x7f72b81341f0 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b809dd70 con 0x7f72b813a350 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b809e050 con 0x7f72b8133700 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.717+0000 7f72bf586640 10 monclient(hunting): authenticate will time out at 889.222911s 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72bdb23640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f72b813a350 0x7f72b813c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33826/0 (socket says 172.21.15.171:33826) 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72bdb23640 1 -- 172.21.15.171:0/4256486553 learned_addr learned my addr 172.21.15.171:0/4256486553 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:33.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362284595 0 0) 0x7f72b809dd70 con 0x7f72b813a350 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): my global_id is 4858 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72a4003520 con 0x7f72b813a350 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162919975 0 0) 0x7f72b809da90 con 0x7f72b81341f0 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): my global_id is 54123 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b809dd70 con 0x7f72b81341f0 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130863788 0 0) 0x7f72b809e050 con 0x7f72b8133700 2024-09-16T04:08:33.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient(hunting): my global_id is 4889 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b809da90 con 0x7f72b8133700 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 426418326 0 0) 0x7f72a4003520 con 0x7f72b813a350 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 >> v1:172.21.15.148:6790/0 conn(0x7f72b8133700 legacy=0x7f72b8133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 >> v1:172.21.15.148:6789/0 conn(0x7f72b81341f0 legacy=0x7f72b8138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 monclient: found mon.noname-c 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b809d4f0 con 0x7f72b813a350 2024-09-16T04:08:33.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72b0001990 con 0x7f72b813a350 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: dump: 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.729 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72bf586640 5 monclient: authenticate success, global_id 4858 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72bf586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72b0001cb0 con 0x7f72b813a350 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 1 -- 172.21.15.171:0/4256486553 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72b0002080 con 0x7f72b813a350 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72bf586640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72bf586640 4 set_mon_vals no callback set 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.718+0000 7f72affff640 10 monclient: dump: 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:33.730 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 10 monclient: shutdown 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 1 -- 172.21.15.171:0/4256486553 >> v1:172.21.15.171:6789/0 conn(0x7f72b813a350 legacy=0x7f72b813c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 1 -- 172.21.15.171:0/4256486553 shutdown_connections 2024-09-16T04:08:33.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 1 -- 172.21.15.171:0/4256486553 >> 172.21.15.171:0/4256486553 conn(0x7f72b80a5450 msgr2=0x7f72b80a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 1 -- 172.21.15.171:0/4256486553 shutdown_connections 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 1 -- 172.21.15.171:0/4256486553 wait complete. 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.719+0000 7f72bf586640 10 monclient: monmap: 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.732 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 1 Processor -- start 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 1 -- start start 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient: init 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient(hunting): picked mon.noname-a con 0x7f72b8133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient(hunting): picked mon.noname-b con 0x7f72b81341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient(hunting): picked mon.noname-c con 0x7f72b813a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b809b6c0 con 0x7f72b8133700 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b80727d0 con 0x7f72b81341f0 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b8121100 con 0x7f72b813a350 2024-09-16T04:08:33.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bf586640 10 monclient(hunting): authenticate will time out at 889.225602s 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bcb21640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f72b81341f0 0x7f72b81d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33840/0 (socket says 172.21.15.171:33840) 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bcb21640 1 -- 172.21.15.171:0/496342555 learned_addr learned my addr 172.21.15.171:0/496342555 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035108694 0 0) 0x7f72b80727d0 con 0x7f72b81341f0 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): my global_id is 4861 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bd322640 1 -- 172.21.15.171:0/496342555 >> v1:172.21.15.148:6789/0 conn(0x7f72b8133700 legacy=0x7f72b8132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bd322640 1 -- 172.21.15.171:0/496342555 >> v1:172.21.15.148:6789/0 conn(0x7f72b8133700 legacy=0x7f72b8132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f729c003520 con 0x7f72b81341f0 2024-09-16T04:08:33.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bd322640 1 --1- 172.21.15.171:0/496342555 >> v1:172.21.15.148:6789/0 conn(0x7f72b8133700 0x7f72b8132730 :-1 s=OPENED pgs=31 cs=1 l=1).handle_message read tag failed 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72bd322640 1 --1- 172.21.15.171:0/496342555 >> v1:172.21.15.148:6789/0 conn(0x7f72b8133700 0x7f72b8132730 :-1 s=OPENED pgs=31 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 301897470 0 0) 0x7f72b8121100 con 0x7f72b813a350 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 10 monclient(hunting): my global_id is 4892 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.720+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b80727d0 con 0x7f72b813a350 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 353042677 0 0) 0x7f729c003520 con 0x7f72b81341f0 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72b8121100 con 0x7f72b81341f0 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72b40018b0 con 0x7f72b81341f0 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:33.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient(hunting): dump: 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2561475795 0 0) 0x7f72b8121100 con 0x7f72b81341f0 2024-09-16T04:08:33.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 >> v1:172.21.15.148:6790/0 conn(0x7f72b813a350 legacy=0x7f72b81d3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 >> v1:172.21.15.148:6789/0 conn(0x7f72b8133700 legacy=0x7f72b8132730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 monclient: found mon.b 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b8120860 con 0x7f72b81341f0 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 5 monclient: authenticate success, global_id 4861 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 10 monclient: _renew_subs 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72b81cb7e0 con 0x7f72b81341f0 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72b40035f0 con 0x7f72b81341f0 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72b4003940 con 0x7f72b81341f0 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72adffb640 10 monclient: dump: 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.738 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 10 monclient: _renew_subs 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72af7fe640 4 set_mon_vals no callback set 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.721+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72b8133020 con 0x7f72b81341f0 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f72b4003ae0 con 0x7f72b81341f0 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72bf586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72bf586640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72bf586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b8138330 con 0x7f72b81341f0 2024-09-16T04:08:33.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7f72b4046f40 con 0x7f72b81341f0 2024-09-16T04:08:33.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.722+0000 7f72bd322640 10 monclient: get_auth_request con 0x7f729c038be0 auth_method 0 2024-09-16T04:08:33.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.727+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f72b4015a60 con 0x7f72b81341f0 2024-09-16T04:08:33.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.727+0000 7f72adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.727+0000 7f72adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.719+0000 7f7d615f1640 10 monclient: get_monmap_and_config 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.719+0000 7f7d615f1640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.719+0000 7f7d615f1640 10 monclient: monmap: 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:33.720667+0000 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:33.720667+0000 2024-09-16T04:08:33.749 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 1 Processor -- start 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 1 -- start start 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient: init 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7d5c0a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7d5c09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5c09b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c09e040 con 0x7f7d5c09b170 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c09d660 con 0x7f7d5c09b5c0 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c09c0d0 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d615f1640 10 monclient(hunting): authenticate will time out at 934.345860s 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d5affd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7d5c0a0e90 0x7f7d5c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59340/0 (socket says 172.21.15.148:59340) 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d5a7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7d5c09b170 0x7f7d5c1383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54764/0 (socket says 172.21.15.148:54764) 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.720+0000 7f7d5affd640 1 -- 172.21.15.148:0/366382888 learned_addr learned my addr 172.21.15.148:0/366382888 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3714260684 0 0) 0x7f7d50001110 con 0x7f7d5c09b170 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): my global_id is 54126 2024-09-16T04:08:33.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d44003520 con 0x7f7d5c09b170 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408467135 0 0) 0x7f7d54001490 con 0x7f7d5c09b5c0 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): my global_id is 4864 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d50001110 con 0x7f7d5c09b5c0 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158109232 0 0) 0x7f7d5c09c0d0 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient(hunting): my global_id is 4895 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d54001490 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1088201783 0 0) 0x7f7d44003520 con 0x7f7d5c09b170 2024-09-16T04:08:33.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 >> v1:172.21.15.148:6790/0 conn(0x7f7d5c0a0e90 legacy=0x7f7d5c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 >> v1:172.21.15.171:6789/0 conn(0x7f7d5c09b5c0 legacy=0x7f7d5c13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 monclient: found mon.noname-a 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c09d380 con 0x7f7d5c09b170 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d500017e0 con 0x7f7d5c09b170 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 5 monclient: authenticate success, global_id 54126 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: dump: 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d50001a40 con 0x7f7d5c09b170 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 1 -- 172.21.15.148:0/366382888 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d500047e0 con 0x7f7d5c09b170 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 4 set_mon_vals no callback set 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d59ffb640 10 monclient: dump: 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.755 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 10 monclient: shutdown 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.721+0000 7f7d615f1640 1 -- 172.21.15.148:0/366382888 >> v1:172.21.15.148:6789/0 conn(0x7f7d5c09b170 legacy=0x7f7d5c1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.722+0000 7f7d615f1640 1 -- 172.21.15.148:0/366382888 shutdown_connections 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.722+0000 7f7d615f1640 1 -- 172.21.15.148:0/366382888 >> 172.21.15.148:0/366382888 conn(0x7f7d5c0a5640 msgr2=0x7f7d5c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.722+0000 7f7d615f1640 1 -- 172.21.15.148:0/366382888 shutdown_connections 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.722+0000 7f7d615f1640 1 -- 172.21.15.148:0/366382888 wait complete. 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.722+0000 7f7d615f1640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.722+0000 7f7d615f1640 10 monclient: monmap: 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 1 Processor -- start 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 1 -- start start 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient: init 2024-09-16T04:08:33.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient(hunting): picked mon.noname-c con 0x7f7d5c09b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient(hunting): picked mon.noname-b con 0x7f7d5c09b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient(hunting): picked mon.noname-a con 0x7f7d5c0a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c072c10 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c0740a0 con 0x7f7d5c09b5c0 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c1cc5c0 con 0x7f7d5c09b170 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d615f1640 10 monclient(hunting): authenticate will time out at 934.348561s 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d5affd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7d5c09b170 0x7f7d5c13ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59346/0 (socket says 172.21.15.148:59346) 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d5affd640 1 -- 172.21.15.148:0/2960051631 learned_addr learned my addr 172.21.15.148:0/2960051631 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3127282009 0 0) 0x7f7d5c1cc5c0 con 0x7f7d5c09b170 2024-09-16T04:08:33.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): my global_id is 4898 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d48003520 con 0x7f7d5c09b170 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158688976 0 0) 0x7f7d5c072c10 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 10 monclient(hunting): my global_id is 54129 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.723+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d5c1cc5c0 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4146387180 0 0) 0x7f7d48003520 con 0x7f7d5c09b170 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d5c072c10 con 0x7f7d5c09b170 2024-09-16T04:08:33.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d4c001830 con 0x7f7d5c09b170 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): dump: 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3837881482 0 0) 0x7f7d5c1cc5c0 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d48003520 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d54001dc0 con 0x7f7d5c0a0e90 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient(hunting): dump: 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.761 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3493337429 0 0) 0x7f7d5c072c10 con 0x7f7d5c09b170 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.171:6789/0 conn(0x7f7d5c09b5c0 legacy=0x7f7d5c1d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.148:6789/0 conn(0x7f7d5c0a0e90 legacy=0x7f7d5c1d4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 monclient: found mon.c 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c127b30 con 0x7f7d5c09b170 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 5 monclient: authenticate success, global_id 4898 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 10 monclient: _renew_subs 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5c13f160 con 0x7f7d5c09b170 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 10 monclient: _renew_subs 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d5c1260b0 con 0x7f7d5c09b170 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d4c0047e0 con 0x7f7d5c09b170 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d4c004b30 con 0x7f7d5c09b170 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d597fa640 4 set_mon_vals no callback set 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 10 monclient: dump: 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.724+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7d4c004cd0 con 0x7f7d5c09b170 2024-09-16T04:08:33.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.725+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7f7d4c047130 con 0x7f7d5c09b170 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.725+0000 7f7d615f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.725+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7d3c0065c0 con 0x7f7d3c002860 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.725+0000 7f7d5a7fc640 10 monclient: get_auth_request con 0x7f7d48038f10 auth_method 0 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.726+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104015212 0 0) 0x7f7d3c0065c0 con 0x7f7d3c002860 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.726+0000 7f7d437fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.726+0000 7f7d437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.726+0000 7f7d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.726+0000 7f7d437fe640 10 monclient(hunting): my global_id is 4867 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d5c072c10 con 0x7f7d3c002860 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1237638485 0 0) 0x7f7d5c072c10 con 0x7f7d3c002860 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d3c0065c0 con 0x7f7d3c002860 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d54003b10 con 0x7f7d3c002860 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 10 monclient: dump: 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1758502888 0 0) 0x7f7d3c0065c0 con 0x7f7d3c002860 2024-09-16T04:08:33.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d3c00a150 con 0x7f7d3c002860 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7d54003dc0 con 0x7f7d3c002860 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.171:6789/0 conn(0x7f7d3c002860 legacy=0x7f7d3c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.727+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.171:6789/0 conn(0x7f7d3c002860 legacy=0x7f7d3c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.733+0000 7f7d615f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.733+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7d3c009e20 con 0x7f7d3c0090e0 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104520513 0 0) 0x7f7d3c009e20 con 0x7f7d3c0090e0 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient(hunting): my global_id is 4870 2024-09-16T04:08:33.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d3c0065c0 con 0x7f7d3c0090e0 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2576962257 0 0) 0x7f7d3c0065c0 con 0x7f7d3c0090e0 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d3c009e20 con 0x7f7d3c0090e0 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d540043d0 con 0x7f7d3c0090e0 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient: dump: 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.768 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3551622349 0 0) 0x7f7d3c009e20 con 0x7f7d3c0090e0 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d3c00c390 con 0x7f7d3c0090e0 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 649219032) 0x7f7d54005630 con 0x7f7d3c0090e0 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.171:6789/0 conn(0x7f7d3c0090e0 legacy=0x7f7d3c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.734+0000 7f7d437fe640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.171:6789/0 conn(0x7f7d3c0090e0 legacy=0x7f7d3c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 16, 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 137772, 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.770 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.771 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.772 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:33.773 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:33.774 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:33.775 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.776 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.777 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.778 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.779 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:33.780 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:33.781 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:33.782 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:33.782 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:33.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:33.782 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:33.782 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:33.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.171:6812/7327273 conn(0x7f7d48038f10 legacy=0x7f7d4803b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 10 monclient: shutdown 2024-09-16T04:08:33.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 >> v1:172.21.15.148:6790/0 conn(0x7f7d5c09b170 legacy=0x7f7d5c13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d5b7fe640 1 -- 172.21.15.148:0/2960051631 reap_dead start 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d5b7fe640 1 -- 172.21.15.148:0/2960051631 reap_dead start 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 shutdown_connections 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 >> 172.21.15.148:0/2960051631 conn(0x7f7d5c0a5640 msgr2=0x7f7d5c0a3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 shutdown_connections 2024-09-16T04:08:33.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.735+0000 7f7d615f1640 1 -- 172.21.15.148:0/2960051631 wait complete. 2024-09-16T04:08:33.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:33.784 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:08:33.785 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:08:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.879+0000 7f5687991640 10 monclient: get_monmap_and_config 2024-09-16T04:08:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.879+0000 7f5687991640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.879+0000 7f5687991640 10 monclient: monmap: 2024-09-16T04:08:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.881 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:33.880330+0000 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:33.880330+0000 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:33.882 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.879+0000 7f5687991640 1 Processor -- start 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 1 -- start start 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient: init 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient(hunting): picked mon.noname-b con 0x7f56800a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient(hunting): picked mon.noname-c con 0x7f568009b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient(hunting): picked mon.noname-a con 0x7f568009b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568009e040 con 0x7f568009b5c0 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568009d660 con 0x7f568009b170 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568009c0d0 con 0x7f56800a0e90 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f5687991640 10 monclient(hunting): authenticate will time out at 934.505772s 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f568572d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f56800a0e90 0x7f568009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59352/0 (socket says 172.21.15.148:59352) 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.880+0000 7f568572d640 1 -- 172.21.15.148:0/1441642929 learned_addr learned my addr 172.21.15.148:0/1441642929 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 987580288 0 0) 0x7f568009c0d0 con 0x7f56800a0e90 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): my global_id is 4901 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f566c003520 con 0x7f56800a0e90 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3484228326 0 0) 0x7f5678001490 con 0x7f568009b5c0 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): my global_id is 54132 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f568009c0d0 con 0x7f568009b5c0 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704836674 0 0) 0x7f567c001110 con 0x7f568009b170 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient(hunting): my global_id is 4873 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5678001490 con 0x7f568009b170 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1041897426 0 0) 0x7f566c003520 con 0x7f56800a0e90 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 >> v1:172.21.15.171:6789/0 conn(0x7f568009b170 legacy=0x7f56801383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 >> v1:172.21.15.148:6789/0 conn(0x7f568009b5c0 legacy=0x7f568013bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 monclient: found mon.noname-b 2024-09-16T04:08:33.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568009d380 con 0x7f56800a0e90 2024-09-16T04:08:33.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5674001610 con 0x7f56800a0e90 2024-09-16T04:08:33.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 5 monclient: authenticate success, global_id 4901 2024-09-16T04:08:33.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:33.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: dump: 2024-09-16T04:08:33.901 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.901 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.901 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.901 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.901 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.902 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.902 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.902 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.902 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.902 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5674001840 con 0x7f56800a0e90 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 1 -- 172.21.15.148:0/1441642929 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5674001c50 con 0x7f56800a0e90 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 4 set_mon_vals no callback set 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5673fff640 10 monclient: dump: 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.904 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.905 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.905 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.905 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 10 monclient: shutdown 2024-09-16T04:08:33.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:33.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 1 -- 172.21.15.148:0/1441642929 >> v1:172.21.15.148:6790/0 conn(0x7f56800a0e90 legacy=0x7f568009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.881+0000 7f5687991640 1 -- 172.21.15.148:0/1441642929 shutdown_connections 2024-09-16T04:08:33.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.882+0000 7f5687991640 1 -- 172.21.15.148:0/1441642929 >> 172.21.15.148:0/1441642929 conn(0x7f56800a5640 msgr2=0x7f56800a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:33.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.882+0000 7f5687991640 1 -- 172.21.15.148:0/1441642929 shutdown_connections 2024-09-16T04:08:33.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.882+0000 7f5687991640 1 -- 172.21.15.148:0/1441642929 wait complete. 2024-09-16T04:08:33.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.882+0000 7f5687991640 10 monclient: build_initial_monmap 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.882+0000 7f5687991640 10 monclient: monmap: 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.907 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.882+0000 7f5687991640 1 Processor -- start 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 1 -- start start 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient: init 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient(hunting): picked mon.noname-b con 0x7f568009b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:33.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient(hunting): picked mon.noname-c con 0x7f568009b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient(hunting): picked mon.noname-a con 0x7f56800a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56801c3a60 con 0x7f56800a0e90 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56801a81f0 con 0x7f568009b170 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56801aacb0 con 0x7f568009b5c0 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5687991640 10 monclient(hunting): authenticate will time out at 934.508433s 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5684f2c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f568009b5c0 0x7f56801ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59362/0 (socket says 172.21.15.148:59362) 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5684f2c640 1 -- 172.21.15.148:0/1405545324 learned_addr learned my addr 172.21.15.148:0/1405545324 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749620834 0 0) 0x7f56801c3a60 con 0x7f56800a0e90 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): my global_id is 54135 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5664003520 con 0x7f56800a0e90 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251034808 0 0) 0x7f56801aacb0 con 0x7f568009b5c0 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): my global_id is 4904 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56801c3a60 con 0x7f568009b5c0 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1332024232 0 0) 0x7f56801a81f0 con 0x7f568009b170 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:33.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): my global_id is 4876 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56801aacb0 con 0x7f568009b170 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3730997831 0 0) 0x7f5664003520 con 0x7f56800a0e90 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56801a81f0 con 0x7f56800a0e90 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f56780018c0 con 0x7f56800a0e90 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.911 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.883+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2808088034 0 0) 0x7f56801c3a60 con 0x7f568009b5c0 2024-09-16T04:08:33.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5664003520 con 0x7f568009b5c0 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f567c001a00 con 0x7f568009b5c0 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.913 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2197985920 0 0) 0x7f56801aacb0 con 0x7f568009b170 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56801c3a60 con 0x7f568009b170 2024-09-16T04:08:33.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5674001b90 con 0x7f568009b170 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.915 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1174717732 0 0) 0x7f5664003520 con 0x7f568009b5c0 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 >> v1:172.21.15.171:6789/0 conn(0x7f568009b170 legacy=0x7f56801c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 >> v1:172.21.15.148:6789/0 conn(0x7f56800a0e90 legacy=0x7f56801d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 monclient: found mon.c 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56801a89e0 con 0x7f568009b5c0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 5 monclient: authenticate success, global_id 4904 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 10 monclient: _renew_subs 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56801aa9d0 con 0x7f568009b5c0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f567c001ba0 con 0x7f568009b5c0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f567c003a60 con 0x7f568009b5c0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:33.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5671ffb640 10 monclient: dump: 2024-09-16T04:08:33.918 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:33.918 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:33.918 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.918 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:33.918 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:33.919 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:33.919 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 10 monclient: _renew_subs 2024-09-16T04:08:33.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f56737fe640 4 set_mon_vals no callback set 2024-09-16T04:08:33.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.884+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f56801c5dc0 con 0x7f568009b5c0 2024-09-16T04:08:33.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f567c003a60 con 0x7f568009b5c0 2024-09-16T04:08:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(59..59 src has 1..59) ==== 4387+0+0 (unknown 1439843745 0 0) 0x7f567c046cd0 con 0x7f568009b5c0 2024-09-16T04:08:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f568572d640 10 monclient: get_auth_request con 0x7f56640391f0 auth_method 0 2024-09-16T04:08:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f5687991640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f5687991640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f5687991640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:33.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.885+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56800740a0 con 0x7f568009b5c0 2024-09-16T04:08:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.890+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f568009d660 con 0x7f568009b5c0 2024-09-16T04:08:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.890+0000 7f5671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:33.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:33.890+0000 7f5671ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:33.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.897+0000 7f72bf586640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:33.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.897+0000 7f72bf586640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:33.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.897+0000 7f72bf586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:33.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:33.897+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7f72b81cba10 con 0x7f72b81341f0 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.121+0000 7f5687991640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.121+0000 7f5687991640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.121+0000 7f5687991640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.121+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f56801a3670 con 0x7f568009b5c0 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.122+0000 7f5671ffb640 1 -- 172.21.15.148:0/1405545324 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f567c015930 con 0x7f568009b5c0 2024-09-16T04:08:34.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.122+0000 7f5671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:08:34.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.122+0000 7f5671ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:34.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.123+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v60) ==== 110+0+0 (unknown 1535433929 0 0) 0x7f72b4019710 con 0x7f72b81341f0 2024-09-16T04:08:34.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.123+0000 7f72adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.123+0000 7f72adffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:34.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.123+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.126 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.123+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=10,kv=9,logm=67} crc {auth=549221487,config=692619927,health=2439085623,kv=2272835141,logm=254863479}) 2024-09-16T04:08:34.126 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.123+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=11,kv=10,logm=68} crc {auth=907629644,config=692619927,health=3172947059,kv=2270513034,logm=2383336917}) 2024-09-16T04:08:34.126 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.123+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.126 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.123+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=10,kv=9,logm=67} crc {auth=549221487,config=692619927,health=2439085623,kv=2272835141,logm=254863479}) 2024-09-16T04:08:34.126 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.123+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=549221487,config=692619927,health=3897446751,kv=2270513034,logm=1212421161}) 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 >> v1:172.21.15.171:6812/7327273 conn(0x7f56640391f0 legacy=0x7f566403b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 10 monclient: shutdown 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 >> v1:172.21.15.148:6790/0 conn(0x7f568009b5c0 legacy=0x7f56801ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 shutdown_connections 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 >> 172.21.15.148:0/1405545324 conn(0x7f56800a5640 msgr2=0x7f56800a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 shutdown_connections 2024-09-16T04:08:34.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f5687991640 1 -- 172.21.15.148:0/1405545324 wait complete. 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128932763}) 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2123428946}) 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128932763}) 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.124+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101998693}) 2024-09-16T04:08:34.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218894704}) 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367541712}) 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218894704}) 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3839550449}) 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406707710}) 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410196764}) 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.132 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406707710}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.125+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2695232398}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.126+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.126+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880965141}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.126+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513736810}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.126+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.126+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880965141}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.126+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239045644}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.127+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.127+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94383429}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.127+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236753854}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.127+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.127+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94383429}) 2024-09-16T04:08:34.133 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.127+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1138825318}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8,monmap=3} crc {logm=1830718512,mdsmap=1097804592,mgr=397011007,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3491511243,mon_config_key=2720514793,monmap=2633035914}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10} crc {logm=4005285376,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3491511243,mon_config_key=1198105214}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8,monmap=3} crc {logm=1830718512,mdsmap=1097804592,mgr=397011007,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3491511243,mon_config_key=2720514793,monmap=2633035914}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=1} crc {logm=4289717555,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3491511243,mon_config_key=888502937,monmap=1290997623}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.128+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=91} crc {osd_metadata=4007021953,osd_pg_creating=2918134708,osd_snap=841190636,osdmap=1764052487}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=91} crc {osd_metadata=4007021953,osd_pg_creating=2918134708,osd_snap=841190635,osdmap=4025211677}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=91} crc {osd_metadata=4007021953,osd_pg_creating=2918134708,osd_snap=841190636,osdmap=1764052487}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=6,osd_snap=2,osdmap=92} crc {osd_metadata=4007021953,osd_snap=841190635,osdmap=1391463886}) 2024-09-16T04:08:34.134 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:34.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=3340706261}) 2024-09-16T04:08:34.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:08:34.129+0000 7f4b1aabd640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=13} crc {osdmap=1840571226}) 2024-09-16T04:08:34.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:08:34.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.225+0000 7f72bf586640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.225+0000 7f72bf586640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.225+0000 7f72bf586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.225+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7f72b80008d0 con 0x7f72b81341f0 2024-09-16T04:08:34.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.226+0000 7f72adffb640 1 -- 172.21.15.171:0/496342555 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v60) ==== 134+0+0 (unknown 1774726514 0 0) 0x7f72b401dfd0 con 0x7f72b81341f0 2024-09-16T04:08:34.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.226+0000 7f72adffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.226+0000 7f72adffb640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T04:08:34.229 INFO:tasks.workunit.client.0.smithi171.stderr:already renamed to 'foo' 2024-09-16T04:08:34.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 >> v1:172.21.15.171:6812/7327273 conn(0x7f729c038be0 legacy=0x7f729c03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 10 monclient: shutdown 2024-09-16T04:08:34.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:34.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 >> v1:172.21.15.171:6789/0 conn(0x7f72b81341f0 legacy=0x7f72b81d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 shutdown_connections 2024-09-16T04:08:34.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 >> 172.21.15.171:0/496342555 conn(0x7f72b80a5450 msgr2=0x7f72b81395c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:34.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 shutdown_connections 2024-09-16T04:08:34.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.228+0000 7f72bf586640 1 -- 172.21.15.171:0/496342555 wait complete. 2024-09-16T04:08:34.239 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-16T04:08:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.318+0000 7fe877ae6640 10 monclient: get_monmap_and_config 2024-09-16T04:08:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.318+0000 7fe877ae6640 10 monclient: build_initial_monmap 2024-09-16T04:08:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.318+0000 7fe877ae6640 10 monclient: monmap: 2024-09-16T04:08:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:34.319243+0000 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:34.319243+0000 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 1 Processor -- start 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 1 -- start start 2024-09-16T04:08:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient: init 2024-09-16T04:08:34.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient(hunting): picked mon.noname-b con 0x7fe870072aa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8700702c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient(hunting): picked mon.noname-c con 0x7fe87013b0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8700a0590 con 0x7fe8700702c0 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87009f000 con 0x7fe87013b0e0 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8700a02b0 con 0x7fe870072aa0 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe877ae6640 10 monclient(hunting): authenticate will time out at 889.824798s 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe876083640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe87013b0e0 0x7fe87013d4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33856/0 (socket says 172.21.15.171:33856) 2024-09-16T04:08:34.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.319+0000 7fe876083640 1 -- 172.21.15.171:0/916617286 learned_addr learned my addr 172.21.15.171:0/916617286 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 114742013 0 0) 0x7fe87009f000 con 0x7fe87013b0e0 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): my global_id is 4879 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe85c003520 con 0x7fe87013b0e0 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632829410 0 0) 0x7fe8700a02b0 con 0x7fe870072aa0 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): my global_id is 4907 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87009f000 con 0x7fe870072aa0 2024-09-16T04:08:34.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796916735 0 0) 0x7fe8700a0590 con 0x7fe8700702c0 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient(hunting): my global_id is 54138 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8700a02b0 con 0x7fe8700702c0 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1341480381 0 0) 0x7fe85c003520 con 0x7fe87013b0e0 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 >> v1:172.21.15.148:6790/0 conn(0x7fe870072aa0 legacy=0x7fe870072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 >> v1:172.21.15.148:6789/0 conn(0x7fe8700702c0 legacy=0x7fe8701399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: _finish_hunting 0 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 monclient: found mon.noname-c 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87009f280 con 0x7fe87013b0e0 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: _finish_auth 0 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: _check_auth_tickets 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe868001910 con 0x7fe87013b0e0 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe877ae6640 5 monclient: authenticate success, global_id 4879 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe877ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: dump: 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.337 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe877ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe868001b70 con 0x7fe87013b0e0 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 1 -- 172.21.15.171:0/916617286 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe868001f40 con 0x7fe87013b0e0 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe877ae6640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe877ae6640 4 set_mon_vals no callback set 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:34.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe874880640 10 monclient: dump: 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.320+0000 7fe877ae6640 10 monclient: shutdown 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:34.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 1 -- 172.21.15.171:0/916617286 >> v1:172.21.15.171:6789/0 conn(0x7fe87013b0e0 legacy=0x7fe87013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 1 -- 172.21.15.171:0/916617286 shutdown_connections 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 1 -- 172.21.15.171:0/916617286 >> 172.21.15.171:0/916617286 conn(0x7fe8700a77f0 msgr2=0x7fe8700a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 1 -- 172.21.15.171:0/916617286 shutdown_connections 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 1 -- 172.21.15.171:0/916617286 wait complete. 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 10 monclient: build_initial_monmap 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 10 monclient: monmap: 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:34.340 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.321+0000 7fe877ae6640 1 Processor -- start 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 1 -- start start 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient: init 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:34.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8700702c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient(hunting): picked mon.noname-c con 0x7fe870072aa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient(hunting): picked mon.noname-a con 0x7fe87013b0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87013ab10 con 0x7fe87013b0e0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87011eaa0 con 0x7fe8700702c0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe870134740 con 0x7fe870072aa0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe877ae6640 10 monclient(hunting): authenticate will time out at 889.827560s 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe875882640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe8700702c0 0x7fe8701339a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33866/0 (socket says 172.21.15.171:33866) 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe875882640 1 -- 172.21.15.171:0/3074241126 learned_addr learned my addr 172.21.15.171:0/3074241126 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298834740 0 0) 0x7fe87011eaa0 con 0x7fe8700702c0 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): my global_id is 4882 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe854003520 con 0x7fe8700702c0 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175994920 0 0) 0x7fe870134740 con 0x7fe870072aa0 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): my global_id is 4910 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87011eaa0 con 0x7fe870072aa0 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715747842 0 0) 0x7fe87013ab10 con 0x7fe87013b0e0 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): my global_id is 54141 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe870134740 con 0x7fe87013b0e0 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3437228294 0 0) 0x7fe854003520 con 0x7fe8700702c0 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe87013ab10 con 0x7fe8700702c0 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe86c001a00 con 0x7fe8700702c0 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:34.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.322+0000 7fe8667fc640 10 monclient(hunting): dump: 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.345 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3303618069 0 0) 0x7fe87011eaa0 con 0x7fe870072aa0 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe854003520 con 0x7fe870072aa0 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe8600018b0 con 0x7fe870072aa0 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): dump: 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.346 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.347 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 564376877 0 0) 0x7fe870134740 con 0x7fe87013b0e0 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe87011eaa0 con 0x7fe87013b0e0 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe868001ec0 con 0x7fe87013b0e0 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:34.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient(hunting): dump: 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.349 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1194173038 0 0) 0x7fe87013ab10 con 0x7fe8700702c0 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 >> v1:172.21.15.148:6790/0 conn(0x7fe870072aa0 legacy=0x7fe8701ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 >> v1:172.21.15.148:6789/0 conn(0x7fe87013b0e0 legacy=0x7fe8701d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 monclient: found mon.b 2024-09-16T04:08:34.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8701340b0 con 0x7fe8700702c0 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 5 monclient: authenticate success, global_id 4882 2024-09-16T04:08:34.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 10 monclient: _renew_subs 2024-09-16T04:08:34.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe87011dbb0 con 0x7fe8700702c0 2024-09-16T04:08:34.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 10 monclient: _renew_subs 2024-09-16T04:08:34.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe870139330 con 0x7fe8700702c0 2024-09-16T04:08:34.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe86c0038c0 con 0x7fe8700702c0 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe86c003c90 con 0x7fe8700702c0 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe867fff640 4 set_mon_vals no callback set 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.323+0000 7fe8667fc640 10 monclient: dump: 2024-09-16T04:08:34.353 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.354 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.355 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.355 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe86c003c90 con 0x7fe8700702c0 2024-09-16T04:08:34.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(60..60 src has 1..60) ==== 4382+0+0 (unknown 228546661 0 0) 0x7fe86c046ff0 con 0x7fe8700702c0 2024-09-16T04:08:34.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe877ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:34.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe877ae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:34.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe877ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe858005080 con 0x7fe8700702c0 2024-09-16T04:08:34.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.324+0000 7fe875081640 10 monclient: get_auth_request con 0x7fe854039110 auth_method 0 2024-09-16T04:08:34.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.328+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe86c015be0 con 0x7fe8700702c0 2024-09-16T04:08:34.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.328+0000 7fe8667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:34.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.328+0000 7fe8667fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:34.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.496+0000 7fe877ae6640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.496+0000 7fe877ae6640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.496+0000 7fe877ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.496+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7fe858005370 con 0x7fe8700702c0 2024-09-16T04:08:34.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.498+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v60) ==== 134+0+0 (unknown 1774726514 0 0) 0x7fe86c0154e0 con 0x7fe8700702c0 2024-09-16T04:08:34.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.498+0000 7fe8667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.498+0000 7fe8667fc640 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-16T04:08:34.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.600+0000 7fe877ae6640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.601+0000 7fe877ae6640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.601+0000 7fe877ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.601+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) -- 0x7fe858000c20 con 0x7fe8700702c0 2024-09-16T04:08:34.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.601+0000 7fe8667fc640 1 -- 172.21.15.171:0/3074241126 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v60) ==== 134+0+0 (unknown 1774726514 0 0) 0x7fe86c045b00 con 0x7fe8700702c0 2024-09-16T04:08:34.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.601+0000 7fe8667fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-16T04:08:34.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.601+0000 7fe8667fc640 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-16T04:08:34.605 INFO:tasks.workunit.client.0.smithi171.stderr:already renamed to 'foo' 2024-09-16T04:08:34.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 >> v1:172.21.15.171:6812/7327273 conn(0x7fe854039110 legacy=0x7fe85403b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 10 monclient: shutdown 2024-09-16T04:08:34.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:34.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 >> v1:172.21.15.171:6789/0 conn(0x7fe8700702c0 legacy=0x7fe8701339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 shutdown_connections 2024-09-16T04:08:34.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 >> 172.21.15.171:0/3074241126 conn(0x7fe8700a77f0 msgr2=0x7fe870071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:34.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 shutdown_connections 2024-09-16T04:08:34.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.604+0000 7fe877ae6640 1 -- 172.21.15.171:0/3074241126 wait complete. 2024-09-16T04:08:34.615 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-16T04:08:34.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.692+0000 7f66366e6640 10 monclient: get_monmap_and_config 2024-09-16T04:08:34.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.692+0000 7f66366e6640 10 monclient: build_initial_monmap 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.692+0000 7f66366e6640 10 monclient: monmap: 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:34.693626+0000 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:34.693626+0000 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:34.695 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.693+0000 7f66366e6640 1 Processor -- start 2024-09-16T04:08:34.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.693+0000 7f66366e6640 1 -- start start 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.693+0000 7f66366e6640 10 monclient: init 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6630133760 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient(hunting): picked mon.noname-a con 0x7f663009aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient(hunting): picked mon.noname-b con 0x7f663013d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f663009b4c0 con 0x7f663009aa20 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6630075d70 con 0x7f6630133760 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6630134250 con 0x7f663013d0e0 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient(hunting): authenticate will time out at 890.199135s 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6630133760 0x7f6630133b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33868/0 (socket says 172.21.15.171:33868) 2024-09-16T04:08:34.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662ffff640 1 -- 172.21.15.171:0/2141379412 learned_addr learned my addr 172.21.15.171:0/2141379412 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683935542 0 0) 0x7f6630075d70 con 0x7f6630133760 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): my global_id is 4885 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6618003520 con 0x7f6630133760 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775751883 0 0) 0x7f6630134250 con 0x7f663013d0e0 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): my global_id is 4913 2024-09-16T04:08:34.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6630075d70 con 0x7f663013d0e0 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2748261501 0 0) 0x7f6624001110 con 0x7f663009aa20 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient(hunting): my global_id is 54144 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6630134250 con 0x7f663009aa20 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1707539295 0 0) 0x7f6618003520 con 0x7f6630133760 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 >> v1:172.21.15.148:6790/0 conn(0x7f663013d0e0 legacy=0x7f663013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 >> v1:172.21.15.148:6789/0 conn(0x7f663009aa20 legacy=0x7f663013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 monclient: found mon.noname-c 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:34.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663009c440 con 0x7f6630133760 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: _finish_auth 0 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 5 monclient: authenticate success, global_id 4885 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6620001610 con 0x7f6630133760 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: dump: 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.706 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6620001930 con 0x7f6630133760 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 1 -- 172.21.15.171:0/2141379412 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6620001d00 con 0x7f6630133760 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:34.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f66366e6640 4 set_mon_vals no callback set 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.694+0000 7f662effd640 10 monclient: dump: 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.708 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 10 monclient: shutdown 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 1 -- 172.21.15.171:0/2141379412 >> v1:172.21.15.171:6789/0 conn(0x7f6630133760 legacy=0x7f6630133b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 1 -- 172.21.15.171:0/2141379412 shutdown_connections 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 1 -- 172.21.15.171:0/2141379412 >> 172.21.15.171:0/2141379412 conn(0x7f66300a6330 msgr2=0x7f66300a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 1 -- 172.21.15.171:0/2141379412 shutdown_connections 2024-09-16T04:08:34.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 1 -- 172.21.15.171:0/2141379412 wait complete. 2024-09-16T04:08:34.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 10 monclient: build_initial_monmap 2024-09-16T04:08:34.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.695+0000 7f66366e6640 10 monclient: monmap: 2024-09-16T04:08:34.710 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 1 Processor -- start 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 1 -- start start 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 10 monclient: init 2024-09-16T04:08:34.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 10 monclient(hunting): picked mon.noname-c con 0x7f663009aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6630133760 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.696+0000 7f66366e6640 10 monclient(hunting): picked mon.noname-b con 0x7f663013d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f663011cc60 con 0x7f6630133760 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f663013c2c0 con 0x7f663013d0e0 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f663011d810 con 0x7f663009aa20 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 10 monclient(hunting): authenticate will time out at 890.202038s 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f6634c83640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f663013d0e0 0x7f66301d3e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33884/0 (socket says 172.21.15.171:33884) 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f6634c83640 1 -- 172.21.15.171:0/4225380870 learned_addr learned my addr 172.21.15.171:0/4225380870 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:34.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545550056 0 0) 0x7f663013c2c0 con 0x7f663013d0e0 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): my global_id is 4888 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f661c003520 con 0x7f663013d0e0 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2208512154 0 0) 0x7f663011cc60 con 0x7f6630133760 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): my global_id is 54147 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f663013c2c0 con 0x7f6630133760 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956963215 0 0) 0x7f663011d810 con 0x7f663009aa20 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:34.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): my global_id is 4916 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f663011cc60 con 0x7f663009aa20 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2058357949 0 0) 0x7f661c003520 con 0x7f663013d0e0 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f663011d810 con 0x7f663013d0e0 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6628001c30 con 0x7f663013d0e0 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): dump: 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.714 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2895824756 0 0) 0x7f663011cc60 con 0x7f663009aa20 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f661c003520 con 0x7f663009aa20 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6620001430 con 0x7f663009aa20 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:34.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:34.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): dump: 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1375030852 0 0) 0x7f663013c2c0 con 0x7f6630133760 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f663011cc60 con 0x7f6630133760 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6624001a00 con 0x7f6630133760 2024-09-16T04:08:34.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient(hunting): dump: 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.718 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.719 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2885479133 0 0) 0x7f663011d810 con 0x7f663013d0e0 2024-09-16T04:08:34.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 >> v1:172.21.15.148:6790/0 conn(0x7f663009aa20 legacy=0x7f6630132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 >> v1:172.21.15.148:6789/0 conn(0x7f6630133760 legacy=0x7f66301d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:34.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:34.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 monclient: found mon.b 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663011c920 con 0x7f663013d0e0 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 5 monclient: authenticate success, global_id 4888 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 10 monclient: _renew_subs 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6630132e20 con 0x7f663013d0e0 2024-09-16T04:08:34.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 10 monclient: _renew_subs 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f66301cb890 con 0x7f663013d0e0 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6628003990 con 0x7f663013d0e0 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6628003d60 con 0x7f663013d0e0 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662e7fc640 4 set_mon_vals no callback set 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.697+0000 7f662cff9640 10 monclient: dump: 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:34.721 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.698+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6628003f80 con 0x7f663013d0e0 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.698+0000 7f66366e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.698+0000 7f66366e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.698+0000 7f66366e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.698+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6610005080 con 0x7f663013d0e0 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.699+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(60..60 src has 1..60) ==== 4382+0+0 (unknown 228546661 0 0) 0x7f66280471c0 con 0x7f663013d0e0 2024-09-16T04:08:34.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.699+0000 7f662ffff640 10 monclient: get_auth_request con 0x7f661c039110 auth_method 0 2024-09-16T04:08:34.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.703+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6628015ef0 con 0x7f663013d0e0 2024-09-16T04:08:34.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.703+0000 7f662cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:34.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.703+0000 7f662cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:34.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.870+0000 7f66366e6640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T04:08:34.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.870+0000 7f66366e6640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T04:08:34.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.870+0000 7f66366e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:34.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:34.870+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) -- 0x7f6610005370 con 0x7f663013d0e0 2024-09-16T04:08:35.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.120+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v61) ==== 110+0+0 (unknown 932981054 0 0) 0x7f66280157f0 con 0x7f663013d0e0 2024-09-16T04:08:35.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.120+0000 7f662cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T04:08:35.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.120+0000 7f662cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.138 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:08:35.138 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.223+0000 7f66366e6640 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.223+0000 7f66366e6640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.223+0000 7f66366e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.223+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) -- 0x7f6610005bd0 con 0x7f663013d0e0 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.224+0000 7f662cff9640 1 -- 172.21.15.171:0/4225380870 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v61) ==== 134+0+0 (unknown 4173158973 0 0) 0x7f662801e3a0 con 0x7f663013d0e0 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.224+0000 7f662cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.224+0000 7f662cff9640 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:already renamed to 'bar' 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 >> v1:172.21.15.171:6812/7327273 conn(0x7f661c039110 legacy=0x7f661c03b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 10 monclient: shutdown 2024-09-16T04:08:35.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 >> v1:172.21.15.171:6789/0 conn(0x7f663013d0e0 legacy=0x7f66301d3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 shutdown_connections 2024-09-16T04:08:35.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 >> 172.21.15.171:0/4225380870 conn(0x7f66300a6330 msgr2=0x7f6630136430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 shutdown_connections 2024-09-16T04:08:35.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.226+0000 7f66366e6640 1 -- 172.21.15.171:0/4225380870 wait complete. 2024-09-16T04:08:35.238 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls 2024-09-16T04:08:35.238 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false grep foo-asdf 2024-09-16T04:08:35.238 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:35.238 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo-asdf 2024-09-16T04:08:35.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.243+0000 7fe0b2f5f640 10 monclient: get_monmap_and_config 2024-09-16T04:08:35.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.243+0000 7fe0b2f5f640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.243+0000 7fe0b2f5f640 10 monclient: monmap: 2024-09-16T04:08:35.246 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.246 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.246 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:35.243941+0000 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:35.243941+0000 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:35.247 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.243+0000 7fe0b2f5f640 1 Processor -- start 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.243+0000 7fe0b2f5f640 1 -- start start 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.243+0000 7fe0b2f5f640 10 monclient: init 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac0a22b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac0a1b70 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac13d0d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ac0a0440 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ac0a4d90 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ac0a08f0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b2f5f640 10 monclient(hunting): authenticate will time out at 935.869413s 2024-09-16T04:08:35.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0abfff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe0ac0a1b70 0x7fe0ac13b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59364/0 (socket says 172.21.15.148:59364) 2024-09-16T04:08:35.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0abfff640 1 -- 172.21.15.148:0/3868658951 learned_addr learned my addr 172.21.15.148:0/3868658951 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0b14fc640 1 --1- 172.21.15.148:0/3868658951 >> v1:172.21.15.148:6789/0 conn(0x7fe0ac13d0d0 0x7fe0ac13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54808/0 (socket says 172.21.15.148:54808) 2024-09-16T04:08:35.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510483700 0 0) 0x7fe0ac0a0440 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): my global_id is 54150 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe094003520 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362999163 0 0) 0x7fe0ac0a08f0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): my global_id is 4919 2024-09-16T04:08:35.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ac0a0440 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3338901380 0 0) 0x7fe0ac0a4d90 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 10 monclient(hunting): my global_id is 4891 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.244+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ac0a08f0 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3211841620 0 0) 0x7fe094003520 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 >> v1:172.21.15.148:6790/0 conn(0x7fe0ac0a1b70 legacy=0x7fe0ac13b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 >> v1:172.21.15.171:6789/0 conn(0x7fe0ac0a22b0 legacy=0x7fe0ac0a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 monclient: found mon.noname-a 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac0a2da0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0a4001990 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 5 monclient: authenticate success, global_id 54150 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: dump: 2024-09-16T04:08:35.265 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.266 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0a4001cb0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 1 -- 172.21.15.148:0/3868658951 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0a40020c0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 4 set_mon_vals no callback set 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0ab7fe640 10 monclient: dump: 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.267 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 10 monclient: shutdown 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/3868658951 >> v1:172.21.15.148:6789/0 conn(0x7fe0ac13d0d0 legacy=0x7fe0ac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/3868658951 shutdown_connections 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/3868658951 >> 172.21.15.148:0/3868658951 conn(0x7fe0ac0732c0 msgr2=0x7fe0ac0736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/3868658951 shutdown_connections 2024-09-16T04:08:35.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.245+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/3868658951 wait complete. 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.246+0000 7fe0b2f5f640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.246+0000 7fe0b2f5f640 10 monclient: monmap: 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.246+0000 7fe0b2f5f640 1 Processor -- start 2024-09-16T04:08:35.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.246+0000 7fe0b2f5f640 1 -- start start 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.246+0000 7fe0b2f5f640 10 monclient: init 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe0ac0a1b70 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe0ac0a22b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe0ac13d0d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ac1408d0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ac0a0130 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ac0a80e0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b2f5f640 10 monclient(hunting): authenticate will time out at 935.872435s 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b0cfb640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe0ac0a1b70 0x7fe0ac13c010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54812/0 (socket says 172.21.15.148:54812) 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b0cfb640 1 -- 172.21.15.148:0/4007716224 learned_addr learned my addr 172.21.15.148:0/4007716224 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0b14fc640 1 --1- 172.21.15.148:0/4007716224 >> v1:172.21.15.148:6790/0 conn(0x7fe0ac13d0d0 0x7fe0ac1d3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59380/0 (socket says 172.21.15.148:59380) 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3663531998 0 0) 0x7fe0ac1408d0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): my global_id is 54153 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe098003520 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2202987608 0 0) 0x7fe0ac0a80e0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): my global_id is 4922 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ac1408d0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325871366 0 0) 0x7fe0ac0a0130 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 10 monclient(hunting): my global_id is 4894 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ac0a80e0 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3867898712 0 0) 0x7fe098003520 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0ac0a0130 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.247+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe09c001950 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:35.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): dump: 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2642858798 0 0) 0x7fe0ac1408d0 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe098003520 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3720797434 0 0) 0x7fe0ac0a80e0 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0ac1408d0 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0a4001e60 con 0x7fe0ac13d0d0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): dump: 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.274 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0a0001870 con 0x7fe0ac0a22b0 2024-09-16T04:08:35.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient(hunting): dump: 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.276 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1857623462 0 0) 0x7fe0ac0a0130 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 >> v1:172.21.15.148:6790/0 conn(0x7fe0ac13d0d0 legacy=0x7fe0ac1d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 >> v1:172.21.15.171:6789/0 conn(0x7fe0ac0a22b0 legacy=0x7fe0ac1d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 monclient: found mon.a 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac123690 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 5 monclient: authenticate success, global_id 54153 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 10 monclient: _renew_subs 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ac13c720 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 10 monclient: _renew_subs 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0ac1223c0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe09c001cd0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe09c003a40 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0aaffd640 4 set_mon_vals no callback set 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.248+0000 7fe0a97fa640 10 monclient: dump: 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.279 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe09c003be0 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0b2f5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0b2f5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0b2f5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe08c005080 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7fe09c047080 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.249+0000 7fe0abfff640 10 monclient: get_auth_request con 0x7fe098039230 auth_method 0 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.254+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe09c015c70 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.254+0000 7fe0a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.254+0000 7fe0a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:35.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.315+0000 7f42fd6e6640 10 monclient: get_monmap_and_config 2024-09-16T04:08:35.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.315+0000 7f42fd6e6640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.315+0000 7f42fd6e6640 10 monclient: monmap: 2024-09-16T04:08:35.321 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:35.321 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:35.316389+0000 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:35.316389+0000 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:35.322 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 1 Processor -- start 2024-09-16T04:08:35.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 1 -- start start 2024-09-16T04:08:35.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient: init 2024-09-16T04:08:35.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:35.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient(hunting): picked mon.noname-a con 0x7f42f8136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient(hunting): picked mon.noname-b con 0x7f42f8139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient(hunting): picked mon.noname-c con 0x7f42f813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f80a1340 con 0x7f42f8136b60 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f8137650 con 0x7f42f813d620 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f80a0aa0 con 0x7f42f8139b10 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42fd6e6640 10 monclient(hunting): authenticate will time out at 890.821867s 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42f77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42f813d620 0x7f42f813fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33888/0 (socket says 172.21.15.171:33888) 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.316+0000 7f42f77fe640 1 -- 172.21.15.171:0/4131945013 learned_addr learned my addr 172.21.15.171:0/4131945013 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:35.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840451565 0 0) 0x7f42f8137650 con 0x7f42f813d620 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): my global_id is 4897 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e4003520 con 0x7f42f813d620 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167537420 0 0) 0x7f42f80a1340 con 0x7f42f8136b60 2024-09-16T04:08:35.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): my global_id is 54156 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f8137650 con 0x7f42f8136b60 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121372732 0 0) 0x7f42f80a0aa0 con 0x7f42f8139b10 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient(hunting): my global_id is 4925 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f80a1340 con 0x7f42f8139b10 2024-09-16T04:08:35.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 683636782 0 0) 0x7f42e4003520 con 0x7f42f813d620 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 >> v1:172.21.15.148:6790/0 conn(0x7f42f8139b10 legacy=0x7f42f813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 >> v1:172.21.15.148:6789/0 conn(0x7f42f8136b60 legacy=0x7f42f8136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f80a0550 con 0x7f42f813d620 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42fd6e6640 5 monclient: authenticate success, global_id 4897 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42fd6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42f0001990 con 0x7f42f813d620 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: dump: 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.341 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42fd6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42f0001cb0 con 0x7f42f813d620 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42fd6e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 1 -- 172.21.15.171:0/4131945013 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42f0002080 con 0x7f42f813d620 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42fd6e6640 4 set_mon_vals no callback set 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.317+0000 7f42f67fc640 10 monclient: dump: 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 10 monclient: shutdown 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 1 -- 172.21.15.171:0/4131945013 >> v1:172.21.15.171:6789/0 conn(0x7f42f813d620 legacy=0x7f42f813fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 1 -- 172.21.15.171:0/4131945013 shutdown_connections 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 1 -- 172.21.15.171:0/4131945013 >> 172.21.15.171:0/4131945013 conn(0x7f42f80a98b0 msgr2=0x7f42f80a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 1 -- 172.21.15.171:0/4131945013 shutdown_connections 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 1 -- 172.21.15.171:0/4131945013 wait complete. 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 10 monclient: monmap: 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:35.344 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.318+0000 7f42fd6e6640 1 Processor -- start 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 1 -- start start 2024-09-16T04:08:35.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient: init 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient(hunting): picked mon.noname-c con 0x7f42f8136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient(hunting): picked mon.noname-b con 0x7f42f8139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient(hunting): picked mon.noname-a con 0x7f42f813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f811fb70 con 0x7f42f813d620 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f8120ce0 con 0x7f42f8139b10 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f8120fc0 con 0x7f42f8136b60 2024-09-16T04:08:35.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42fd6e6640 10 monclient(hunting): authenticate will time out at 890.824542s 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42effff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42f8139b10 0x7f42f81d0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33902/0 (socket says 172.21.15.171:33902) 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42effff640 1 -- 172.21.15.171:0/3282143610 learned_addr learned my addr 172.21.15.171:0/3282143610 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333122685 0 0) 0x7f42f8120ce0 con 0x7f42f8139b10 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): my global_id is 4900 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42d8003520 con 0x7f42f8139b10 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323214322 0 0) 0x7f42f811fb70 con 0x7f42f813d620 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): my global_id is 54159 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f8120ce0 con 0x7f42f813d620 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1252192689 0 0) 0x7f42f8120fc0 con 0x7f42f8136b60 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 10 monclient(hunting): my global_id is 4928 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f811fb70 con 0x7f42f8136b60 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 941903298 0 0) 0x7f42d8003520 con 0x7f42f8139b10 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42f8120fc0 con 0x7f42f8139b10 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.319+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42e0001910 con 0x7f42f8139b10 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:35.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): dump: 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.349 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1968339519 0 0) 0x7f42f8120ce0 con 0x7f42f813d620 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42d8003520 con 0x7f42f813d620 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42f00017b0 con 0x7f42f813d620 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): dump: 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.350 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2432693065 0 0) 0x7f42f811fb70 con 0x7f42f8136b60 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42f8120ce0 con 0x7f42f8136b60 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42e80017f0 con 0x7f42f8136b60 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient(hunting): dump: 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 913285712 0 0) 0x7f42f8120fc0 con 0x7f42f8139b10 2024-09-16T04:08:35.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 >> v1:172.21.15.148:6790/0 conn(0x7f42f8136b60 legacy=0x7f42f81cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 >> v1:172.21.15.148:6789/0 conn(0x7f42f813d620 legacy=0x7f42f81d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 monclient: found mon.b 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f813cf20 con 0x7f42f8139b10 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 5 monclient: authenticate success, global_id 4900 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 10 monclient: _renew_subs 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42f80a0240 con 0x7f42f8139b10 2024-09-16T04:08:35.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 10 monclient: _renew_subs 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42f809e400 con 0x7f42f8139b10 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42e0001d00 con 0x7f42f8139b10 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42e0003a60 con 0x7f42f8139b10 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42f5ffb640 4 set_mon_vals no callback set 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.320+0000 7f42ef7fe640 10 monclient: dump: 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.354 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f42e0003a60 con 0x7f42f8139b10 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42fd6e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42fd6e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42fd6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42dc005080 con 0x7f42f8139b10 2024-09-16T04:08:35.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7f42e0046ee0 con 0x7f42f8139b10 2024-09-16T04:08:35.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.321+0000 7f42f6ffd640 10 monclient: get_auth_request con 0x7f42d8039210 auth_method 0 2024-09-16T04:08:35.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.325+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f42e0015ad0 con 0x7f42f8139b10 2024-09-16T04:08:35.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.325+0000 7f42ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.325+0000 7f42ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:35.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.484+0000 7fe0b2f5f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:35.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.484+0000 7fe0b2f5f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:35.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.484+0000 7fe0b2f5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.484+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe08c005370 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.485+0000 7fe0a97fa640 1 -- 172.21.15.148:0/4007716224 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2569773466) 0x7fe09c015570 con 0x7fe0ac0a1b70 2024-09-16T04:08:35.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.485+0000 7fe0a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:35.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.485+0000 7fe0a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.491 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.486+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 >> v1:172.21.15.171:6812/7327273 conn(0x7fe098039230 legacy=0x7fe09803b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.486+0000 7fe0b2f5f640 10 monclient: shutdown 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.486+0000 7fe0b2f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.486+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 >> v1:172.21.15.148:6789/0 conn(0x7fe0ac0a1b70 legacy=0x7fe0ac13c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.487+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 shutdown_connections 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.487+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 >> 172.21.15.148:0/4007716224 conn(0x7fe0ac0732c0 msgr2=0x7fe0ac0a6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.487+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 shutdown_connections 2024-09-16T04:08:35.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.487+0000 7fe0b2f5f640 1 -- 172.21.15.148:0/4007716224 wait complete. 2024-09-16T04:08:35.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.493+0000 7f42fd6e6640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.493+0000 7f42fd6e6640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.493+0000 7f42fd6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.493+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f42dc005370 con 0x7f42f8139b10 2024-09-16T04:08:35.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.494+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+60 (unknown 231978741 0 1596542489) 0x7f42e00153d0 con 0x7f42f8139b10 2024-09-16T04:08:35.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.494+0000 7f42ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.494+0000 7f42ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.500 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:08:35.500 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:08:35.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:08:35.501 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:08:35.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42fd6e6640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42fd6e6640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42fd6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7f42dc000bc0 con 0x7f42f8139b10 2024-09-16T04:08:35.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42ef7fe640 1 -- 172.21.15.171:0/3282143610 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+60 (unknown 231978741 0 1596542489) 0x7f42e0019720 con 0x7f42f8139b10 2024-09-16T04:08:35.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42ef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.594+0000 7f42ef7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:35.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 >> v1:172.21.15.171:6812/7327273 conn(0x7f42d8039210 legacy=0x7f42d803b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 10 monclient: shutdown 2024-09-16T04:08:35.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 >> v1:172.21.15.171:6789/0 conn(0x7f42f8139b10 legacy=0x7f42f81d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 shutdown_connections 2024-09-16T04:08:35.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 >> 172.21.15.171:0/3282143610 conn(0x7f42f80a98b0 msgr2=0x7f42f80a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 shutdown_connections 2024-09-16T04:08:35.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.596+0000 7f42fd6e6640 1 -- 172.21.15.171:0/3282143610 wait complete. 2024-09-16T04:08:35.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.601+0000 7f2723fc4640 10 monclient: get_monmap_and_config 2024-09-16T04:08:35.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.601+0000 7f2723fc4640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.601+0000 7f2723fc4640 10 monclient: monmap: 2024-09-16T04:08:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:35.602628+0000 2024-09-16T04:08:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:35.602628+0000 2024-09-16T04:08:35.604 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.605 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.605 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.605 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:35.605 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:35.605 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 1 Processor -- start 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 1 -- start start 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 10 monclient: init 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f271c134920 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f271c137770 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f271c13b280 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c06bff0 con 0x7f271c137770 2024-09-16T04:08:35.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.602+0000 7f2723fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c09d870 con 0x7f271c134920 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c09db50 con 0x7f271c13b280 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 10 monclient(hunting): authenticate will time out at 936.228034s 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f272155f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f271c137770 0x7f271c139b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54822/0 (socket says 172.21.15.148:54822) 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f272155f640 1 -- 172.21.15.148:0/1351021705 learned_addr learned my addr 172.21.15.148:0/1351021705 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1119542703 0 0) 0x7f271c06bff0 con 0x7f271c137770 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): my global_id is 54162 2024-09-16T04:08:35.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2708003520 con 0x7f271c137770 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721980371 0 0) 0x7f271c09db50 con 0x7f271c13b280 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): my global_id is 4931 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c06bff0 con 0x7f271c13b280 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4075739720 0 0) 0x7f271c09d870 con 0x7f271c134920 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient(hunting): my global_id is 4903 2024-09-16T04:08:35.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c09db50 con 0x7f271c134920 2024-09-16T04:08:35.614 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:35.614 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule ls 2024-09-16T04:08:35.614 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false grep bar-asdf 2024-09-16T04:08:35.614 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:35.614 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep bar-asdf 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1819847638 0 0) 0x7f2708003520 con 0x7f271c137770 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 >> v1:172.21.15.148:6790/0 conn(0x7f271c13b280 legacy=0x7f271c13d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 >> v1:172.21.15.171:6789/0 conn(0x7f271c134920 legacy=0x7f271c134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 monclient: found mon.noname-a 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c0633c0 con 0x7f271c137770 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c001610 con 0x7f271c137770 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 5 monclient: authenticate success, global_id 54162 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: dump: 2024-09-16T04:08:35.626 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f270c001930 con 0x7f271c137770 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 1 -- 172.21.15.148:0/1351021705 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c001d40 con 0x7f271c137770 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2723fc4640 4 set_mon_vals no callback set 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.603+0000 7f2720d5e640 10 monclient: dump: 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.628 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 10 monclient: shutdown 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 1 -- 172.21.15.148:0/1351021705 >> v1:172.21.15.148:6789/0 conn(0x7f271c137770 legacy=0x7f271c139b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 1 -- 172.21.15.148:0/1351021705 shutdown_connections 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 1 -- 172.21.15.148:0/1351021705 >> 172.21.15.148:0/1351021705 conn(0x7f271c062630 msgr2=0x7f271c062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 1 -- 172.21.15.148:0/1351021705 shutdown_connections 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 1 -- 172.21.15.148:0/1351021705 wait complete. 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.604+0000 7f2723fc4640 10 monclient: monmap: 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:35.630 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 1 Processor -- start 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 1 -- start start 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient: init 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f271c134920 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f271c137770 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f271c13b280 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c09adc0 con 0x7f271c13b280 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c120900 con 0x7f271c137770 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c09b920 con 0x7f271c134920 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2723fc4640 10 monclient(hunting): authenticate will time out at 936.230660s 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2721d60640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f271c134920 0x7f271c1cb690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59398/0 (socket says 172.21.15.148:59398) 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f2721d60640 1 -- 172.21.15.148:0/831826922 learned_addr learned my addr 172.21.15.148:0/831826922 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1756518236 0 0) 0x7f271c09b920 con 0x7f271c134920 2024-09-16T04:08:35.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): my global_id is 4934 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2700003520 con 0x7f271c134920 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799846452 0 0) 0x7f271c09adc0 con 0x7f271c13b280 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 10 monclient(hunting): my global_id is 54165 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.605+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c09b920 con 0x7f271c13b280 2024-09-16T04:08:35.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1555405135 0 0) 0x7f271c120900 con 0x7f271c137770 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): my global_id is 4906 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c09adc0 con 0x7f271c137770 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1653359845 0 0) 0x7f2700003520 con 0x7f271c134920 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f271c120900 con 0x7f271c134920 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2718001a10 con 0x7f271c134920 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:35.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): dump: 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.635 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 445704462 0 0) 0x7f271c09b920 con 0x7f271c13b280 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2700003520 con 0x7f271c13b280 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2714001410 con 0x7f271c13b280 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): dump: 2024-09-16T04:08:35.636 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2051665801 0 0) 0x7f271c09adc0 con 0x7f271c137770 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f271c09b920 con 0x7f271c137770 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c0047e0 con 0x7f271c137770 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient(hunting): dump: 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.638 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3791659405 0 0) 0x7f271c120900 con 0x7f271c134920 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.171:6789/0 conn(0x7f271c137770 legacy=0x7f271c1d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.148:6789/0 conn(0x7f271c13b280 legacy=0x7f271c1d3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 monclient: found mon.c 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c06e000 con 0x7f271c134920 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 5 monclient: authenticate success, global_id 4934 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 10 monclient: _renew_subs 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f271c09b220 con 0x7f271c134920 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 10 monclient: _renew_subs 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f271c072c70 con 0x7f271c134920 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27180038a0 con 0x7f271c134920 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2718003c70 con 0x7f271c134920 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f2713fff640 4 set_mon_vals no callback set 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.606+0000 7f27127fc640 10 monclient: dump: 2024-09-16T04:08:35.641 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f2723fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f27040065c0 con 0x7f2704002860 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2718003e90 con 0x7f271c134920 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7f2718047020 con 0x7f271c134920 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2262947460 0 0) 0x7f27040065c0 con 0x7f2704002860 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f2722561640 10 monclient: get_auth_request con 0x7f27000391a0 auth_method 0 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient(hunting): my global_id is 54168 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c120900 con 0x7f2704002860 2024-09-16T04:08:35.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1101639103 0 0) 0x7f271c120900 con 0x7f2704002860 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27040065c0 con 0x7f2704002860 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c004b20 con 0x7f2704002860 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.607+0000 7f27127fc640 10 monclient: dump: 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.644 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1878139381 0 0) 0x7f27040065c0 con 0x7f2704002860 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f270400a150 con 0x7f2704002860 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f270c004ab0 con 0x7f2704002860 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.148:6789/0 conn(0x7f2704002860 legacy=0x7f2704004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.608+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.148:6789/0 conn(0x7f2704002860 legacy=0x7f2704004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f2723fc4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2704009e20 con 0x7f27040090e0 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 422482672 0 0) 0x7f2704009e20 con 0x7f27040090e0 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient(hunting): my global_id is 54171 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27040065c0 con 0x7f27040090e0 2024-09-16T04:08:35.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1866155838 0 0) 0x7f27040065c0 con 0x7f27040090e0 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2704009e20 con 0x7f27040090e0 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f270c005090 con 0x7f27040090e0 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 10 monclient: dump: 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.647 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 150565269 0 0) 0x7f2704009e20 con 0x7f27040090e0 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.614+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f270400c390 con 0x7f27040090e0 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3977 (unknown 0 0 2848847952) 0x7f270c005280 con 0x7f27040090e0 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f27127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f27127fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.148:6789/0 conn(0x7f27040090e0 legacy=0x7f2704009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f27127fc640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.148:6789/0 conn(0x7f27040090e0 legacy=0x7f2704009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:35.649 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:35.649 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:35.649 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:35.649 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:08:35.649 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 16, 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 2585, 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:35.650 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.651 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.652 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:35.653 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:35.654 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.655 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:35.656 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:35.657 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.658 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.659 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.660 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:35.660 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:35.660 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.660 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.661 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.662 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.663 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.665 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:35.665 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:35.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.171:6812/7327273 conn(0x7f27000391a0 legacy=0x7f270003b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 10 monclient: shutdown 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 >> v1:172.21.15.148:6790/0 conn(0x7f271c134920 legacy=0x7f271c1cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2722561640 1 -- 172.21.15.148:0/831826922 reap_dead start 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2722561640 1 -- 172.21.15.148:0/831826922 reap_dead start 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 shutdown_connections 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 >> 172.21.15.148:0/831826922 conn(0x7f271c062630 msgr2=0x7f271c1366f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 shutdown_connections 2024-09-16T04:08:35.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.615+0000 7f2723fc4640 1 -- 172.21.15.148:0/831826922 wait complete. 2024-09-16T04:08:35.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:35.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:08:35.668 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:08:35.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.685+0000 7fb6fc0f1640 10 monclient: get_monmap_and_config 2024-09-16T04:08:35.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.685+0000 7fb6fc0f1640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.685+0000 7fb6fc0f1640 10 monclient: monmap: 2024-09-16T04:08:35.687 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:35.687 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:35.686367+0000 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:35.686367+0000 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:35.688 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 1 Processor -- start 2024-09-16T04:08:35.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 1 -- start start 2024-09-16T04:08:35.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient: init 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f409f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f40a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f409e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f40a1260 con 0x7fb6f409e1d0 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f409d0c0 con 0x7fb6f40a0130 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f409d3a0 con 0x7fb6f409f640 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6fc0f1640 10 monclient(hunting): authenticate will time out at 891.191858s 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6f968c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb6f40a0130 0x7fb6f4138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33904/0 (socket says 172.21.15.171:33904) 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.686+0000 7fb6f968c640 1 -- 172.21.15.171:0/3928466842 learned_addr learned my addr 172.21.15.171:0/3928466842 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:35.697 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554284280 0 0) 0x7fb6f409d0c0 con 0x7fb6f40a0130 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): my global_id is 4909 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6e0003520 con 0x7fb6f40a0130 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3172344482 0 0) 0x7fb6f409d3a0 con 0x7fb6f409f640 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): my global_id is 4937 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f409d0c0 con 0x7fb6f409f640 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1392404595 0 0) 0x7fb6f40a1260 con 0x7fb6f409e1d0 2024-09-16T04:08:35.698 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient(hunting): my global_id is 54174 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f409d3a0 con 0x7fb6f409e1d0 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 381085555 0 0) 0x7fb6e0003520 con 0x7fb6f40a0130 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 >> v1:172.21.15.148:6790/0 conn(0x7fb6f409f640 legacy=0x7fb6f409fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 >> v1:172.21.15.148:6789/0 conn(0x7fb6f409e1d0 legacy=0x7fb6f413bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 monclient: found mon.noname-c 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f409d680 con 0x7fb6f40a0130 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f0001610 con 0x7fb6f40a0130 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6fc0f1640 5 monclient: authenticate success, global_id 4909 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6fc0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: dump: 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.700 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6fc0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6f00018c0 con 0x7fb6f40a0130 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 1 -- 172.21.15.171:0/3928466842 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f0001c90 con 0x7fb6f40a0130 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6fc0f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6fc0f1640 4 set_mon_vals no callback set 2024-09-16T04:08:35.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.687+0000 7fb6f8e8b640 10 monclient: dump: 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 10 monclient: shutdown 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/3928466842 >> v1:172.21.15.171:6789/0 conn(0x7fb6f40a0130 legacy=0x7fb6f4138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/3928466842 shutdown_connections 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/3928466842 >> 172.21.15.171:0/3928466842 conn(0x7fb6f40a52a0 msgr2=0x7fb6f40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/3928466842 shutdown_connections 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/3928466842 wait complete. 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.688+0000 7fb6fc0f1640 10 monclient: monmap: 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:35.703 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 1 Processor -- start 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 1 -- start start 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient: init 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f409e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f409f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f40a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f41a5e90 con 0x7fb6f409e1d0 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f41a8900 con 0x7fb6f40a0130 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f41c1690 con 0x7fb6f409f640 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fc0f1640 10 monclient(hunting): authenticate will time out at 891.194522s 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fa68e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb6f40a0130 0x7fb6f41cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33914/0 (socket says 172.21.15.171:33914) 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6fa68e640 1 -- 172.21.15.171:0/1320914453 learned_addr learned my addr 172.21.15.171:0/1320914453 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4025684844 0 0) 0x7fb6f41a8900 con 0x7fb6f40a0130 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): my global_id is 4912 2024-09-16T04:08:35.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6d8003550 con 0x7fb6f40a0130 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 334664143 0 0) 0x7fb6f41c1690 con 0x7fb6f409f640 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): my global_id is 4940 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f41a8900 con 0x7fb6f409f640 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841215236 0 0) 0x7fb6f41a5e90 con 0x7fb6f409e1d0 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 10 monclient(hunting): my global_id is 54177 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.689+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f41c1690 con 0x7fb6f409e1d0 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2433988038 0 0) 0x7fb6d8003550 con 0x7fb6f40a0130 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6f41a5e90 con 0x7fb6f40a0130 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6ec001c90 con 0x7fb6f40a0130 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): dump: 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.707 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1764212775 0 0) 0x7fb6f41a8900 con 0x7fb6f409f640 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6d8003550 con 0x7fb6f409f640 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6f0001c10 con 0x7fb6f409f640 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): dump: 2024-09-16T04:08:35.708 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 215422585 0 0) 0x7fb6f41c1690 con 0x7fb6f409e1d0 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb6f41a8900 con 0x7fb6f409e1d0 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6e80017a0 con 0x7fb6f409e1d0 2024-09-16T04:08:35.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient(hunting): dump: 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 838330175 0 0) 0x7fb6f41a5e90 con 0x7fb6f40a0130 2024-09-16T04:08:35.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 >> v1:172.21.15.148:6790/0 conn(0x7fb6f409f640 legacy=0x7fb6f41cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 >> v1:172.21.15.148:6789/0 conn(0x7fb6f409e1d0 legacy=0x7fb6f41c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 monclient: found mon.b 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f406ceb0 con 0x7fb6f40a0130 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 5 monclient: authenticate success, global_id 4912 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 10 monclient: _renew_subs 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f4072390 con 0x7fb6f40a0130 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 10 monclient: _renew_subs 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb6f41a8620 con 0x7fb6f40a0130 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6ec003990 con 0x7fb6f40a0130 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6ec003d60 con 0x7fb6f40a0130 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e67fc640 10 monclient: dump: 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.712 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.690+0000 7fb6e7fff640 4 set_mon_vals no callback set 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb6ec003f80 con 0x7fb6f40a0130 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7fb6ec0471a0 con 0x7fb6f40a0130 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6fc0f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6fc0f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6fc0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6f41a16b0 con 0x7fb6f40a0130 2024-09-16T04:08:35.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.691+0000 7fb6f9e8d640 10 monclient: get_auth_request con 0x7fb6d8039280 auth_method 0 2024-09-16T04:08:35.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.695+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb6ec015ed0 con 0x7fb6f40a0130 2024-09-16T04:08:35.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.695+0000 7fb6e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.695+0000 7fb6e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.766+0000 7f882f8d7640 10 monclient: get_monmap_and_config 2024-09-16T04:08:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.766+0000 7f882f8d7640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.766+0000 7f882f8d7640 10 monclient: monmap: 2024-09-16T04:08:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.768 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:35.767170+0000 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:35.767170+0000 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:35.769 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.766+0000 7f882f8d7640 1 Processor -- start 2024-09-16T04:08:35.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 1 -- start start 2024-09-16T04:08:35.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient: init 2024-09-16T04:08:35.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient(hunting): picked mon.noname-b con 0x7f88280a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient(hunting): picked mon.noname-c con 0x7f882809b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient(hunting): picked mon.noname-a con 0x7f882809b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882809e040 con 0x7f882809b5c0 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882809d660 con 0x7f882809b170 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882809c0d0 con 0x7f88280a0e90 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882f8d7640 10 monclient(hunting): authenticate will time out at 936.392570s 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882d673640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88280a0e90 0x7f882809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59412/0 (socket says 172.21.15.148:59412) 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f882d673640 1 -- 172.21.15.148:0/2712129041 learned_addr learned my addr 172.21.15.148:0/2712129041 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174052043 0 0) 0x7f882809c0d0 con 0x7f88280a0e90 2024-09-16T04:08:35.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): my global_id is 4943 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8814003520 con 0x7f88280a0e90 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202937275 0 0) 0x7f8820001110 con 0x7f882809b5c0 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 10 monclient(hunting): my global_id is 54180 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.767+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882809c0d0 con 0x7f882809b5c0 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612677979 0 0) 0x7f8818001110 con 0x7f882809b170 2024-09-16T04:08:35.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient(hunting): my global_id is 4915 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8820001110 con 0x7f882809b170 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3000793922 0 0) 0x7f8814003520 con 0x7f88280a0e90 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 >> v1:172.21.15.171:6789/0 conn(0x7f882809b170 legacy=0x7f88281383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 >> v1:172.21.15.148:6789/0 conn(0x7f882809b5c0 legacy=0x7f882813bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 monclient: found mon.noname-b 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882809d380 con 0x7f88280a0e90 2024-09-16T04:08:35.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8824001990 con 0x7f88280a0e90 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 5 monclient: authenticate success, global_id 4943 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: dump: 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.782 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8824001c40 con 0x7f88280a0e90 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 1 -- 172.21.15.148:0/2712129041 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8824002050 con 0x7f88280a0e90 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 4 set_mon_vals no callback set 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f881ffff640 10 monclient: dump: 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 10 monclient: shutdown 2024-09-16T04:08:35.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 1 -- 172.21.15.148:0/2712129041 >> v1:172.21.15.148:6790/0 conn(0x7f88280a0e90 legacy=0x7f882809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 1 -- 172.21.15.148:0/2712129041 shutdown_connections 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 1 -- 172.21.15.148:0/2712129041 >> 172.21.15.148:0/2712129041 conn(0x7f88280a5640 msgr2=0x7f88280a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 1 -- 172.21.15.148:0/2712129041 shutdown_connections 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.768+0000 7f882f8d7640 1 -- 172.21.15.148:0/2712129041 wait complete. 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.769+0000 7f882f8d7640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.769+0000 7f882f8d7640 10 monclient: monmap: 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:35.785 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.769+0000 7f882f8d7640 1 Processor -- start 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.769+0000 7f882f8d7640 1 -- start start 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.769+0000 7f882f8d7640 10 monclient: init 2024-09-16T04:08:35.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient(hunting): picked mon.noname-a con 0x7f882809b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient(hunting): picked mon.noname-b con 0x7f882809b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient(hunting): picked mon.noname-c con 0x7f88280a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88281ad100 con 0x7f882809b170 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882806fc60 con 0x7f882809b5c0 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88281cc5f0 con 0x7f88280a0e90 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882f8d7640 10 monclient(hunting): authenticate will time out at 936.395321s 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882d673640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f882809b170 0x7f88281cbee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54866/0 (socket says 172.21.15.148:54866) 2024-09-16T04:08:35.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f882d673640 1 -- 172.21.15.148:0/1127926386 learned_addr learned my addr 172.21.15.148:0/1127926386 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351461763 0 0) 0x7f88281cc5f0 con 0x7f88280a0e90 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): my global_id is 4946 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f880c003550 con 0x7f88280a0e90 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110878823 0 0) 0x7f88281ad100 con 0x7f882809b170 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): my global_id is 54183 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88281cc5f0 con 0x7f882809b170 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095405034 0 0) 0x7f882806fc60 con 0x7f882809b5c0 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): my global_id is 4918 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88281ad100 con 0x7f882809b5c0 2024-09-16T04:08:35.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3475564169 0 0) 0x7f880c003550 con 0x7f88280a0e90 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f882806fc60 con 0x7f88280a0e90 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88200014c0 con 0x7f88280a0e90 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): dump: 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.790 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1493549387 0 0) 0x7f88281cc5f0 con 0x7f882809b170 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f880c003550 con 0x7f882809b170 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8824001f90 con 0x7f882809b170 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): dump: 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.791 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 74859625 0 0) 0x7f88281ad100 con 0x7f882809b5c0 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88281cc5f0 con 0x7f882809b5c0 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88180019c0 con 0x7f882809b5c0 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 10 monclient(hunting): dump: 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.770+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3128752113 0 0) 0x7f882806fc60 con 0x7f88280a0e90 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.171:6789/0 conn(0x7f882809b5c0 legacy=0x7f88281d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.148:6789/0 conn(0x7f882809b170 legacy=0x7f88281cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 1 monclient: found mon.c 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88281ac7f0 con 0x7f88280a0e90 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 5 monclient: authenticate success, global_id 4946 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 10 monclient: _renew_subs 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88281c5d80 con 0x7f88280a0e90 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88200017e0 con 0x7f88280a0e90 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8820003a40 con 0x7f88280a0e90 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881dffb640 10 monclient: dump: 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.796 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f881f7fe640 4 set_mon_vals no callback set 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 10 monclient: _renew_subs 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.771+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f88281ab300 con 0x7f88280a0e90 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f882000ecd0 con 0x7f88280a0e90 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f882f8d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8828068e70 con 0x7f882802d7c0 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7f8820045dc0 con 0x7f88280a0e90 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218594952 0 0) 0x7f882809d380 con 0x7f882802d7c0 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient(hunting): my global_id is 4949 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882806fc60 con 0x7f882802d7c0 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f882ce72640 10 monclient: get_auth_request con 0x7f880c039220 auth_method 0 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2362093163 0 0) 0x7f882806fc60 con 0x7f882802d7c0 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f882809d380 con 0x7f882802d7c0 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88240051e0 con 0x7f882802d7c0 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 10 monclient: dump: 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.798 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 314538297 0 0) 0x7f882809d380 con 0x7f882802d7c0 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.772+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f882802dba0 con 0x7f882802d7c0 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.773+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8824007510 con 0x7f882802d7c0 2024-09-16T04:08:35.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.773+0000 7f881dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.773+0000 7f881dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:35.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.773+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.148:6790/0 conn(0x7f882802d7c0 legacy=0x7f8828068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.773+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.148:6790/0 conn(0x7f882802d7c0 legacy=0x7f8828068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:35.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f882f8d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88281a3680 con 0x7f88281cb710 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3971001357 0 0) 0x7f88281a3680 con 0x7f88281cb710 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient(hunting): my global_id is 4952 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882809d380 con 0x7f88281cb710 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3382691622 0 0) 0x7f882809d380 con 0x7f88281cb710 2024-09-16T04:08:35.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88281a3680 con 0x7f88281cb710 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88240058a0 con 0x7f88281cb710 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient: dump: 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.803 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2784795803 0 0) 0x7f88281a3680 con 0x7f88281cb710 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88280a1270 con 0x7f88281cb710 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 3480380625) 0x7f88240076c0 con 0x7f88281cb710 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.148:6790/0 conn(0x7f88281cb710 legacy=0x7f88281a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.779+0000 7f881dffb640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.148:6790/0 conn(0x7f88281cb710 legacy=0x7f88281a2f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 16, 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 139857, 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:35.805 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.806 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.807 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:35.808 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.809 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.810 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:35.811 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.812 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.813 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.815 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.816 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:35.817 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.171:6812/7327273 conn(0x7f880c039220 legacy=0x7f880c03b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 10 monclient: shutdown 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 >> v1:172.21.15.148:6790/0 conn(0x7f88280a0e90 legacy=0x7f88281d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882de74640 1 -- 172.21.15.148:0/1127926386 reap_dead start 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882de74640 1 -- 172.21.15.148:0/1127926386 reap_dead start 2024-09-16T04:08:35.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 shutdown_connections 2024-09-16T04:08:35.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 >> 172.21.15.148:0/1127926386 conn(0x7f88280a5640 msgr2=0x7f88280a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 shutdown_connections 2024-09-16T04:08:35.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.780+0000 7f882f8d7640 1 -- 172.21.15.148:0/1127926386 wait complete. 2024-09-16T04:08:35.819 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:35.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:08:35.819 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:08:35.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6fc0f1640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6fc0f1640 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6fc0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fb6f406bcf0 con 0x7fb6f40a0130 2024-09-16T04:08:35.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+60 (unknown 231978741 0 1596542489) 0x7fb6ec0157d0 con 0x7fb6f40a0130 2024-09-16T04:08:35.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.867+0000 7fb6e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.916+0000 7f3556791640 10 monclient: get_monmap_and_config 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.916+0000 7f3556791640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.916+0000 7f3556791640 10 monclient: monmap: 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:35.917003+0000 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:35.917003+0000 2024-09-16T04:08:35.918 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.919 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.919 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.919 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:35.919 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:35.919 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.916+0000 7f3556791640 1 Processor -- start 2024-09-16T04:08:35.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.916+0000 7f3556791640 1 -- start start 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.916+0000 7f3556791640 10 monclient: init 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient(hunting): picked mon.noname-c con 0x7f35500a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient(hunting): picked mon.noname-b con 0x7f35500a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient(hunting): picked mon.noname-a con 0x7f355013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500a0e00 con 0x7f355013d060 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500a10e0 con 0x7f35500a2410 2024-09-16T04:08:35.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500a4970 con 0x7f35500a1b10 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f3556791640 10 monclient(hunting): authenticate will time out at 936.542470s 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354f7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f35500a1b10 0x7f355013b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59426/0 (socket says 172.21.15.148:59426) 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354f7fe640 1 -- 172.21.15.148:0/1620653264 learned_addr learned my addr 172.21.15.148:0/1620653264 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683217564 0 0) 0x7f35500a4970 con 0x7f35500a1b10 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): my global_id is 4955 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f353c003520 con 0x7f35500a1b10 2024-09-16T04:08:35.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3807767789 0 0) 0x7f35500a0e00 con 0x7f355013d060 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): my global_id is 54186 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35500a4970 con 0x7f355013d060 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004752260 0 0) 0x7f35500a10e0 con 0x7f35500a2410 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 10 monclient(hunting): my global_id is 4921 2024-09-16T04:08:35.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.917+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35500a0e00 con 0x7f35500a2410 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2978698710 0 0) 0x7f35500a4970 con 0x7f355013d060 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 >> v1:172.21.15.148:6790/0 conn(0x7f35500a1b10 legacy=0x7f355013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 >> v1:172.21.15.171:6789/0 conn(0x7f35500a2410 legacy=0x7f35500a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 monclient: found mon.noname-a 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355006dfd0 con 0x7f355013d060 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f35480018f0 con 0x7f355013d060 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 5 monclient: authenticate success, global_id 54186 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: dump: 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.934 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3548001bd0 con 0x7f355013d060 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 1 -- 172.21.15.148:0/1620653264 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3548001fe0 con 0x7f355013d060 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 4 set_mon_vals no callback set 2024-09-16T04:08:35.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f354effd640 10 monclient: dump: 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.936 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 10 monclient: shutdown 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 1 -- 172.21.15.148:0/1620653264 >> v1:172.21.15.148:6789/0 conn(0x7f355013d060 legacy=0x7f355013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 1 -- 172.21.15.148:0/1620653264 shutdown_connections 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 1 -- 172.21.15.148:0/1620653264 >> 172.21.15.148:0/1620653264 conn(0x7f35500a97b0 msgr2=0x7f35500a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 1 -- 172.21.15.148:0/1620653264 shutdown_connections 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.918+0000 7f3556791640 1 -- 172.21.15.148:0/1620653264 wait complete. 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.919+0000 7f3556791640 10 monclient: build_initial_monmap 2024-09-16T04:08:35.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.919+0000 7f3556791640 10 monclient: monmap: 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.919+0000 7f3556791640 1 Processor -- start 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.919+0000 7f3556791640 1 -- start start 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.919+0000 7f3556791640 10 monclient: init 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient(hunting): picked mon.noname-c con 0x7f35500a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient(hunting): picked mon.noname-b con 0x7f35500a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient(hunting): picked mon.noname-a con 0x7f355013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35501375f0 con 0x7f355013d060 2024-09-16T04:08:35.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35501378d0 con 0x7f35500a2410 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355013c8f0 con 0x7f35500a1b10 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f3556791640 10 monclient(hunting): authenticate will time out at 936.545439s 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354ffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f35500a1b10 0x7f355009c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59432/0 (socket says 172.21.15.148:59432) 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354ffff640 1 -- 172.21.15.148:0/381221976 learned_addr learned my addr 172.21.15.148:0/381221976 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819289321 0 0) 0x7f355013c8f0 con 0x7f35500a1b10 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): my global_id is 4958 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3530003520 con 0x7f35500a1b10 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492307019 0 0) 0x7f35501375f0 con 0x7f355013d060 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): my global_id is 54189 2024-09-16T04:08:35.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f355013c8f0 con 0x7f355013d060 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280840101 0 0) 0x7f35501378d0 con 0x7f35500a2410 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): my global_id is 4924 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35501375f0 con 0x7f35500a2410 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1189765794 0 0) 0x7f3530003520 con 0x7f35500a1b10 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35501378d0 con 0x7f35500a1b10 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3540001950 con 0x7f35500a1b10 2024-09-16T04:08:35.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): dump: 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.943 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1664668947 0 0) 0x7f355013c8f0 con 0x7f355013d060 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3530003520 con 0x7f355013d060 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3548001d80 con 0x7f355013d060 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.920+0000 7f354cff9640 10 monclient(hunting): dump: 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.944 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2682095115 0 0) 0x7f35501375f0 con 0x7f35500a2410 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f355013c8f0 con 0x7f35500a2410 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f35380013b0 con 0x7f35500a2410 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient(hunting): dump: 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.946 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4105012507 0 0) 0x7f35501378d0 con 0x7f35500a1b10 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.171:6789/0 conn(0x7f35500a2410 legacy=0x7f355009d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.148:6789/0 conn(0x7f355013d060 legacy=0x7f355009d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 monclient: found mon.c 2024-09-16T04:08:35.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3550140790 con 0x7f35500a1b10 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 5 monclient: authenticate success, global_id 4958 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 10 monclient: _renew_subs 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3550137310 con 0x7f35500a1b10 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 10 monclient: _renew_subs 2024-09-16T04:08:35.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f35501d4b50 con 0x7f35500a1b10 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3540001d20 con 0x7f35500a1b10 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3540003a40 con 0x7f35500a1b10 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354e7fc640 4 set_mon_vals no callback set 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 10 monclient: dump: 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.949 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3540003a40 con 0x7f35500a1b10 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.921+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f35340065c0 con 0x7f3534002860 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7f3540046f00 con 0x7f35500a1b10 2024-09-16T04:08:35.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3211018318 0 0) 0x7f35340065c0 con 0x7f3534002860 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient(hunting): my global_id is 4927 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35501378d0 con 0x7f3534002860 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f3554d2e640 10 monclient: get_auth_request con 0x7f35300391b0 auth_method 0 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 361592073 0 0) 0x7f35501378d0 con 0x7f3534002860 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35340065c0 con 0x7f3534002860 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3538004980 con 0x7f3534002860 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 10 monclient: dump: 2024-09-16T04:08:35.952 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.952 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.952 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3291088519 0 0) 0x7f35340065c0 con 0x7f3534002860 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.922+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f353400a150 con 0x7f3534002860 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.923+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3538004b60 con 0x7f3534002860 2024-09-16T04:08:35.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.923+0000 7f354cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:35.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.923+0000 7f354cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:35.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.923+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.171:6789/0 conn(0x7f3534002860 legacy=0x7f3534004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.923+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.171:6789/0 conn(0x7f3534002860 legacy=0x7f3534004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:35.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f3556791640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:35.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3534009e40 con 0x7f3534009100 2024-09-16T04:08:35.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982289619 0 0) 0x7f3534009e40 con 0x7f3534009100 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient(hunting): my global_id is 4930 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35340065c0 con 0x7f3534009100 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 52134525 0 0) 0x7f35340065c0 con 0x7f3534009100 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3534009e40 con 0x7f3534009100 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f35380050e0 con 0x7f3534009100 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 10 monclient: dump: 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:35.956 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4102974871 0 0) 0x7f3534009e40 con 0x7f3534009100 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.929+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f353400c3d0 con 0x7f3534009100 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 153338642) 0x7f3538005340 con 0x7f3534009100 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f354cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f354cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:35.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.171:6789/0 conn(0x7f3534009100 legacy=0x7f3534009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f354cff9640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.171:6789/0 conn(0x7f3534009100 legacy=0x7f3534009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 16, 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 139968, 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:08:35.958 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.959 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.960 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:35.961 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:35.962 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.963 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.964 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.965 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:35.966 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:35.967 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.968 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:35.969 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:35.970 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:35.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:35.970 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:35.970 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:35.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.171:6812/7327273 conn(0x7f35300391b0 legacy=0x7f353003b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 10 monclient: shutdown 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 >> v1:172.21.15.148:6790/0 conn(0x7f35500a1b10 legacy=0x7f355009c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3554d2e640 1 -- 172.21.15.148:0/381221976 reap_dead start 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3554d2e640 1 -- 172.21.15.148:0/381221976 reap_dead start 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 shutdown_connections 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 >> 172.21.15.148:0/381221976 conn(0x7f35500a97b0 msgr2=0x7f35500a7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 shutdown_connections 2024-09-16T04:08:35.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:35.930+0000 7f3556791640 1 -- 172.21.15.148:0/381221976 wait complete. 2024-09-16T04:08:35.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:35.973 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:08:35.973 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:08:35.973 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:08:35.973 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:08:35.973 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:08:35.973 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6fc0f1640 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6fc0f1640 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6fc0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule ls"} v 0) -- 0x7fb6f41a73b0 con 0x7fb6f40a0130 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6e67fc640 1 -- 172.21.15.171:0/1320914453 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v61) ==== 65+0+60 (unknown 231978741 0 1596542489) 0x7fb6ec01e380 con 0x7fb6f40a0130 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6e67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.970+0000 7fb6e67fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 >> v1:172.21.15.171:6812/7327273 conn(0x7fb6d8039280 legacy=0x7fb6d803b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 10 monclient: shutdown 2024-09-16T04:08:35.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:35.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 >> v1:172.21.15.171:6789/0 conn(0x7fb6f40a0130 legacy=0x7fb6f41cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:35.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 shutdown_connections 2024-09-16T04:08:35.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 >> 172.21.15.171:0/1320914453 conn(0x7fb6f40a52a0 msgr2=0x7fb6f40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:35.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 shutdown_connections 2024-09-16T04:08:35.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:35.972+0000 7fb6fc0f1640 1 -- 172.21.15.171:0/1320914453 wait complete. 2024-09-16T04:08:35.983 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:08:35.983 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:08:35.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:35.984 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:35.984 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm foo 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.062+0000 7fe1cf0e4640 10 monclient: get_monmap_and_config 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.062+0000 7fe1cf0e4640 10 monclient: build_initial_monmap 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.062+0000 7fe1cf0e4640 10 monclient: monmap: 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:36.063235+0000 2024-09-16T04:08:36.064 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:36.063235+0000 2024-09-16T04:08:36.065 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:36.065 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.065 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:36.065 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:36.065 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:36.065 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 1 Processor -- start 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 1 -- start start 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient: init 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe1c81338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe1c809aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe1c813d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c81343d0 con 0x7fe1c809aa20 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c8075db0 con 0x7fe1c81338e0 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c809b1e0 con 0x7fe1c813d240 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:36.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cf0e4640 10 monclient(hunting): authenticate will time out at 891.568758s 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1c7fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe1c809aa20 legacy=0x7fe1c813bb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cce80640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe1c81338e0 0x7fe1c8133cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33930/0 (socket says 172.21.15.171:33930) 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.063+0000 7fe1cce80640 1 -- 172.21.15.171:0/1324968137 learned_addr learned my addr 172.21.15.171:0/1324968137 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385679589 0 0) 0x7fe1c8075db0 con 0x7fe1c81338e0 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): my global_id is 4933 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b8003520 con 0x7fe1c81338e0 2024-09-16T04:08:36.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170454537 0 0) 0x7fe1c809b1e0 con 0x7fe1c813d240 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient(hunting): my global_id is 4961 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c8075db0 con 0x7fe1c813d240 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3642536958 0 0) 0x7fe1b8003520 con 0x7fe1c81338e0 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 >> v1:172.21.15.148:6790/0 conn(0x7fe1c813d240 legacy=0x7fe1c813f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 >> v1:172.21.15.148:6789/0 conn(0x7fe1c809aa20 legacy=0x7fe1c813bb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:36.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c809b4c0 con 0x7fe1c81338e0 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1bc001610 con 0x7fe1c81338e0 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: dump: 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.075 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 5 monclient: authenticate success, global_id 4933 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1bc0018c0 con 0x7fe1c81338e0 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 1 -- 172.21.15.171:0/1324968137 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1bc001c90 con 0x7fe1c81338e0 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:36.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 4 set_mon_vals no callback set 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1c77fe640 10 monclient: dump: 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 10 monclient: shutdown 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/1324968137 >> v1:172.21.15.171:6789/0 conn(0x7fe1c81338e0 legacy=0x7fe1c8133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/1324968137 shutdown_connections 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.064+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/1324968137 >> 172.21.15.171:0/1324968137 conn(0x7fe1c80a28d0 msgr2=0x7fe1c80a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/1324968137 shutdown_connections 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/1324968137 wait complete. 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 10 monclient: build_initial_monmap 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 10 monclient: monmap: 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:36.078 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 1 Processor -- start 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 1 -- start start 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.065+0000 7fe1cf0e4640 10 monclient: init 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe1c809aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe1c81338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe1c813d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:36.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c811dd70 con 0x7fe1c809aa20 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c811e050 con 0x7fe1c81338e0 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1c80a01b0 con 0x7fe1c813d240 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cf0e4640 10 monclient(hunting): authenticate will time out at 891.571616s 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1cce80640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe1c809aa20 legacy=0x7fe1c8132690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe1c81338e0 0x7fe1c81d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33934/0 (socket says 172.21.15.171:33934) 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c7fff640 1 -- 172.21.15.171:0/3173043278 learned_addr learned my addr 172.21.15.171:0/3173043278 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 842688312 0 0) 0x7fe1c811e050 con 0x7fe1c81338e0 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): my global_id is 4936 2024-09-16T04:08:36.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1ac003520 con 0x7fe1c81338e0 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 493913132 0 0) 0x7fe1c80a01b0 con 0x7fe1c813d240 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 10 monclient(hunting): my global_id is 4964 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.066+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1c811e050 con 0x7fe1c813d240 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 512206397 0 0) 0x7fe1ac003520 con 0x7fe1c81338e0 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1c80a01b0 con 0x7fe1c81338e0 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1b4004180 con 0x7fe1c81338e0 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): dump: 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:36.081 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3918315224 0 0) 0x7fe1c811e050 con 0x7fe1c813d240 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1ac003520 con 0x7fe1c813d240 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1c0001c30 con 0x7fe1c813d240 2024-09-16T04:08:36.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient(hunting): dump: 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 106113344 0 0) 0x7fe1c80a01b0 con 0x7fe1c81338e0 2024-09-16T04:08:36.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 >> v1:172.21.15.148:6790/0 conn(0x7fe1c813d240 legacy=0x7fe1c81d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 >> v1:172.21.15.148:6789/0 conn(0x7fe1c809aa20 legacy=0x7fe1c8132690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 monclient: found mon.b 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c809d0c0 con 0x7fe1c81338e0 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 5 monclient: authenticate success, global_id 4936 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 10 monclient: _renew_subs 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c811da90 con 0x7fe1c81338e0 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 10 monclient: _renew_subs 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1c813b620 con 0x7fe1c81338e0 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1b4004430 con 0x7fe1c81338e0 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1b4004800 con 0x7fe1c81338e0 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c6ffd640 4 set_mon_vals no callback set 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.067+0000 7fe1c57fa640 10 monclient: dump: 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.085 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe1b40049a0 con 0x7fe1c81338e0 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7fe1b4047060 con 0x7fe1c81338e0 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1cf0e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1cf0e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1cf0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a8005080 con 0x7fe1c81338e0 2024-09-16T04:08:36.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.068+0000 7fe1cce80640 10 monclient: get_auth_request con 0x7fe1ac038f60 auth_method 0 2024-09-16T04:08:36.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.073+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe1b4015bf0 con 0x7fe1c81338e0 2024-09-16T04:08:36.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.073+0000 7fe1c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:36.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.073+0000 7fe1c57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:36.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.117+0000 7fb2732f8640 10 monclient: get_monmap_and_config 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.118+0000 7fb2732f8640 10 monclient: build_initial_monmap 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.118+0000 7fb2732f8640 10 monclient: monmap: 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:36.118929+0000 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:36.118929+0000 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:36.120 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.118+0000 7fb2732f8640 1 Processor -- start 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.118+0000 7fb2732f8640 1 -- start start 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.118+0000 7fb2732f8640 10 monclient: init 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient(hunting): picked mon.noname-c con 0x7fb26c0733a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient(hunting): picked mon.noname-a con 0x7fb26c073780 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:36.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient(hunting): picked mon.noname-b con 0x7fb26c13d0b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26c0a3360 con 0x7fb26c073780 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26c06e760 con 0x7fb26c0733a0 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26c06db00 con 0x7fb26c13d0b0 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient(hunting): authenticate will time out at 936.744311s 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb270893640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb26c073780 legacy=0x7fb26c13b990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb271895640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fb26c13d0b0 0x7fb26c13f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59444/0 (socket says 172.21.15.148:59444) 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb271895640 1 -- 172.21.15.148:0/3164429755 learned_addr learned my addr 172.21.15.148:0/3164429755 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093858530 0 0) 0x7fb26c06db00 con 0x7fb26c13d0b0 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): my global_id is 4967 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb258003520 con 0x7fb26c13d0b0 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1462667741 0 0) 0x7fb26c06e760 con 0x7fb26c0733a0 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient(hunting): my global_id is 4939 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb26c06db00 con 0x7fb26c0733a0 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2312602246 0 0) 0x7fb258003520 con 0x7fb26c13d0b0 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 >> v1:172.21.15.171:6789/0 conn(0x7fb26c0733a0 legacy=0x7fb26c09ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 >> v1:172.21.15.148:6789/0 conn(0x7fb26c073780 legacy=0x7fb26c13b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 monclient: found mon.noname-b 2024-09-16T04:08:36.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26c09e570 con 0x7fb26c13d0b0 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb264001990 con 0x7fb26c13d0b0 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: dump: 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.130 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 5 monclient: authenticate success, global_id 4967 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb264001c40 con 0x7fb26c13d0b0 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 1 -- 172.21.15.148:0/3164429755 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb264002010 con 0x7fb26c13d0b0 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb2732f8640 4 set_mon_vals no callback set 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.119+0000 7fb263fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:36.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb263fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb263fff640 10 monclient: dump: 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 10 monclient: shutdown 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 1 -- 172.21.15.148:0/3164429755 >> v1:172.21.15.148:6790/0 conn(0x7fb26c13d0b0 legacy=0x7fb26c13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 1 -- 172.21.15.148:0/3164429755 shutdown_connections 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 1 -- 172.21.15.148:0/3164429755 >> 172.21.15.148:0/3164429755 conn(0x7fb26c06bb90 msgr2=0x7fb26c06bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 1 -- 172.21.15.148:0/3164429755 shutdown_connections 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.120+0000 7fb2732f8640 1 -- 172.21.15.148:0/3164429755 wait complete. 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient: build_initial_monmap 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient: monmap: 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 1 Processor -- start 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 1 -- start start 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient: init 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient(hunting): picked mon.noname-c con 0x7fb26c0733a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient(hunting): picked mon.noname-b con 0x7fb26c073780 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient(hunting): picked mon.noname-a con 0x7fb26c13d0b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26c124470 con 0x7fb26c13d0b0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26c122630 con 0x7fb26c073780 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb26c124750 con 0x7fb26c0733a0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb2732f8640 10 monclient(hunting): authenticate will time out at 936.746934s 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.121+0000 7fb271895640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fb26c13d0b0 legacy=0x7fb26c1d3e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb271094640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fb26c0733a0 0x7fb26c121810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59456/0 (socket says 172.21.15.148:59456) 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb271094640 1 -- 172.21.15.148:0/1171414710 learned_addr learned my addr 172.21.15.148:0/1171414710 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 777890078 0 0) 0x7fb26c124750 con 0x7fb26c0733a0 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): my global_id is 4970 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb250003520 con 0x7fb26c0733a0 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171860591 0 0) 0x7fb26c122630 con 0x7fb26c073780 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): my global_id is 4942 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb26c124750 con 0x7fb26c073780 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 218608449 0 0) 0x7fb26c124750 con 0x7fb26c073780 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb26c122630 con 0x7fb26c073780 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3627063069 0 0) 0x7fb250003520 con 0x7fb26c0733a0 2024-09-16T04:08:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb26c124750 con 0x7fb26c0733a0 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb25c0040c0 con 0x7fb26c073780 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.137 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb2680017d0 con 0x7fb26c0733a0 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient(hunting): dump: 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1052322200 0 0) 0x7fb26c122630 con 0x7fb26c073780 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 >> v1:172.21.15.148:6790/0 conn(0x7fb26c0733a0 legacy=0x7fb26c121810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 >> v1:172.21.15.148:6789/0 conn(0x7fb26c13d0b0 legacy=0x7fb26c1d3e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 monclient: found mon.b 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb26c0a4cb0 con 0x7fb26c073780 2024-09-16T04:08:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 5 monclient: authenticate success, global_id 4942 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 10 monclient: _renew_subs 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb26c1405e0 con 0x7fb26c073780 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb25c004650 con 0x7fb26c073780 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb25c004a20 con 0x7fb26c073780 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb261ffb640 10 monclient: dump: 2024-09-16T04:08:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2637fe640 4 set_mon_vals no callback set 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 10 monclient: _renew_subs 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.122+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb26c122d40 con 0x7fb26c073780 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb25c00fed0 con 0x7fb26c073780 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7fb25c046ff0 con 0x7fb26c073780 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb2732f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb2732f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb2732f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26c0a1f00 con 0x7fb26c073780 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.123+0000 7fb271094640 10 monclient: get_auth_request con 0x7fb250038eb0 auth_method 0 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.128+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb26c0a1f00 con 0x7fb26c073780 2024-09-16T04:08:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.128+0000 7fb261ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.128+0000 7fb261ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:36.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.242+0000 7fe1cf0e4640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:36.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.242+0000 7fe1cf0e4640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:36.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.242+0000 7fe1cf0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:36.242+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7fe1a8005370 con 0x7fe1c81338e0 2024-09-16T04:08:36.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.358+0000 7fb2732f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:36.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.358+0000 7fb2732f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:36.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.358+0000 7fb2732f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:36.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.358+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb26c071830 con 0x7fb26c073780 2024-09-16T04:08:36.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.359+0000 7fb261ffb640 1 -- 172.21.15.148:0/1171414710 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 271008810) 0x7fb25c015c00 con 0x7fb26c073780 2024-09-16T04:08:36.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.359+0000 7fb261ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:36.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.359+0000 7fb261ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:36.362 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:36.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 >> v1:172.21.15.171:6812/7327273 conn(0x7fb250038eb0 legacy=0x7fb25003b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 10 monclient: shutdown 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 >> v1:172.21.15.171:6789/0 conn(0x7fb26c073780 legacy=0x7fb26c121f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 shutdown_connections 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 >> 172.21.15.148:0/1171414710 conn(0x7fb26c06bb90 msgr2=0x7fb26c06bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 shutdown_connections 2024-09-16T04:08:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:36.361+0000 7fb2732f8640 1 -- 172.21.15.148:0/1171414710 wait complete. 2024-09-16T04:08:39.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:39.066+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:39.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:39.066+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:39.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:39.066+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:39.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:39.066+0000 7fe1b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:39.375 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.480+0000 7f234c082640 10 monclient: get_monmap_and_config 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.480+0000 7f234c082640 10 monclient: build_initial_monmap 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.480+0000 7f234c082640 10 monclient: monmap: 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:39.481103+0000 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:39.481103+0000 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:39.482 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:39.483 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:39.483 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:39.483 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:39.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.480+0000 7f234c082640 1 Processor -- start 2024-09-16T04:08:39.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 1 -- start start 2024-09-16T04:08:39.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient: init 2024-09-16T04:08:39.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:39.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:39.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient(hunting): picked mon.noname-a con 0x7f2344098400 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient(hunting): picked mon.noname-c con 0x7f23440987e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient(hunting): picked mon.noname-b con 0x7f2344098bf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f234409ced0 con 0x7f2344098400 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2344137480 con 0x7f23440987e0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f234409cbf0 con 0x7f2344098bf0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234c082640 10 monclient(hunting): authenticate will time out at 940.106606s 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f2349e1e640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2344098400 legacy=0x7f2344136d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234a61f640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2344098bf0 0x7f234413f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59462/0 (socket says 172.21.15.148:59462) 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.481+0000 7f234a61f640 1 -- 172.21.15.148:0/2641152846 learned_addr learned my addr 172.21.15.148:0/2641152846 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555578595 0 0) 0x7f234409cbf0 con 0x7f2344098bf0 2024-09-16T04:08:39.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient(hunting): _negotiate 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient(hunting): my global_id is 4973 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2330003520 con 0x7f2344098bf0 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1635713098 0 0) 0x7f2330003520 con 0x7f2344098bf0 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 >> v1:172.21.15.171:6789/0 conn(0x7f23440987e0 legacy=0x7f234413bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 >> v1:172.21.15.148:6789/0 conn(0x7f2344098400 legacy=0x7f2344136d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: _finish_hunting 0 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 monclient: found mon.noname-b 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:39.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23440a16d0 con 0x7f2344098bf0 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: _finish_auth 0 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: _check_auth_tickets 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f233c001990 con 0x7f2344098bf0 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 5 monclient: authenticate success, global_id 4973 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:39.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: dump: 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f233c001c80 con 0x7f2344098bf0 2024-09-16T04:08:39.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 1 -- 172.21.15.148:0/2641152846 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f233c002090 con 0x7f2344098bf0 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f2348e1c640 10 monclient: dump: 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:39.494 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 4 set_mon_vals no callback set 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 10 monclient: shutdown 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 1 -- 172.21.15.148:0/2641152846 >> v1:172.21.15.148:6790/0 conn(0x7f2344098bf0 legacy=0x7f234413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 1 -- 172.21.15.148:0/2641152846 shutdown_connections 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 1 -- 172.21.15.148:0/2641152846 >> 172.21.15.148:0/2641152846 conn(0x7f23440a5860 msgr2=0x7f23440a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.482+0000 7f234c082640 1 -- 172.21.15.148:0/2641152846 shutdown_connections 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.483+0000 7f234c082640 1 -- 172.21.15.148:0/2641152846 wait complete. 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.483+0000 7f234c082640 10 monclient: build_initial_monmap 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.483+0000 7f234c082640 10 monclient: monmap: 2024-09-16T04:08:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.483+0000 7f234c082640 1 Processor -- start 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.483+0000 7f234c082640 1 -- start start 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.483+0000 7f234c082640 10 monclient: init 2024-09-16T04:08:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient(hunting): picked mon.noname-a con 0x7f2344098400 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient(hunting): picked mon.noname-b con 0x7f23440987e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient(hunting): picked mon.noname-c con 0x7f2344098bf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f234413e2a0 con 0x7f2344098400 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f234409ed60 con 0x7f23440987e0 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23440a49b0 con 0x7f2344098bf0 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f2349e1e640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2344098400 legacy=0x7f234413c340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234c082640 10 monclient(hunting): authenticate will time out at 940.109387s 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234a61f640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2344098bf0 0x7f23441d3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59472/0 (socket says 172.21.15.148:59472) 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f234a61f640 1 -- 172.21.15.148:0/3461970753 learned_addr learned my addr 172.21.15.148:0/3461970753 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669405310 0 0) 0x7f23440a16d0 con 0x7f2344098bf0 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): my global_id is 4976 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2328003520 con 0x7f2344098bf0 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1029788826 0 0) 0x7f2344137480 con 0x7f23440987e0 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): my global_id is 4948 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23440a16d0 con 0x7f23440987e0 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2054856817 0 0) 0x7f2328003520 con 0x7f2344098bf0 2024-09-16T04:08:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2344137480 con 0x7f2344098bf0 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f233c0017b0 con 0x7f2344098bf0 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): dump: 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:39.499 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.484+0000 7f233a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1010365769 0 0) 0x7f2344137480 con 0x7f2344098bf0 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 >> v1:172.21.15.171:6789/0 conn(0x7f23440987e0 legacy=0x7f234413ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 >> v1:172.21.15.148:6789/0 conn(0x7f2344098400 legacy=0x7f234413c340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 monclient: found mon.c 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234409c630 con 0x7f2344098bf0 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 5 monclient: authenticate success, global_id 4976 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 10 monclient: _renew_subs 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f234413d160 con 0x7f2344098bf0 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 10 monclient: _renew_subs 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f234413fb00 con 0x7f2344098bf0 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f233c005040 con 0x7f2344098bf0 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f233c0051f0 con 0x7f2344098bf0 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233bfff640 4 set_mon_vals no callback set 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.485+0000 7f233a7fc640 10 monclient: dump: 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f233c010620 con 0x7f2344098bf0 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f234c082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f234c082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f234c082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f232c005080 con 0x7f2344098bf0 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7f233c047350 con 0x7f2344098bf0 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.486+0000 7f2349e1e640 10 monclient: get_auth_request con 0x7f2328038d30 auth_method 0 2024-09-16T04:08:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.490+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f234409c630 con 0x7f2344098bf0 2024-09-16T04:08:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.490+0000 7f233a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.490+0000 7f233a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:39.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f234c082640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f234c082640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f234c082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f232c005370 con 0x7f2344098bf0 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f233a7fc640 1 -- 172.21.15.148:0/3461970753 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3066092216) 0x7f23440a49b0 con 0x7f2344098bf0 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f233a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.720+0000 7f233a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:39.723 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.722+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 >> v1:172.21.15.171:6812/7327273 conn(0x7f2328038d30 legacy=0x7f232803b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.722+0000 7f234c082640 10 monclient: shutdown 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.722+0000 7f234c082640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.722+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 >> v1:172.21.15.148:6790/0 conn(0x7f2344098bf0 legacy=0x7f23441d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.722+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 shutdown_connections 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.722+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 >> 172.21.15.148:0/3461970753 conn(0x7f23440a5860 msgr2=0x7f234409bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.723+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 shutdown_connections 2024-09-16T04:08:39.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:39.723+0000 7f234c082640 1 -- 172.21.15.148:0/3461970753 wait complete. 2024-09-16T04:08:40.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:40.066+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:40.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:40.066+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:40.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:40.066+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:41.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:41.066+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:41.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:41.066+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:41.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:41.066+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:42.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:42.067+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:42.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:42.067+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:42.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:42.067+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:42.736 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.842+0000 7f140b131640 10 monclient: get_monmap_and_config 2024-09-16T04:08:42.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.842+0000 7f140b131640 10 monclient: build_initial_monmap 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.842+0000 7f140b131640 10 monclient: monmap: 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:42.843668+0000 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:42.843668+0000 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:42.845 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.843+0000 7f140b131640 1 Processor -- start 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.843+0000 7f140b131640 1 -- start start 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.843+0000 7f140b131640 10 monclient: init 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient(hunting): picked mon.noname-a con 0x7f1404133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient(hunting): picked mon.noname-b con 0x7f140406c430 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:42.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient(hunting): picked mon.noname-c con 0x7f140413d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140409b0d0 con 0x7f1404133910 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140409b340 con 0x7f140413d0e0 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140409b620 con 0x7f140406c430 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f1408ecd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f1404133cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f140b131640 10 monclient(hunting): authenticate will time out at 943.469122s 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f1403fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f140406c430 0x7f140413b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56006/0 (socket says 172.21.15.148:56006) 2024-09-16T04:08:42.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:42.844+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 learned_addr learned my addr 172.21.15.148:0/110552659 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:43.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:43.045+0000 7f1408ecd640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f1404133cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:43.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:43.066+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:43.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:43.066+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:43.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:43.066+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:43.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:43.446+0000 7f1408ecd640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f1404133cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:44.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:44.067+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:44.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:44.067+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:44.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:44.067+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:44.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:44.247+0000 7f1408ecd640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f1404133cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:45.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:45.067+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:45.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:45.067+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:45.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:45.067+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:45.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): tick 2024-09-16T04:08:46.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:08:46.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:08:46.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6790/0 conn(0x7f140406c430 legacy=0x7f140413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.171:6789/0 conn(0x7f140413d0e0 legacy=0x7f140413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f1404133cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): picked mon.noname-b con 0x7f13e80036c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): picked mon.noname-a con 0x7f1404133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): picked mon.noname-c con 0x7f140413d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14040633c0 con 0x7f1404133910 2024-09-16T04:08:46.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140409b0d0 con 0x7f140413d0e0 2024-09-16T04:08:46.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e8010360 con 0x7f13e80036c0 2024-09-16T04:08:46.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f14027fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:46.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:45.843+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:46.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:46.044+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:46.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:46.067+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:46.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:46.067+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:46.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:46.067+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:46.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:46.445+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:47.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:47.067+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:47.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:47.067+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:47.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:47.067+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:47.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:47.246+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:48.067+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:48.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:48.067+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:48.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:48.067+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): tick 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6790/0 conn(0x7f13e80036c0 legacy=0x7f13e8005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.171:6789/0 conn(0x7f140413d0e0 legacy=0x7f13e800cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8009320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): picked mon.noname-c con 0x7f13e800f410 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): picked mon.noname-a con 0x7f1404133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:48.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): picked mon.noname-b con 0x7f13e80036c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:48.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e80115b0 con 0x7f1404133910 2024-09-16T04:08:48.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14040633c0 con 0x7f13e800f410 2024-09-16T04:08:48.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e80100d0 con 0x7f13e80036c0 2024-09-16T04:08:48.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f14027fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:48.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:48.844+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8010530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:49.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:49.045+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8010530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:49.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:49.068+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:49.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:49.068+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:49.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:49.068+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:49.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:49.068+0000 7fe1b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:49.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:49.446+0000 7f1403fff640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8010530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.068+0000 7fe1b3fff640 10 monclient: tick 2024-09-16T04:08:50.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.068+0000 7fe1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.068+0000 7fe1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2892086879 0 0) 0x7f13e80100d0 con 0x7f13e80036c0 2024-09-16T04:08:50.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient(hunting): my global_id is 4979 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13f4003520 con 0x7f13e80036c0 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3225581181 0 0) 0x7f13f4003520 con 0x7f13e80036c0 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.171:6789/0 conn(0x7f13e800f410 legacy=0x7f13e800f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f13e8010530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 monclient: found mon.noname-b 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e800cb60 con 0x7f13e80036c0 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13fc001890 con 0x7f13e80036c0 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f140b131640 5 monclient: authenticate success, global_id 4979 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f140b131640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: dump: 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.134 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f140b131640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13fc001b20 con 0x7f13e80036c0 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 1 -- 172.21.15.148:0/110552659 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13fc001f30 con 0x7f13e80036c0 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f14037fe640 10 monclient: dump: 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f140b131640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.129+0000 7f140b131640 4 set_mon_vals no callback set 2024-09-16T04:08:50.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 10 monclient: shutdown 2024-09-16T04:08:50.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 1 -- 172.21.15.148:0/110552659 >> v1:172.21.15.148:6790/0 conn(0x7f13e80036c0 legacy=0x7f13e8005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 1 -- 172.21.15.148:0/110552659 shutdown_connections 2024-09-16T04:08:50.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 1 -- 172.21.15.148:0/110552659 >> 172.21.15.148:0/110552659 conn(0x7f1404062630 msgr2=0x7f1404062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 1 -- 172.21.15.148:0/110552659 shutdown_connections 2024-09-16T04:08:50.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.130+0000 7f140b131640 1 -- 172.21.15.148:0/110552659 wait complete. 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.131+0000 7f140b131640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.131+0000 7f140b131640 10 monclient: monmap: 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.141 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.131+0000 7f140b131640 1 Processor -- start 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.131+0000 7f140b131640 1 -- start start 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.131+0000 7f140b131640 10 monclient: init 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient(hunting): picked mon.noname-c con 0x7f13e80036c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient(hunting): picked mon.noname-b con 0x7f13e800f410 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient(hunting): picked mon.noname-a con 0x7f1404133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140413bd60 con 0x7f1404133910 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140413a030 con 0x7f13e800f410 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140413a4d0 con 0x7f13e80036c0 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f14096ce640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f14041d4030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f140b131640 10 monclient(hunting): authenticate will time out at 950.757637s 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f1408ecd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f13e80036c0 0x7f1404130870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56040/0 (socket says 172.21.15.148:56040) 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.132+0000 7f1408ecd640 1 -- 172.21.15.148:0/1648424177 learned_addr learned my addr 172.21.15.148:0/1648424177 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443101513 0 0) 0x7f140413a030 con 0x7f13e800f410 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): my global_id is 4951 2024-09-16T04:08:50.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13e80115b0 con 0x7f13e800f410 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1931068540 0 0) 0x7f13e80115b0 con 0x7f13e800f410 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f140413a030 con 0x7f13e800f410 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13f0001830 con 0x7f13e800f410 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): dump: 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.144 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086397197 0 0) 0x7f140413a4d0 con 0x7f13e80036c0 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient(hunting): my global_id is 4982 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13e80115b0 con 0x7f13e80036c0 2024-09-16T04:08:50.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1367147022 0 0) 0x7f140413a030 con 0x7f13e800f410 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 >> v1:172.21.15.148:6790/0 conn(0x7f13e80036c0 legacy=0x7f1404130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 >> v1:172.21.15.148:6789/0 conn(0x7f1404133910 legacy=0x7f14041d4030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 monclient: found mon.b 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140413ab90 con 0x7f13e800f410 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f14017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 5 monclient: authenticate success, global_id 4951 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 10 monclient: _renew_subs 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f140413a910 con 0x7f13e800f410 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 10 monclient: _renew_subs 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.136+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14041cc470 con 0x7f13e800f410 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13f00047e0 con 0x7f13e800f410 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13f0004b30 con 0x7f13e800f410 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 10 monclient: dump: 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.147 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f1402ffd640 4 set_mon_vals no callback set 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.137+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f13f0004d50 con 0x7f13e800f410 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.138+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(61..61 src has 1..61) ==== 4377+0+0 (unknown 3033558676 0 0) 0x7f13f0047040 con 0x7f13e800f410 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.138+0000 7f140b131640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.138+0000 7f140b131640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.138+0000 7f140b131640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.138+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e4005080 con 0x7f13e800f410 2024-09-16T04:08:50.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.138+0000 7f1408ecd640 10 monclient: get_auth_request con 0x7f13e8039f80 auth_method 0 2024-09-16T04:08:50.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.143+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f13f0015c30 con 0x7f13e800f410 2024-09-16T04:08:50.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.143+0000 7f14017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.143+0000 7f14017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:50.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.192+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v62) ==== 80+0+0 (unknown 657111845 0 0) 0x7fe1b40198a0 con 0x7fe1c81338e0 2024-09-16T04:08:50.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.193+0000 7fe1c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.193+0000 7fe1c57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.299+0000 7fe1cf0e4640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.299+0000 7fe1cf0e4640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.299+0000 7fe1cf0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.299+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7fe1a8002210 con 0x7fe1c81338e0 2024-09-16T04:08:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.300+0000 7fe1c57fa640 1 -- 172.21.15.171:0/3173043278 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v62) ==== 103+0+0 (unknown 4238528152 0 0) 0x7fe1b4045bd0 con 0x7fe1c81338e0 2024-09-16T04:08:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.300+0000 7fe1c57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.300+0000 7fe1c57fa640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T04:08:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo does not exist 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 >> v1:172.21.15.171:6812/7327273 conn(0x7fe1ac038f60 legacy=0x7fe1ac03b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 10 monclient: shutdown 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 >> v1:172.21.15.171:6789/0 conn(0x7fe1c81338e0 legacy=0x7fe1c81d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 shutdown_connections 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 >> 172.21.15.171:0/3173043278 conn(0x7fe1c80a28d0 msgr2=0x7fe1c80a44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 shutdown_connections 2024-09-16T04:08:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.302+0000 7fe1cf0e4640 1 -- 172.21.15.171:0/3173043278 wait complete. 2024-09-16T04:08:50.313 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm foo 2024-09-16T04:08:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.382+0000 7f140b131640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.382+0000 7f140b131640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.382+0000 7f140b131640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.382+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f13e4005370 con 0x7f13e800f410 2024-09-16T04:08:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.383+0000 7f14017fa640 1 -- 172.21.15.148:0/1648424177 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 4107659938) 0x7f13f0015530 con 0x7f13e800f410 2024-09-16T04:08:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.383+0000 7f14017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.383+0000 7f14017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:50.385 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 >> v1:172.21.15.171:6812/7327273 conn(0x7f13e8039f80 legacy=0x7f13e803c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 10 monclient: shutdown 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 >> v1:172.21.15.171:6789/0 conn(0x7f13e800f410 legacy=0x7f14041d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 shutdown_connections 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 >> 172.21.15.148:0/1648424177 conn(0x7f1404062630 msgr2=0x7f14040a3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 shutdown_connections 2024-09-16T04:08:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.385+0000 7f140b131640 1 -- 172.21.15.148:0/1648424177 wait complete. 2024-09-16T04:08:50.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.390+0000 7fbdc6531640 10 monclient: get_monmap_and_config 2024-09-16T04:08:50.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.390+0000 7fbdc6531640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.390+0000 7fbdc6531640 10 monclient: monmap: 2024-09-16T04:08:50.392 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:50.392 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.392 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:50.391448+0000 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:50.391448+0000 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:50.393 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.402 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:08:50.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:08:50.403 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:08:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 1 Processor -- start 2024-09-16T04:08:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 1 -- start start 2024-09-16T04:08:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient: init 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient(hunting): picked mon.noname-c con 0x7fbdc0132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient(hunting): picked mon.noname-b con 0x7fbdc009ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient(hunting): picked mon.noname-a con 0x7fbdc0132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc01332b0 con 0x7fbdc0132e60 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc0134310 con 0x7fbdc0132a80 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc009d980 con 0x7fbdc009ec20 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.391+0000 7fbdc6531640 10 monclient(hunting): authenticate will time out at 905.896944s 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc4ace640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbdc0132e60 legacy=0x7fbdc013c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbdc0132a80 0x7fbdc01334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51076/0 (socket says 172.21.15.171:51076) 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbffff640 1 -- 172.21.15.171:0/292349068 learned_addr learned my addr 172.21.15.171:0/292349068 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 600967972 0 0) 0x7fbdc0134310 con 0x7fbdc0132a80 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): my global_id is 4954 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbda8003520 con 0x7fbdc0132a80 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436600255 0 0) 0x7fbdc009d980 con 0x7fbdc009ec20 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient(hunting): my global_id is 4985 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc0134310 con 0x7fbdc009ec20 2024-09-16T04:08:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 367617150 0 0) 0x7fbda8003520 con 0x7fbdc0132a80 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 >> v1:172.21.15.148:6790/0 conn(0x7fbdc009ec20 legacy=0x7fbdc0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 >> v1:172.21.15.148:6789/0 conn(0x7fbdc0132e60 legacy=0x7fbdc013c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 monclient: found mon.noname-c 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc009dc60 con 0x7fbdc0132a80 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdb0001610 con 0x7fbdc0132a80 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 5 monclient: authenticate success, global_id 4954 2024-09-16T04:08:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: dump: 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.407 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdb00018c0 con 0x7fbdc0132a80 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 1 -- 172.21.15.171:0/292349068 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdb0001c90 con 0x7fbdc0132a80 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 4 set_mon_vals no callback set 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdbeffd640 10 monclient: dump: 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 10 monclient: shutdown 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.392+0000 7fbdc6531640 1 -- 172.21.15.171:0/292349068 >> v1:172.21.15.171:6789/0 conn(0x7fbdc0132a80 legacy=0x7fbdc01334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 1 -- 172.21.15.171:0/292349068 shutdown_connections 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 1 -- 172.21.15.171:0/292349068 >> 172.21.15.171:0/292349068 conn(0x7fbdc00a52a0 msgr2=0x7fbdc00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 1 -- 172.21.15.171:0/292349068 shutdown_connections 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 1 -- 172.21.15.171:0/292349068 wait complete. 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 10 monclient: monmap: 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.393+0000 7fbdc6531640 1 Processor -- start 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 1 -- start start 2024-09-16T04:08:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient: init 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient(hunting): picked mon.noname-c con 0x7fbdc009ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient(hunting): picked mon.noname-a con 0x7fbdc0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient(hunting): picked mon.noname-b con 0x7fbdc0132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc0131a10 con 0x7fbdc0132a80 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc011dc10 con 0x7fbdc0132e60 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc00a3410 con 0x7fbdc009ec20 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc6531640 10 monclient(hunting): authenticate will time out at 905.899648s 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbf7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbdc0132a80 legacy=0x7fbdc011d500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc4ace640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbdc0132e60 0x7fbdc01d3a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51088/0 (socket says 172.21.15.171:51088) 2024-09-16T04:08:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdc4ace640 1 -- 172.21.15.171:0/2021922572 learned_addr learned my addr 172.21.15.171:0/2021922572 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3644451851 0 0) 0x7fbdc011dc10 con 0x7fbdc0132e60 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): my global_id is 4957 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdac003520 con 0x7fbdc0132e60 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3198414888 0 0) 0x7fbdc00a3410 con 0x7fbdc009ec20 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 10 monclient(hunting): my global_id is 4988 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.394+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc011dc10 con 0x7fbdc009ec20 2024-09-16T04:08:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2342795427 0 0) 0x7fbdac003520 con 0x7fbdc0132e60 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc00a3410 con 0x7fbdc0132e60 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdb8004440 con 0x7fbdc0132e60 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): dump: 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.413 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 255646032 0 0) 0x7fbdc011dc10 con 0x7fbdc009ec20 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdac003520 con 0x7fbdc009ec20 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdb0001c10 con 0x7fbdc009ec20 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient(hunting): dump: 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.414 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1748129593 0 0) 0x7fbdc00a3410 con 0x7fbdc0132e60 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 >> v1:172.21.15.148:6790/0 conn(0x7fbdc009ec20 legacy=0x7fbdc011cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 >> v1:172.21.15.148:6789/0 conn(0x7fbdc0132a80 legacy=0x7fbdc011d500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 monclient: found mon.b 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc01239f0 con 0x7fbdc0132e60 2024-09-16T04:08:50.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 5 monclient: authenticate success, global_id 4957 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 10 monclient: _renew_subs 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc0123cd0 con 0x7fbdc0132e60 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 10 monclient: _renew_subs 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbdc011df20 con 0x7fbdc0132e60 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdb80046f0 con 0x7fbdc0132e60 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdb8004ac0 con 0x7fbdc0132e60 2024-09-16T04:08:50.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbe7fc640 4 set_mon_vals no callback set 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.395+0000 7fbdbcff9640 10 monclient: dump: 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.417 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbdb8004c60 con 0x7fbdc0132e60 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdc6531640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdc6531640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdc6531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda0005080 con 0x7fbdc0132e60 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4322+0+0 (unknown 1588246179 0 0) 0x7fbdb8047320 con 0x7fbdc0132e60 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.396+0000 7fbdbffff640 10 monclient: get_auth_request con 0x7fbdac038f60 auth_method 0 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.400+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbdb8015f40 con 0x7fbdc0132e60 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.400+0000 7fbdbcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.400+0000 7fbdbcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:50.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.514+0000 7f77b4e8d640 10 monclient: get_monmap_and_config 2024-09-16T04:08:50.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.514+0000 7f77b4e8d640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.514+0000 7f77b4e8d640 10 monclient: monmap: 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:50.515414+0000 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:50.515414+0000 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:50.517 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 1 Processor -- start 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 1 -- start start 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient: init 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f77b009bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f77b009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f77b009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b00a1010 con 0x7f77b009c660 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b009f030 con 0x7f77b009bb70 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b009dc60 con 0x7f77b009cc30 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77b4e8d640 10 monclient(hunting): authenticate will time out at 951.140747s 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77a7fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f77b009c660 legacy=0x7f77b0138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77aed9d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f77b009cc30 0x7f77b013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56054/0 (socket says 172.21.15.148:56054) 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.515+0000 7f77aed9d640 1 -- 172.21.15.148:0/4126072415 learned_addr learned my addr 172.21.15.148:0/4126072415 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3394553252 0 0) 0x7f77b009dc60 con 0x7f77b009cc30 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): my global_id is 4991 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f779c003520 con 0x7f77b009cc30 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566818860 0 0) 0x7f77b009f030 con 0x7f77b009bb70 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient(hunting): my global_id is 4960 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77b009dc60 con 0x7f77b009bb70 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3989762202 0 0) 0x7f779c003520 con 0x7f77b009cc30 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 >> v1:172.21.15.171:6789/0 conn(0x7f77b009bb70 legacy=0x7f77b009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 >> v1:172.21.15.148:6789/0 conn(0x7f77b009c660 legacy=0x7f77b0138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 monclient: found mon.noname-b 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b009df40 con 0x7f77b009cc30 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a8001990 con 0x7f77b009cc30 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: dump: 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 5 monclient: authenticate success, global_id 4991 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77a8001c40 con 0x7f77b009cc30 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 1 -- 172.21.15.148:0/4126072415 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a8002010 con 0x7f77b009cc30 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77add9b640 10 monclient: dump: 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 4 set_mon_vals no callback set 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 10 monclient: shutdown 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 1 -- 172.21.15.148:0/4126072415 >> v1:172.21.15.148:6790/0 conn(0x7f77b009cc30 legacy=0x7f77b013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 1 -- 172.21.15.148:0/4126072415 shutdown_connections 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 1 -- 172.21.15.148:0/4126072415 >> 172.21.15.148:0/4126072415 conn(0x7f77b00a5640 msgr2=0x7f77b00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 1 -- 172.21.15.148:0/4126072415 shutdown_connections 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.516+0000 7f77b4e8d640 1 -- 172.21.15.148:0/4126072415 wait complete. 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.517+0000 7f77b4e8d640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.517+0000 7f77b4e8d640 10 monclient: monmap: 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.517+0000 7f77b4e8d640 1 Processor -- start 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.517+0000 7f77b4e8d640 1 -- start start 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.517+0000 7f77b4e8d640 10 monclient: init 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f77b009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f77b009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f77b009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b011cd30 con 0x7f77b009bb70 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b00740a0 con 0x7f77b009cc30 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b006e340 con 0x7f77b009c660 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77b4e8d640 10 monclient(hunting): authenticate will time out at 951.143420s 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77ae59c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f77b009bb70 legacy=0x7f77b01c9a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a7fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f77b009c660 0x7f77b01cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56070/0 (socket says 172.21.15.148:56070) 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a7fff640 1 -- 172.21.15.148:0/482014106 learned_addr learned my addr 172.21.15.148:0/482014106 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984769719 0 0) 0x7f77b006e340 con 0x7f77b009c660 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): my global_id is 4994 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7790003520 con 0x7f77b009c660 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514792064 0 0) 0x7f77b00740a0 con 0x7f77b009cc30 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): my global_id is 4963 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77b006e340 con 0x7f77b009cc30 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1580447049 0 0) 0x7f7790003520 con 0x7f77b009c660 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77b00740a0 con 0x7f77b009c660 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7798004150 con 0x7f77b009c660 2024-09-16T04:08:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.536 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.518+0000 7f77a6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2059144357 0 0) 0x7f77b006e340 con 0x7f77b009cc30 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7790003520 con 0x7f77b009cc30 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1063241678 0 0) 0x7f77b00740a0 con 0x7f77b009c660 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.171:6789/0 conn(0x7f77b009cc30 legacy=0x7f77b01d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.148:6789/0 conn(0x7f77b009bb70 legacy=0x7f77b01c9a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 monclient: found mon.c 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b011c770 con 0x7f77b009c660 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 5 monclient: authenticate success, global_id 4994 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 10 monclient: _renew_subs 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77b011ca50 con 0x7f77b009c660 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 10 monclient: _renew_subs 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f77b013b520 con 0x7f77b009c660 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7798004400 con 0x7f77b009c660 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77980047d0 con 0x7f77b009c660 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77ad59a640 4 set_mon_vals no callback set 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 10 monclient: dump: 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f77940065c0 con 0x7f7794002860 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.519+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7798004970 con 0x7f77b009c660 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4322+0+0 (unknown 1588246179 0 0) 0x7f7798047080 con 0x7f77b009c660 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959026159 0 0) 0x7f77940065c0 con 0x7f7794002860 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient(hunting): my global_id is 4997 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77b00740a0 con 0x7f7794002860 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77aed9d640 10 monclient: get_auth_request con 0x7f7790038eb0 auth_method 0 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4040679505 0 0) 0x7f77b00740a0 con 0x7f7794002860 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77940065c0 con 0x7f7794002860 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a00015e0 con 0x7f7794002860 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient: dump: 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2542778353 0 0) 0x7f77940065c0 con 0x7f7794002860 2024-09-16T04:08:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f779400a150 con 0x7f7794002860 2024-09-16T04:08:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f77a0001920 con 0x7f7794002860 2024-09-16T04:08:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.148:6790/0 conn(0x7f7794002860 legacy=0x7f7794004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.520+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.148:6790/0 conn(0x7f7794002860 legacy=0x7f7794004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.526+0000 7f77b4e8d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.526+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7794009e40 con 0x7f7794009100 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2222711130 0 0) 0x7f7794009e40 con 0x7f7794009100 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient(hunting): my global_id is 5000 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77940065c0 con 0x7f7794009100 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2310010837 0 0) 0x7f77940065c0 con 0x7f7794009100 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7794009e40 con 0x7f7794009100 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a0003610 con 0x7f7794009100 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient: dump: 2024-09-16T04:08:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1662312234 0 0) 0x7f7794009e40 con 0x7f7794009100 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f779400c3d0 con 0x7f7794009100 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4133 (unknown 0 0 3435228722) 0x7f77a00057c0 con 0x7f7794009100 2024-09-16T04:08:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.148:6790/0 conn(0x7f7794009100 legacy=0x7f7794009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77a6ffd640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.148:6790/0 conn(0x7f7794009100 legacy=0x7f7794009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 18, 2024-09-16T04:08:50.546 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 154606, 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:50.547 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:50.548 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:50.549 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:50.550 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:50.558 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:50.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.171:6812/7327273 conn(0x7f7790038eb0 legacy=0x7f779003b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77b4e8d640 10 monclient: shutdown 2024-09-16T04:08:50.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77b4e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.527+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 >> v1:172.21.15.148:6790/0 conn(0x7f77b009c660 legacy=0x7f77b01cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.528+0000 7f77aed9d640 1 -- 172.21.15.148:0/482014106 reap_dead start 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.528+0000 7f77aed9d640 1 -- 172.21.15.148:0/482014106 reap_dead start 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.528+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 shutdown_connections 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.528+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 >> 172.21.15.148:0/482014106 conn(0x7f77b00a5640 msgr2=0x7f77b006e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.528+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 shutdown_connections 2024-09-16T04:08:50.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.528+0000 7f77b4e8d640 1 -- 172.21.15.148:0/482014106 wait complete. 2024-09-16T04:08:50.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:08:50.560 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:08:50.561 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:08:50.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.572+0000 7fbdc6531640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.572+0000 7fbdc6531640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.572+0000 7fbdc6531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.572+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7fbda0005370 con 0x7fbdc0132e60 2024-09-16T04:08:50.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.573+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v62) ==== 103+0+0 (unknown 4238528152 0 0) 0x7fbdb8015840 con 0x7fbdc0132e60 2024-09-16T04:08:50.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.573+0000 7fbdbcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.573+0000 7fbdbcff9640 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-16T04:08:50.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.656+0000 7f9354118640 10 monclient: get_monmap_and_config 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.656+0000 7f9354118640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.657+0000 7f9354118640 10 monclient: monmap: 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:08:50.657919+0000 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:08:50.657919+0000 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:50.659 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:50.660 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.657+0000 7f9354118640 1 Processor -- start 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.657+0000 7f9354118640 1 -- start start 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.657+0000 7f9354118640 10 monclient: init 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:50.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient(hunting): picked mon.noname-c con 0x7f934c06de70 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient(hunting): picked mon.noname-b con 0x7f934c0a3e20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient(hunting): picked mon.noname-a con 0x7f934c13d1b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934c06e8f0 con 0x7f934c13d1b0 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934c0a0840 con 0x7f934c06de70 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934c0a4d70 con 0x7f934c0a3e20 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient(hunting): authenticate will time out at 951.283321s 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f93526b5640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f934c13d1b0 legacy=0x7f934c13f5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f93516b3640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f934c0a3e20 0x7f934c13ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56100/0 (socket says 172.21.15.148:56100) 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f93516b3640 1 -- 172.21.15.148:0/2034071485 learned_addr learned my addr 172.21.15.148:0/2034071485 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065760635 0 0) 0x7f934c0a4d70 con 0x7f934c0a3e20 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): my global_id is 5003 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9338003520 con 0x7f934c0a3e20 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2539101279 0 0) 0x7f934c0a0840 con 0x7f934c06de70 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient(hunting): my global_id is 4966 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f934c0a4d70 con 0x7f934c06de70 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 402348725 0 0) 0x7f9338003520 con 0x7f934c0a3e20 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 >> v1:172.21.15.171:6789/0 conn(0x7f934c06de70 legacy=0x7f934c0a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 >> v1:172.21.15.148:6789/0 conn(0x7f934c13d1b0 legacy=0x7f934c13f5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:50.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 monclient: found mon.noname-b 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934c0a2d20 con 0x7f934c0a3e20 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f933c001610 con 0x7f934c0a3e20 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 5 monclient: authenticate success, global_id 5003 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:50.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: dump: 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f933c001930 con 0x7f934c0a3e20 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 1 -- 172.21.15.148:0/2034071485 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f933c001d00 con 0x7f934c0a3e20 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9354118640 4 set_mon_vals no callback set 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.658+0000 7f9350eb2640 10 monclient: dump: 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.673 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 10 monclient: shutdown 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 1 -- 172.21.15.148:0/2034071485 >> v1:172.21.15.148:6790/0 conn(0x7f934c0a3e20 legacy=0x7f934c13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 1 -- 172.21.15.148:0/2034071485 shutdown_connections 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 1 -- 172.21.15.148:0/2034071485 >> 172.21.15.148:0/2034071485 conn(0x7f934c0a5a40 msgr2=0x7f934c0a5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 1 -- 172.21.15.148:0/2034071485 shutdown_connections 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.659+0000 7f9354118640 1 -- 172.21.15.148:0/2034071485 wait complete. 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient: monmap: 2024-09-16T04:08:50.674 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 1 Processor -- start 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 1 -- start start 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient: init 2024-09-16T04:08:50.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient(hunting): picked mon.noname-a con 0x7f934c06de70 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient(hunting): picked mon.noname-c con 0x7f934c0a3e20 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient(hunting): picked mon.noname-b con 0x7f934c13d1b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934c09b2f0 con 0x7f934c06de70 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9351eb4640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f934c06de70 legacy=0x7f934c139850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934c09b8b0 con 0x7f934c13d1b0 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934c13cce0 con 0x7f934c0a3e20 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.660+0000 7f9354118640 10 monclient(hunting): authenticate will time out at 951.285955s 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93516b3640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f934c0a3e20 0x7f934c139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56110/0 (socket says 172.21.15.148:56110) 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93516b3640 1 -- 172.21.15.148:0/4084254045 learned_addr learned my addr 172.21.15.148:0/4084254045 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:08:50.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 662398410 0 0) 0x7f934c13cce0 con 0x7f934c0a3e20 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): my global_id is 5006 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9330003520 con 0x7f934c0a3e20 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4000762774 0 0) 0x7f934c09b8b0 con 0x7f934c13d1b0 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): my global_id is 4969 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f934c13cce0 con 0x7f934c13d1b0 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 222766743 0 0) 0x7f9330003520 con 0x7f934c0a3e20 2024-09-16T04:08:50.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f934c09b8b0 con 0x7f934c0a3e20 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f933c0015d0 con 0x7f934c0a3e20 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): dump: 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.678 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4272703677 0 0) 0x7f934c09b8b0 con 0x7f934c0a3e20 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.171:6789/0 conn(0x7f934c13d1b0 legacy=0x7f934c13a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.148:6789/0 conn(0x7f934c06de70 legacy=0x7f934c139850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 monclient: found mon.c 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934c0a0130 con 0x7f934c0a3e20 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 5 monclient: authenticate success, global_id 5006 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 10 monclient: _renew_subs 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f934c09b010 con 0x7f934c0a3e20 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 10 monclient: _renew_subs 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f934c13c880 con 0x7f934c0a3e20 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f933c004cc0 con 0x7f934c0a3e20 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f933c005010 con 0x7f934c0a3e20 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f9343fff640 4 set_mon_vals no callback set 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.661+0000 7f93427fc640 10 monclient: dump: 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f9354118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f93340065c0 con 0x7f9334002860 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f933c0051b0 con 0x7f934c0a3e20 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(62..62 src has 1..62) ==== 4322+0+0 (unknown 1588246179 0 0) 0x7f933c047050 con 0x7f934c0a3e20 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951132281 0 0) 0x7f93340065c0 con 0x7f9334002860 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 10 monclient(hunting): my global_id is 4972 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f934c09b8b0 con 0x7f9334002860 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.662+0000 7f93526b5640 10 monclient: get_auth_request con 0x7f9330038ef0 auth_method 0 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 406730011 0 0) 0x7f934c09b8b0 con 0x7f9334002860 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93340065c0 con 0x7f9334002860 2024-09-16T04:08:50.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9348001740 con 0x7f9334002860 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 10 monclient: dump: 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.683 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3682962910 0 0) 0x7f93340065c0 con 0x7f9334002860 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f933400a150 con 0x7f9334002860 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9348001aa0 con 0x7f9334002860 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.171:6789/0 conn(0x7f9334002860 legacy=0x7f9334004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.663+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.171:6789/0 conn(0x7f9334002860 legacy=0x7f9334004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:50.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.676+0000 7fbdc6531640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.676+0000 7fbdc6531640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.676+0000 7fbdc6531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.676+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) -- 0x7fbda0002210 con 0x7fbdc0132e60 2024-09-16T04:08:50.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.676+0000 7fbdbcff9640 1 -- 172.21.15.171:0/2021922572 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v62) ==== 103+0+0 (unknown 4238528152 0 0) 0x7fbdb8019b90 con 0x7fbdc0132e60 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.677+0000 7fbdbcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.677+0000 7fbdbcff9640 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo does not exist 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.678+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 >> v1:172.21.15.171:6812/7327273 conn(0x7fbdac038f60 legacy=0x7fbdac03b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.678+0000 7fbdc6531640 10 monclient: shutdown 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.678+0000 7fbdc6531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.678+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 >> v1:172.21.15.171:6789/0 conn(0x7fbdc0132e60 legacy=0x7fbdc01d3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.679+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 shutdown_connections 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.679+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 >> 172.21.15.171:0/2021922572 conn(0x7fbdc00a52a0 msgr2=0x7fbdc01312d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.679+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 shutdown_connections 2024-09-16T04:08:50.687 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.679+0000 7fbdc6531640 1 -- 172.21.15.171:0/2021922572 wait complete. 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.669+0000 7f9354118640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.669+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9334009e40 con 0x7f9334009100 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570723707 0 0) 0x7f9334009e40 con 0x7f9334009100 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient(hunting): my global_id is 4975 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93340065c0 con 0x7f9334009100 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2700971086 0 0) 0x7f93340065c0 con 0x7f9334009100 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9334009e40 con 0x7f9334009100 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f93480037b0 con 0x7f9334009100 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient: dump: 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.689 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 847100624 0 0) 0x7f9334009e40 con 0x7f9334009100 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f933400c3d0 con 0x7f9334009100 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 766062730) 0x7f93480057c0 con 0x7f9334009100 2024-09-16T04:08:50.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.171:6789/0 conn(0x7f9334009100 legacy=0x7f9334009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.670+0000 7f93427fc640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.171:6789/0 conn(0x7f9334009100 legacy=0x7f9334009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:50.691 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 18, 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 154709, 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:50.692 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:50.693 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:50.694 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:08:50.695 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:08:50.696 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:08:50.697 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.698 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.699 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:50.700 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:08:50.701 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:08:50.702 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.703 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:08:50.704 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.705 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:08:50.706 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:08:50.707 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:08:50.708 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm bar 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.171:6812/7327273 conn(0x7f9330038ef0 legacy=0x7f933003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 10 monclient: shutdown 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 >> v1:172.21.15.148:6790/0 conn(0x7f934c0a3e20 legacy=0x7f934c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f93526b5640 1 -- 172.21.15.148:0/4084254045 reap_dead start 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f93526b5640 1 -- 172.21.15.148:0/4084254045 reap_dead start 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 shutdown_connections 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 >> 172.21.15.148:0/4084254045 conn(0x7f934c0a5a40 msgr2=0x7f934c0a6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 shutdown_connections 2024-09-16T04:08:50.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:08:50.671+0000 7f9354118640 1 -- 172.21.15.148:0/4084254045 wait complete. 2024-09-16T04:08:50.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:08:50.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:08:50.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.768+0000 7f71812c9640 10 monclient: get_monmap_and_config 2024-09-16T04:08:50.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.768+0000 7f71812c9640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.768+0000 7f71812c9640 10 monclient: monmap: 2024-09-16T04:08:50.770 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:50.769284+0000 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:50.769284+0000 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:50.771 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 1 Processor -- start 2024-09-16T04:08:50.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 1 -- start start 2024-09-16T04:08:50.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient: init 2024-09-16T04:08:50.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient(hunting): picked mon.noname-b con 0x7f717c136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient(hunting): picked mon.noname-a con 0x7f717c139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient(hunting): picked mon.noname-c con 0x7f717c13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717c0a1340 con 0x7f717c139b10 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717c137650 con 0x7f717c13d620 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717c0a0aa0 con 0x7f717c136b60 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f71812c9640 10 monclient(hunting): authenticate will time out at 906.274809s 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f7173fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f717c139b10 legacy=0x7f717c13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f717b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f717c13d620 0x7f717c13fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51096/0 (socket says 172.21.15.171:51096) 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.769+0000 7f717b7fe640 1 -- 172.21.15.171:0/2053921660 learned_addr learned my addr 172.21.15.171:0/2053921660 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236165953 0 0) 0x7f717c137650 con 0x7f717c13d620 2024-09-16T04:08:50.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): my global_id is 4978 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7168003520 con 0x7f717c13d620 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2687428579 0 0) 0x7f717c0a0aa0 con 0x7f717c136b60 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient(hunting): my global_id is 5009 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f717c137650 con 0x7f717c136b60 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3574252808 0 0) 0x7f7168003520 con 0x7f717c13d620 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 >> v1:172.21.15.148:6790/0 conn(0x7f717c136b60 legacy=0x7f717c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 >> v1:172.21.15.148:6789/0 conn(0x7f717c139b10 legacy=0x7f717c13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717c0a0550 con 0x7f717c13d620 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7174001990 con 0x7f717c13d620 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 5 monclient: authenticate success, global_id 4978 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:50.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: dump: 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7174001cb0 con 0x7f717c13d620 2024-09-16T04:08:50.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 1 -- 172.21.15.171:0/2053921660 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7174002080 con 0x7f717c13d620 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 4 set_mon_vals no callback set 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f717a7fc640 10 monclient: dump: 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.785 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 10 monclient: shutdown 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.770+0000 7f71812c9640 1 -- 172.21.15.171:0/2053921660 >> v1:172.21.15.171:6789/0 conn(0x7f717c13d620 legacy=0x7f717c13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 1 -- 172.21.15.171:0/2053921660 shutdown_connections 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 1 -- 172.21.15.171:0/2053921660 >> 172.21.15.171:0/2053921660 conn(0x7f717c0a98b0 msgr2=0x7f717c0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 1 -- 172.21.15.171:0/2053921660 shutdown_connections 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 1 -- 172.21.15.171:0/2053921660 wait complete. 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 10 monclient: build_initial_monmap 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 10 monclient: monmap: 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:50.787 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 1 Processor -- start 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 1 -- start start 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.771+0000 7f71812c9640 10 monclient: init 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient(hunting): picked mon.noname-b con 0x7f717c136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient(hunting): picked mon.noname-c con 0x7f717c139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient(hunting): picked mon.noname-a con 0x7f717c13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717c1204a0 con 0x7f717c13d620 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717c120780 con 0x7f717c136b60 2024-09-16T04:08:50.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f717b7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f717c13d620 legacy=0x7f717c1d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717c136700 con 0x7f717c139b10 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f717affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f717c136b60 0x7f717c1cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51108/0 (socket says 172.21.15.171:51108) 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f717affd640 1 -- 172.21.15.171:0/335967055 learned_addr learned my addr 172.21.15.171:0/335967055 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71812c9640 10 monclient(hunting): authenticate will time out at 906.277484s 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593519335 0 0) 0x7f717c120780 con 0x7f717c136b60 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): my global_id is 4981 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715c003520 con 0x7f717c136b60 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326468346 0 0) 0x7f717c136700 con 0x7f717c139b10 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): my global_id is 5012 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f717c120780 con 0x7f717c139b10 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 408241501 0 0) 0x7f715c003520 con 0x7f717c136b60 2024-09-16T04:08:50.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f717c136700 con 0x7f717c136b60 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f716c001970 con 0x7f717c136b60 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): dump: 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.791 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2209542641 0 0) 0x7f717c120780 con 0x7f717c139b10 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f715c003520 con 0x7f717c139b10 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f71640040f0 con 0x7f717c139b10 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.772+0000 7f71737fe640 10 monclient(hunting): dump: 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.792 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2184885096 0 0) 0x7f717c136700 con 0x7f717c136b60 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 >> v1:172.21.15.148:6790/0 conn(0x7f717c139b10 legacy=0x7f717c1d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:50.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 >> v1:172.21.15.148:6789/0 conn(0x7f717c13d620 legacy=0x7f717c1d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 monclient: found mon.b 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717c09ee90 con 0x7f717c136b60 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 5 monclient: authenticate success, global_id 4981 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 10 monclient: _renew_subs 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f717c09f2e0 con 0x7f717c136b60 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 10 monclient: _renew_subs 2024-09-16T04:08:50.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f717c09e200 con 0x7f717c136b60 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f716c001c50 con 0x7f717c136b60 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f716c003a60 con 0x7f717c136b60 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f7179ffb640 4 set_mon_vals no callback set 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 10 monclient: dump: 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:50.795 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.773+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f716c003c80 con 0x7f717c136b60 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.774+0000 7f71812c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.774+0000 7f71812c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.774+0000 7f71812c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.774+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7160005080 con 0x7f717c136b60 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.774+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(62..62 src has 1..62) ==== 4322+0+0 (unknown 1588246179 0 0) 0x7f716c046f90 con 0x7f717c136b60 2024-09-16T04:08:50.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.774+0000 7f7173fff640 10 monclient: get_auth_request con 0x7f715c038ed0 auth_method 0 2024-09-16T04:08:50.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.778+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f716c015cb0 con 0x7f717c136b60 2024-09-16T04:08:50.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.778+0000 7f71737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:50.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.778+0000 7f71737fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:50.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.946+0000 7f71812c9640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T04:08:50.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.946+0000 7f71812c9640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T04:08:50.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.946+0000 7f71812c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:50.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.946+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) -- 0x7f7160005370 con 0x7f717c136b60 2024-09-16T04:08:50.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.982+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 v63) ==== 80+0+0 (unknown 2386063302 0 0) 0x7f716c0155b0 con 0x7f717c136b60 2024-09-16T04:08:50.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.982+0000 7f71737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T04:08:50.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:50.982+0000 7f71737fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71812c9640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71812c9640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71812c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) -- 0x7f7160002210 con 0x7f717c136b60 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71737fe640 1 -- 172.21.15.171:0/335967055 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v63) ==== 103+0+0 (unknown 556395957 0 0) 0x7f716c019900 con 0x7f717c136b60 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71737fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.085+0000 7f71737fe640 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-16T04:08:51.087 INFO:tasks.workunit.client.0.smithi171.stderr:rule bar does not exist 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 >> v1:172.21.15.171:6812/7327273 conn(0x7f715c038ed0 legacy=0x7f715c03b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 10 monclient: shutdown 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 >> v1:172.21.15.171:6789/0 conn(0x7f717c136b60 legacy=0x7f717c1cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 shutdown_connections 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 >> 172.21.15.171:0/335967055 conn(0x7f717c0a98b0 msgr2=0x7f717c139200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:51.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 shutdown_connections 2024-09-16T04:08:51.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.087+0000 7f71812c9640 1 -- 172.21.15.171:0/335967055 wait complete. 2024-09-16T04:08:51.098 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create pinning_pool 1 2024-09-16T04:08:51.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.177+0000 7fef79653640 10 monclient: get_monmap_and_config 2024-09-16T04:08:51.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.177+0000 7fef79653640 10 monclient: build_initial_monmap 2024-09-16T04:08:51.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.177+0000 7fef79653640 10 monclient: monmap: 2024-09-16T04:08:51.179 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:51.178490+0000 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:51.178490+0000 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:51.180 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 1 Processor -- start 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 1 -- start start 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient: init 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient(hunting): picked mon.noname-b con 0x7fef74136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:51.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient(hunting): picked mon.noname-a con 0x7fef74139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient(hunting): picked mon.noname-c con 0x7fef7413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef740a1340 con 0x7fef74139b10 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef74137650 con 0x7fef7413d620 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef740a0aa0 con 0x7fef74136b60 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef79653640 10 monclient(hunting): authenticate will time out at 906.683904s 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.178+0000 7fef727fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fef74139b10 legacy=0x7fef7413bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef737fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fef7413d620 0x7fef7413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51118/0 (socket says 172.21.15.171:51118) 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef737fe640 1 -- 172.21.15.171:0/1826983137 learned_addr learned my addr 172.21.15.171:0/1826983137 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100206058 0 0) 0x7fef74137650 con 0x7fef7413d620 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:51.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): my global_id is 4984 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef5c003520 con 0x7fef7413d620 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305212226 0 0) 0x7fef740a0aa0 con 0x7fef74136b60 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient(hunting): my global_id is 5015 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef74137650 con 0x7fef74136b60 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3458907520 0 0) 0x7fef5c003520 con 0x7fef7413d620 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 >> v1:172.21.15.148:6790/0 conn(0x7fef74136b60 legacy=0x7fef74136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 >> v1:172.21.15.148:6789/0 conn(0x7fef74139b10 legacy=0x7fef7413bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:51.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef740a0550 con 0x7fef7413d620 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef6c001990 con 0x7fef7413d620 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef79653640 5 monclient: authenticate success, global_id 4984 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef79653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:51.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: dump: 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef79653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef6c001cb0 con 0x7fef7413d620 2024-09-16T04:08:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 1 -- 172.21.15.171:0/1826983137 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef6c002080 con 0x7fef7413d620 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef71ffb640 10 monclient: dump: 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:51.192 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef79653640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.179+0000 7fef79653640 4 set_mon_vals no callback set 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 10 monclient: shutdown 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 1 -- 172.21.15.171:0/1826983137 >> v1:172.21.15.171:6789/0 conn(0x7fef7413d620 legacy=0x7fef7413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 1 -- 172.21.15.171:0/1826983137 shutdown_connections 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 1 -- 172.21.15.171:0/1826983137 >> 172.21.15.171:0/1826983137 conn(0x7fef740a98b0 msgr2=0x7fef740a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 1 -- 172.21.15.171:0/1826983137 shutdown_connections 2024-09-16T04:08:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 1 -- 172.21.15.171:0/1826983137 wait complete. 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 10 monclient: build_initial_monmap 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.180+0000 7fef79653640 10 monclient: monmap: 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 1 Processor -- start 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 1 -- start start 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient: init 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient(hunting): picked mon.noname-c con 0x7fef74136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient(hunting): picked mon.noname-b con 0x7fef74139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient(hunting): picked mon.noname-a con 0x7fef7413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef741204a0 con 0x7fef7413d620 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef74120780 con 0x7fef74139b10 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef74136700 con 0x7fef74136b60 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef79653640 10 monclient(hunting): authenticate will time out at 906.686650s 2024-09-16T04:08:51.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef737fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fef7413d620 legacy=0x7fef741d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef727fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fef74139b10 0x7fef741d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51134/0 (socket says 172.21.15.171:51134) 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef727fc640 1 -- 172.21.15.171:0/3664895872 learned_addr learned my addr 172.21.15.171:0/3664895872 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1921525226 0 0) 0x7fef74120780 con 0x7fef74139b10 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): my global_id is 4987 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef60003520 con 0x7fef74139b10 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740927087 0 0) 0x7fef74136700 con 0x7fef74136b60 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 10 monclient(hunting): my global_id is 5018 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.181+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef74120780 con 0x7fef74136b60 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 550204989 0 0) 0x7fef60003520 con 0x7fef74139b10 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef74136700 con 0x7fef74139b10 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef680041d0 con 0x7fef74139b10 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): dump: 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 233751171 0 0) 0x7fef74120780 con 0x7fef74136b60 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef60003520 con 0x7fef74136b60 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef640018b0 con 0x7fef74136b60 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient(hunting): dump: 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2054449918 0 0) 0x7fef74136700 con 0x7fef74139b10 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 >> v1:172.21.15.148:6790/0 conn(0x7fef74136b60 legacy=0x7fef741cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 >> v1:172.21.15.148:6789/0 conn(0x7fef7413d620 legacy=0x7fef741d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 monclient: found mon.b 2024-09-16T04:08:51.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7409ee90 con 0x7fef74139b10 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 5 monclient: authenticate success, global_id 4987 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 10 monclient: _renew_subs 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef7409f2e0 con 0x7fef74139b10 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 10 monclient: _renew_subs 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef7409e200 con 0x7fef74139b10 2024-09-16T04:08:51.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef68004480 con 0x7fef74139b10 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef68004850 con 0x7fef74139b10 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef717fa640 4 set_mon_vals no callback set 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.182+0000 7fef5b7fe640 10 monclient: dump: 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:51.201 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fef68004ab0 con 0x7fef74139b10 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(63..63 src has 1..63) ==== 4267+0+0 (unknown 2791903777 0 0) 0x7fef68046fc0 con 0x7fef74139b10 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef72ffd640 10 monclient: get_auth_request con 0x7fef60038ec0 auth_method 0 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef79653640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef79653640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef79653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.183+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef54005080 con 0x7fef74139b10 2024-09-16T04:08:51.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.188+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fef68015d00 con 0x7fef74139b10 2024-09-16T04:08:51.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.188+0000 7fef5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:51.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.188+0000 7fef5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:51.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.354+0000 7fef79653640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T04:08:51.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.355+0000 7fef79653640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T04:08:51.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.355+0000 7fef79653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:51.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:51.355+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) -- 0x7fef54005b80 con 0x7fef74139b10 2024-09-16T04:08:52.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.192+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v64) ==== 127+0+0 (unknown 1349722679 0 0) 0x7fef68015600 con 0x7fef74139b10 2024-09-16T04:08:52.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.192+0000 7fef5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T04:08:52.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.192+0000 7fef5b7fe640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef79653640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef79653640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef79653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) -- 0x7fef54005760 con 0x7fef74139b10 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef5b7fe640 1 -- 172.21.15.171:0/3664895872 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v64) ==== 134+0+0 (unknown 3067261128 0 0) 0x7fef68019950 con 0x7fef74139b10 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef5b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.294+0000 7fef5b7fe640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-16T04:08:52.296 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'pinning_pool' already exists 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 >> v1:172.21.15.171:6812/7327273 conn(0x7fef60038ec0 legacy=0x7fef6003b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 10 monclient: shutdown 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 >> v1:172.21.15.171:6789/0 conn(0x7fef74139b10 legacy=0x7fef741d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 shutdown_connections 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 >> 172.21.15.171:0/3664895872 conn(0x7fef740a98b0 msgr2=0x7fef74139200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 shutdown_connections 2024-09-16T04:08:52.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.296+0000 7fef79653640 1 -- 172.21.15.171:0/3664895872 wait complete. 2024-09-16T04:08:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-16T04:08:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-16T04:08:52.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.386+0000 7f6833d6d640 10 monclient: get_monmap_and_config 2024-09-16T04:08:52.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.386+0000 7f6833d6d640 10 monclient: build_initial_monmap 2024-09-16T04:08:52.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.386+0000 7f6833d6d640 10 monclient: monmap: 2024-09-16T04:08:52.388 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:52.388 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.388 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:52.387210+0000 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:52.387210+0000 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:52.389 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.386+0000 7f6833d6d640 1 Processor -- start 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 1 -- start start 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient: init 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f682c0685d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f682c09bfc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f682c136e10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c099c00 con 0x7f682c09bfc0 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c1373c0 con 0x7f682c136e10 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c09a150 con 0x7f682c0685d0 2024-09-16T04:08:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6833d6d640 10 monclient(hunting): authenticate will time out at 907.892778s 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f683230a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f682c136e10 0x7f682c13f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51138/0 (socket says 172.21.15.171:51138) 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f6831308640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f682c09bfc0 legacy=0x7f682c13bc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.387+0000 7f683230a640 1 -- 172.21.15.171:0/2643405076 learned_addr learned my addr 172.21.15.171:0/2643405076 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254620335 0 0) 0x7f682c1373c0 con 0x7f682c136e10 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient(hunting): my global_id is 4990 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6818003520 con 0x7f682c136e10 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3195776549 0 0) 0x7f6818003520 con 0x7f682c136e10 2024-09-16T04:08:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 >> v1:172.21.15.148:6790/0 conn(0x7f682c0685d0 legacy=0x7f682c09b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 >> v1:172.21.15.148:6789/0 conn(0x7f682c09bfc0 legacy=0x7f682c13bc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: _finish_hunting 0 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 monclient: found mon.noname-c 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c143aa0 con 0x7f682c136e10 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: _finish_auth 0 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: _check_auth_tickets 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6824001990 con 0x7f682c136e10 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 5 monclient: authenticate success, global_id 4990 2024-09-16T04:08:52.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: dump: 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.400 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6824001cf0 con 0x7f682c136e10 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 1 -- 172.21.15.171:0/2643405076 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6824004c20 con 0x7f682c136e10 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 4 set_mon_vals no callback set 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:52.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6830b07640 10 monclient: dump: 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 10 monclient: shutdown 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:52.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 1 -- 172.21.15.171:0/2643405076 >> v1:172.21.15.171:6789/0 conn(0x7f682c136e10 legacy=0x7f682c13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 1 -- 172.21.15.171:0/2643405076 shutdown_connections 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.388+0000 7f6833d6d640 1 -- 172.21.15.171:0/2643405076 >> 172.21.15.171:0/2643405076 conn(0x7f682c0a1e90 msgr2=0x7f682c0a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 1 -- 172.21.15.171:0/2643405076 shutdown_connections 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 1 -- 172.21.15.171:0/2643405076 wait complete. 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 10 monclient: build_initial_monmap 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 10 monclient: monmap: 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.403 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 1 Processor -- start 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 1 -- start start 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.389+0000 7f6833d6d640 10 monclient: init 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f682c0685d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f682c09bfc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:52.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f682c136e10 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c13dd70 con 0x7f682c09bfc0 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c0a3830 con 0x7f682c0685d0 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c0a3300 con 0x7f682c136e10 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6833d6d640 10 monclient(hunting): authenticate will time out at 907.895324s 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6831b09640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f682c0685d0 0x7f682c13a050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51152/0 (socket says 172.21.15.171:51152) 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6831b09640 1 -- 172.21.15.171:0/2950970777 learned_addr learned my addr 172.21.15.171:0/2950970777 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f6831308640 1 -- 172.21.15.171:0/2950970777 >> v1:172.21.15.148:6789/0 conn(0x7f682c09bfc0 legacy=0x7f682c13a760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 799328041 0 0) 0x7f682c0a3830 con 0x7f682c0685d0 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): my global_id is 4993 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6810003520 con 0x7f682c0685d0 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885331389 0 0) 0x7f682c0a3300 con 0x7f682c136e10 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): my global_id is 5024 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f682c0a3830 con 0x7f682c136e10 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3546017321 0 0) 0x7f6810003520 con 0x7f682c0685d0 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f682c0a3300 con 0x7f682c0685d0 2024-09-16T04:08:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f68200018d0 con 0x7f682c0685d0 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): dump: 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.407 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3326485373 0 0) 0x7f682c0a3300 con 0x7f682c0685d0 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 >> v1:172.21.15.148:6790/0 conn(0x7f682c136e10 legacy=0x7f682c1d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 >> v1:172.21.15.148:6789/0 conn(0x7f682c09bfc0 legacy=0x7f682c13a760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 monclient: found mon.b 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c13d7b0 con 0x7f682c0685d0 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.390+0000 7f681e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 5 monclient: authenticate success, global_id 4993 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 monclient: _renew_subs 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f682c13da90 con 0x7f682c0685d0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 monclient: _renew_subs 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f682c13b050 con 0x7f682c0685d0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6820003610 con 0x7f682c0685d0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6820003960 con 0x7f682c0685d0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 10 monclient: dump: 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681ffff640 4 set_mon_vals no callback set 2024-09-16T04:08:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6820003b00 con 0x7f682c0685d0 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.391+0000 7f6833d6d640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682c0008d0 con 0x7f682c0685d0 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.392+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(64..64 src has 1..64) ==== 4618+0+0 (unknown 2913982988 0 0) 0x7f6820047180 con 0x7f682c0685d0 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.392+0000 7f6831308640 10 monclient: get_auth_request con 0x7f6810038fb0 auth_method 0 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.396+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6820019870 con 0x7f682c0685d0 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.396+0000 7f681e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.396+0000 7f681e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f6833d6d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f6833d6d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f6833d6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f6833d6d640 1 -- 172.21.15.171:0/2950970777 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) -- 0x7f682c1d5120 con 0x7f682c0685d0 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f681e7fc640 1 -- 172.21.15.171:0/2950970777 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v64) ==== 132+0+0 (unknown 1029812853 0 0) 0x7f682001e130 con 0x7f682c0685d0 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f681e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.567+0000 7f681e7fc640 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-16T04:08:52.570 INFO:tasks.workunit.client.0.smithi171.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-16T04:08:52.574 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:52.574 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-16T04:08:52.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.653+0000 7f97c6e2e640 10 monclient: get_monmap_and_config 2024-09-16T04:08:52.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.653+0000 7f97c6e2e640 10 monclient: build_initial_monmap 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.653+0000 7f97c6e2e640 10 monclient: monmap: 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:52.654680+0000 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:52.654680+0000 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:52.656 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.654+0000 7f97c6e2e640 1 Processor -- start 2024-09-16T04:08:52.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.654+0000 7f97c6e2e640 1 -- start start 2024-09-16T04:08:52.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.654+0000 7f97c6e2e640 10 monclient: init 2024-09-16T04:08:52.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f97c009b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f97c0136da0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f97c013d320 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c0073a20 con 0x7f97c009b920 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c0099920 con 0x7f97c013d320 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c0099c00 con 0x7f97c0136da0 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient(hunting): authenticate will time out at 908.160241s 2024-09-16T04:08:52.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c4bca640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f97c009b920 legacy=0x7f97c009bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c53cb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f97c013d320 0x7f97c013f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51156/0 (socket says 172.21.15.171:51156) 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c53cb640 1 -- 172.21.15.171:0/460141519 learned_addr learned my addr 172.21.15.171:0/460141519 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 976196236 0 0) 0x7f97c0099920 con 0x7f97c013d320 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): my global_id is 4996 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97a8003520 con 0x7f97c013d320 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690016463 0 0) 0x7f97c0099c00 con 0x7f97c0136da0 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient(hunting): my global_id is 5027 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c0099920 con 0x7f97c0136da0 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3570475429 0 0) 0x7f97a8003520 con 0x7f97c013d320 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 >> v1:172.21.15.148:6790/0 conn(0x7f97c0136da0 legacy=0x7f97c013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 >> v1:172.21.15.148:6789/0 conn(0x7f97c009b920 legacy=0x7f97c009bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c0099ee0 con 0x7f97c013d320 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:52.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97b8001990 con 0x7f97c013d320 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 5 monclient: authenticate success, global_id 4996 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: dump: 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.667 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97b8001c40 con 0x7f97c013d320 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 1 -- 172.21.15.171:0/460141519 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97b8002010 con 0x7f97c013d320 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97c6e2e640 4 set_mon_vals no callback set 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.655+0000 7f97bf7fe640 10 monclient: dump: 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.668 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 10 monclient: shutdown 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 1 -- 172.21.15.171:0/460141519 >> v1:172.21.15.171:6789/0 conn(0x7f97c013d320 legacy=0x7f97c013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 1 -- 172.21.15.171:0/460141519 shutdown_connections 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 1 -- 172.21.15.171:0/460141519 >> 172.21.15.171:0/460141519 conn(0x7f97c00a2150 msgr2=0x7f97c00a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:52.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 1 -- 172.21.15.171:0/460141519 shutdown_connections 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 1 -- 172.21.15.171:0/460141519 wait complete. 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 10 monclient: build_initial_monmap 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.656+0000 7f97c6e2e640 10 monclient: monmap: 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:52.670 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 1 Processor -- start 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 1 -- start start 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient: init 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f97c009b920 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f97c0136da0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f97c013d320 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c012b7b0 con 0x7f97c013d320 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c012ba90 con 0x7f97c009b920 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c012a530 con 0x7f97c0136da0 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.657+0000 7f97c6e2e640 10 monclient(hunting): authenticate will time out at 908.162975s 2024-09-16T04:08:52.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c4bca640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f97c009b920 0x7f97c00982e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51166/0 (socket says 172.21.15.171:51166) 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c4bca640 1 -- 172.21.15.171:0/2449919129 learned_addr learned my addr 172.21.15.171:0/2449919129 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c53cb640 1 -- 172.21.15.171:0/2449919129 >> v1:172.21.15.148:6789/0 conn(0x7f97c013d320 legacy=0x7f97c01d4090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244638728 0 0) 0x7f97c012ba90 con 0x7f97c009b920 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): my global_id is 4999 2024-09-16T04:08:52.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97ac003520 con 0x7f97c009b920 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087933719 0 0) 0x7f97c012a530 con 0x7f97c0136da0 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): my global_id is 5030 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97c012ba90 con 0x7f97c0136da0 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 310503491 0 0) 0x7f97ac003520 con 0x7f97c009b920 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97c012a530 con 0x7f97c009b920 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97b0004100 con 0x7f97c009b920 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:52.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): dump: 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 390341395 0 0) 0x7f97c012a530 con 0x7f97c009b920 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 >> v1:172.21.15.148:6790/0 conn(0x7f97c0136da0 legacy=0x7f97c01d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 >> v1:172.21.15.148:6789/0 conn(0x7f97c013d320 legacy=0x7f97c01d4090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 monclient: found mon.b 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c013c3f0 con 0x7f97c009b920 2024-09-16T04:08:52.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:08:52.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:08:52.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:52.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 5 monclient: authenticate success, global_id 4999 2024-09-16T04:08:52.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 10 monclient: _renew_subs 2024-09-16T04:08:52.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97c013b8b0 con 0x7f97c009b920 2024-09-16T04:08:52.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 10 monclient: _renew_subs 2024-09-16T04:08:52.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97c00989f0 con 0x7f97c009b920 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97b0004360 con 0x7f97c009b920 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97b0004730 con 0x7f97c009b920 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97beffd640 4 set_mon_vals no callback set 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.658+0000 7f97bd7fa640 10 monclient: dump: 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:52.678 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:52.679 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f97b00048d0 con 0x7f97c009b920 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97c6e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97c6e2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97c6e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a0005080 con 0x7f97c009b920 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(64..64 src has 1..64) ==== 4618+0+0 (unknown 2913982988 0 0) 0x7f97b0047030 con 0x7f97c009b920 2024-09-16T04:08:52.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.659+0000 7f97bffff640 10 monclient: get_auth_request con 0x7f97ac039050 auth_method 0 2024-09-16T04:08:52.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.664+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f97b0015b90 con 0x7f97c009b920 2024-09-16T04:08:52.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.664+0000 7f97bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:52.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.664+0000 7f97bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:52.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.829+0000 7f97c6e2e640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:52.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.829+0000 7f97c6e2e640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:52.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.830+0000 7f97c6e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:52.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:52.830+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f97a0005370 con 0x7f97c009b920 2024-09-16T04:08:53.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.196+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v65) ==== 172+0+0 (unknown 1366776915 0 0) 0x7f97b0015490 con 0x7f97c009b920 2024-09-16T04:08:53.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.196+0000 7f97bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:53.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.196+0000 7f97bd7fa640 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.297+0000 7f97c6e2e640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.297+0000 7f97c6e2e640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.297+0000 7f97c6e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.297+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f97a0005ab0 con 0x7f97c009b920 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.298+0000 7f97bd7fa640 1 -- 172.21.15.171:0/2449919129 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v65) ==== 179+0+0 (unknown 90434502 0 0) 0x7f97b001e040 con 0x7f97c009b920 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.298+0000 7f97bd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.298+0000 7f97bd7fa640 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-16T04:08:53.300 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'pinning_pool' does not exist 2024-09-16T04:08:53.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 >> v1:172.21.15.171:6812/7327273 conn(0x7f97ac039050 legacy=0x7f97ac03b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 10 monclient: shutdown 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 >> v1:172.21.15.171:6789/0 conn(0x7f97c009b920 legacy=0x7f97c00982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 shutdown_connections 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 >> 172.21.15.171:0/2449919129 conn(0x7f97c00a2150 msgr2=0x7f97c00a4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 shutdown_connections 2024-09-16T04:08:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.300+0000 7f97c6e2e640 1 -- 172.21.15.171:0/2449919129 wait complete. 2024-09-16T04:08:53.310 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-16T04:08:53.310 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:08:53.310 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket foo osd 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.389+0000 7fbbc142e640 10 monclient: get_monmap_and_config 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.389+0000 7fbbc142e640 10 monclient: build_initial_monmap 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.389+0000 7fbbc142e640 10 monclient: monmap: 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:53.389958+0000 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:53.389958+0000 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.391 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:53.392 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:53.392 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:53.392 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.389+0000 7fbbc142e640 1 Processor -- start 2024-09-16T04:08:53.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.389+0000 7fbbc142e640 1 -- start start 2024-09-16T04:08:53.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.389+0000 7fbbc142e640 10 monclient: init 2024-09-16T04:08:53.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient(hunting): picked mon.noname-c con 0x7fbbbc136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient(hunting): picked mon.noname-a con 0x7fbbbc139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient(hunting): picked mon.noname-b con 0x7fbbbc13d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc0a1340 con 0x7fbbbc139b10 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc137650 con 0x7fbbbc136b60 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc0a0aa0 con 0x7fbbbc13d620 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient(hunting): authenticate will time out at 908.895317s 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbbaffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbbbc136b60 0x7fbbbc136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51170/0 (socket says 172.21.15.171:51170) 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbbaffd640 1 -- 172.21.15.171:0/3692687916 learned_addr learned my addr 172.21.15.171:0/3692687916 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:53.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbba7fc640 1 -- 172.21.15.171:0/3692687916 >> v1:172.21.15.148:6789/0 conn(0x7fbbbc139b10 legacy=0x7fbbbc13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665222167 0 0) 0x7fbbbc137650 con 0x7fbbbc136b60 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient(hunting): my global_id is 5002 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba8003520 con 0x7fbbbc136b60 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2605369703 0 0) 0x7fbba8003520 con 0x7fbbbc136b60 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 >> v1:172.21.15.148:6790/0 conn(0x7fbbbc13d620 legacy=0x7fbbbc13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 >> v1:172.21.15.148:6789/0 conn(0x7fbbbc139b10 legacy=0x7fbbbc13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:53.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc0a0550 con 0x7fbbbc136b60 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 5 monclient: authenticate success, global_id 5002 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbac001990 con 0x7fbbbc136b60 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: dump: 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.402 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbac001cf0 con 0x7fbbbc136b60 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:53.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 1 -- 172.21.15.171:0/3692687916 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbac004c20 con 0x7fbbbc136b60 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbc142e640 4 set_mon_vals no callback set 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.390+0000 7fbbb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbb9ffb640 10 monclient: dump: 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.404 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 10 monclient: shutdown 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 1 -- 172.21.15.171:0/3692687916 >> v1:172.21.15.171:6789/0 conn(0x7fbbbc136b60 legacy=0x7fbbbc136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 1 -- 172.21.15.171:0/3692687916 shutdown_connections 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 1 -- 172.21.15.171:0/3692687916 >> 172.21.15.171:0/3692687916 conn(0x7fbbbc0a98b0 msgr2=0x7fbbbc0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:53.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 1 -- 172.21.15.171:0/3692687916 shutdown_connections 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.391+0000 7fbbc142e640 1 -- 172.21.15.171:0/3692687916 wait complete. 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient: build_initial_monmap 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient: monmap: 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 1 Processor -- start 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 1 -- start start 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient: init 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient(hunting): picked mon.noname-a con 0x7fbbbc136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.392+0000 7fbbc142e640 10 monclient(hunting): picked mon.noname-b con 0x7fbbbc139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient(hunting): picked mon.noname-c con 0x7fbbbc13d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc11fab0 con 0x7fbbbc136b60 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc0a0240 con 0x7fbbbc139b10 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc09efb0 con 0x7fbbbc13d620 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient(hunting): authenticate will time out at 908.898034s 2024-09-16T04:08:53.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbbaffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbbbc136b60 legacy=0x7fbbbc1cbb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbba7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbbbc139b10 0x7fbbbc1d0ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51178/0 (socket says 172.21.15.171:51178) 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbba7fc640 1 -- 172.21.15.171:0/1376777187 learned_addr learned my addr 172.21.15.171:0/1376777187 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788019246 0 0) 0x7fbbbc0a0240 con 0x7fbbbc139b10 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): my global_id is 5005 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb9c003550 con 0x7fbbbc139b10 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079956583 0 0) 0x7fbbbc09efb0 con 0x7fbbbc13d620 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): my global_id is 5036 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbbc0a0240 con 0x7fbbbc13d620 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1276622788 0 0) 0x7fbb9c003550 con 0x7fbbbc139b10 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbbc09efb0 con 0x7fbbbc139b10 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbba4004180 con 0x7fbbbc139b10 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): dump: 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 825020386 0 0) 0x7fbbbc09efb0 con 0x7fbbbc139b10 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 >> v1:172.21.15.148:6790/0 conn(0x7fbbbc13d620 legacy=0x7fbbbc1d42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 >> v1:172.21.15.148:6789/0 conn(0x7fbbbc136b60 legacy=0x7fbbbc1cbb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 monclient: found mon.b 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc120b90 con 0x7fbbbc139b10 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 5 monclient: authenticate success, global_id 5005 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient: _renew_subs 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc120e70 con 0x7fbbbc139b10 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient: _renew_subs 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbc142e640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbbc11f8c0 con 0x7fbbbc139b10 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbba4004430 con 0x7fbbbc139b10 2024-09-16T04:08:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbba4004800 con 0x7fbbbc139b10 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb97fa640 4 set_mon_vals no callback set 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.393+0000 7fbbb37fe640 10 monclient: dump: 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbba40049a0 con 0x7fbbbc139b10 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbc142e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbc142e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbc142e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbc142e640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba0005080 con 0x7fbbbc139b10 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(65..65 src has 1..65) ==== 4267+0+0 (unknown 1207397900 0 0) 0x7fbba4047020 con 0x7fbbbc139b10 2024-09-16T04:08:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.394+0000 7fbbbaffd640 10 monclient: get_auth_request con 0x7fbb9c039010 auth_method 0 2024-09-16T04:08:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.399+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbba4015bc0 con 0x7fbbbc139b10 2024-09-16T04:08:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.399+0000 7fbbb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.399+0000 7fbbb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.567+0000 7fbbc142e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.567+0000 7fbbc142e640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.567+0000 7fbbc142e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.567+0000 7fbbc142e640 1 -- 172.21.15.171:0/1376777187 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) -- 0x7fbba0005b80 con 0x7fbbbc139b10 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.568+0000 7fbbb37fe640 1 -- 172.21.15.171:0/1376777187 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v65) ==== 136+0+0 (unknown 553729052 0 0) 0x7fbba4019870 con 0x7fbbbc139b10 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.568+0000 7fbbb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.568+0000 7fbbb37fe640 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-16T04:08:53.570 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-16T04:08:53.574 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:08:53.574 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket foo root 2024-09-16T04:08:53.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.652+0000 7f53b3fff640 10 monclient: get_monmap_and_config 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.652+0000 7f53b3fff640 10 monclient: build_initial_monmap 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.652+0000 7f53b3fff640 10 monclient: monmap: 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:53.652888+0000 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:53.652888+0000 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:53.654 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.652+0000 7f53b3fff640 1 Processor -- start 2024-09-16T04:08:53.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.652+0000 7f53b3fff640 1 -- start start 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.652+0000 7f53b3fff640 10 monclient: init 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f53b40a3e30 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a0e80 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f53b4136a50 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b406c1f0 con 0x7f53b4136a50 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b4136fe0 con 0x7f53b40a3e30 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b406cc60 con 0x7f53b40a0e80 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient(hunting): authenticate will time out at 909.158241s 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f53b4136a50 legacy=0x7f53b413efe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:53.663 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f53b40a3e30 0x7f53b40a0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51184/0 (socket says 172.21.15.171:51184) 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b2ffd640 1 -- 172.21.15.171:0/2509962868 learned_addr learned my addr 172.21.15.171:0/2509962868 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942083276 0 0) 0x7f53b4136fe0 con 0x7f53b40a3e30 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): my global_id is 5008 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f539c003520 con 0x7f53b40a3e30 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282508094 0 0) 0x7f53b406cc60 con 0x7f53b40a0e80 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.664 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient(hunting): my global_id is 5039 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53b4136fe0 con 0x7f53b40a0e80 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1864172228 0 0) 0x7f539c003520 con 0x7f53b40a3e30 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 >> v1:172.21.15.148:6790/0 conn(0x7f53b40a0e80 legacy=0x7f53b413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 >> v1:172.21.15.148:6789/0 conn(0x7f53b4136a50 legacy=0x7f53b413efe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b406d8e0 con 0x7f53b40a3e30 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:53.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 5 monclient: authenticate success, global_id 5008 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f53a8001610 con 0x7f53b40a3e30 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: dump: 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.666 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53a8001930 con 0x7f53b40a3e30 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b3fff640 4 set_mon_vals no callback set 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 1 -- 172.21.15.171:0/2509962868 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f53a8001d00 con 0x7f53b40a3e30 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.653+0000 7f53b1ffb640 10 monclient: dump: 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.667 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 10 monclient: shutdown 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 1 -- 172.21.15.171:0/2509962868 >> v1:172.21.15.171:6789/0 conn(0x7f53b40a3e30 legacy=0x7f53b40a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 1 -- 172.21.15.171:0/2509962868 shutdown_connections 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 1 -- 172.21.15.171:0/2509962868 >> 172.21.15.171:0/2509962868 conn(0x7f53b40a72e0 msgr2=0x7f53b40a36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:53.668 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 1 -- 172.21.15.171:0/2509962868 shutdown_connections 2024-09-16T04:08:53.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 1 -- 172.21.15.171:0/2509962868 wait complete. 2024-09-16T04:08:53.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 10 monclient: build_initial_monmap 2024-09-16T04:08:53.669 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.654+0000 7f53b3fff640 10 monclient: monmap: 2024-09-16T04:08:53.669 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:53.669 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 1 Processor -- start 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 1 -- start start 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient: init 2024-09-16T04:08:53.672 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f53b40a0e80 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a3e30 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f53b4136a50 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b4134720 con 0x7f53b4136a50 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b4135000 con 0x7f53b40a3e30 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b4072160 con 0x7f53b40a0e80 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b3fff640 10 monclient(hunting): authenticate will time out at 909.160923s 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.655+0000 7f53b27fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f53b40a3e30 0x7f53b41334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51192/0 (socket says 172.21.15.171:51192) 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b27fc640 1 -- 172.21.15.171:0/656901707 learned_addr learned my addr 172.21.15.171:0/656901707 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b37fe640 1 -- 172.21.15.171:0/656901707 >> v1:172.21.15.148:6789/0 conn(0x7f53b4136a50 legacy=0x7f53b41d3760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041029193 0 0) 0x7f53b4135000 con 0x7f53b40a3e30 2024-09-16T04:08:53.673 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): my global_id is 5011 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53a0003520 con 0x7f53b40a3e30 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 967306519 0 0) 0x7f53a0003520 con 0x7f53b40a3e30 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53b4135000 con 0x7f53b40a3e30 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f53a4001890 con 0x7f53b40a3e30 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): dump: 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.674 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310407395 0 0) 0x7f53b4072160 con 0x7f53b40a0e80 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient(hunting): my global_id is 5042 2024-09-16T04:08:53.675 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53a0003520 con 0x7f53b40a0e80 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3188218060 0 0) 0x7f53b4135000 con 0x7f53b40a3e30 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 >> v1:172.21.15.148:6790/0 conn(0x7f53b40a0e80 legacy=0x7f53b4132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 >> v1:172.21.15.148:6789/0 conn(0x7f53b4136a50 legacy=0x7f53b41d3760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 monclient: found mon.b 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b4135280 con 0x7f53b40a3e30 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 5 monclient: authenticate success, global_id 5011 2024-09-16T04:08:53.676 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 10 monclient: _renew_subs 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f53b4133c00 con 0x7f53b40a3e30 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 10 monclient: _renew_subs 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53b41d4e80 con 0x7f53b40a3e30 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f53a4001c80 con 0x7f53b40a3e30 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f53a4003ae0 con 0x7f53b40a3e30 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f53b17fa640 4 set_mon_vals no callback set 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.656+0000 7f539b7fe640 10 monclient: dump: 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:53.677 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f53a4003c80 con 0x7f53b40a3e30 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(65..65 src has 1..65) ==== 4267+0+0 (unknown 1207397900 0 0) 0x7f53a4046f40 con 0x7f53b40a3e30 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f53b2ffd640 10 monclient: get_auth_request con 0x7f53a0039070 auth_method 0 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f53b3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f53b3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:53.678 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.657+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5394005080 con 0x7f53b40a3e30 2024-09-16T04:08:53.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.662+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f53a4015ce0 con 0x7f53b40a3e30 2024-09-16T04:08:53.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.662+0000 7f539b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:53.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.662+0000 7f539b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:53.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.831+0000 7f53b3fff640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T04:08:53.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.831+0000 7f53b3fff640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T04:08:53.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.831+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:53.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:53.831+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) -- 0x7f5394005b80 con 0x7f53b40a3e30 2024-09-16T04:08:54.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.200+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v66) ==== 138+0+0 (unknown 3587260211 0 0) 0x7f53a40155e0 con 0x7f53b40a3e30 2024-09-16T04:08:54.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.200+0000 7f539b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T04:08:54.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.200+0000 7f539b7fe640 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f53b3fff640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f53b3fff640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f53b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) -- 0x7f5394005760 con 0x7f53b40a3e30 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f539b7fe640 1 -- 172.21.15.171:0/656901707 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v66) ==== 126+0+0 (unknown 1017218759 0 0) 0x7f53a401e190 con 0x7f53b40a3e30 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f539b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-16T04:08:54.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.304+0000 7f539b7fe640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T04:08:54.307 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'foo' already exists 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 >> v1:172.21.15.171:6812/7327273 conn(0x7f53a0039070 legacy=0x7f53a003b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 10 monclient: shutdown 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 >> v1:172.21.15.171:6789/0 conn(0x7f53b40a3e30 legacy=0x7f53b41334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 shutdown_connections 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 >> 172.21.15.171:0/656901707 conn(0x7f53b40a72e0 msgr2=0x7f53b40a2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:54.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 shutdown_connections 2024-09-16T04:08:54.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.306+0000 7f53b3fff640 1 -- 172.21.15.171:0/656901707 wait complete. 2024-09-16T04:08:54.317 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 2024-09-16T04:08:54.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.391+0000 7f5714e1e640 10 monclient: get_monmap_and_config 2024-09-16T04:08:54.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.391+0000 7f5714e1e640 10 monclient: build_initial_monmap 2024-09-16T04:08:54.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.391+0000 7f5714e1e640 10 monclient: monmap: 2024-09-16T04:08:54.393 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:54.392464+0000 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:54.392464+0000 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:54.394 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 1 Processor -- start 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 1 -- start start 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient: init 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f57101367a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:54.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f571009aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5710137290 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571009c3b0 con 0x7f571009aa20 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571009b100 con 0x7f5710137290 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571009b3e0 con 0x7f57101367a0 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.392+0000 7f5714e1e640 10 monclient(hunting): authenticate will time out at 909.897918s 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570ed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5710137290 0x7f571013f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51204/0 (socket says 172.21.15.171:51204) 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570ed9d640 1 -- 172.21.15.171:0/3925295078 learned_addr learned my addr 172.21.15.171:0/3925295078 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570dd9b640 1 -- 172.21.15.171:0/3925295078 >> v1:172.21.15.148:6789/0 conn(0x7f571009aa20 legacy=0x7f571013bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077490277 0 0) 0x7f571009b100 con 0x7f5710137290 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:54.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient(hunting): my global_id is 5014 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f8003520 con 0x7f5710137290 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2391377819 0 0) 0x7f56f8003520 con 0x7f5710137290 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 >> v1:172.21.15.148:6790/0 conn(0x7f57101367a0 legacy=0x7f5710136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 >> v1:172.21.15.148:6789/0 conn(0x7f571009aa20 legacy=0x7f571013bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 monclient: found mon.noname-c 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571009b6c0 con 0x7f5710137290 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: _finish_auth 0 2024-09-16T04:08:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5708001990 con 0x7f5710137290 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f5714e1e640 5 monclient: authenticate success, global_id 5014 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f5714e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: dump: 2024-09-16T04:08:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:54.406 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:54.406 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.406 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f5714e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5708001c80 con 0x7f5710137290 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f5714e1e640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 1 -- 172.21.15.171:0/3925295078 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5708002090 con 0x7f5710137290 2024-09-16T04:08:54.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f5714e1e640 4 set_mon_vals no callback set 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.393+0000 7f570d59a640 10 monclient: dump: 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:54.408 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:54.409 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:54.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 10 monclient: shutdown 2024-09-16T04:08:54.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 1 -- 172.21.15.171:0/3925295078 >> v1:172.21.15.171:6789/0 conn(0x7f5710137290 legacy=0x7f571013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 1 -- 172.21.15.171:0/3925295078 shutdown_connections 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 1 -- 172.21.15.171:0/3925295078 >> 172.21.15.171:0/3925295078 conn(0x7f57100a28d0 msgr2=0x7f57100a2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 1 -- 172.21.15.171:0/3925295078 shutdown_connections 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 1 -- 172.21.15.171:0/3925295078 wait complete. 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 10 monclient: build_initial_monmap 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.394+0000 7f5714e1e640 10 monclient: monmap: 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:54.410 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 1 Processor -- start 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 1 -- start start 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient: init 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f571009aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:54.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f57101367a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5710137290 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57100a50a0 con 0x7f571009aa20 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57101cba00 con 0x7f57101367a0 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5710134eb0 con 0x7f5710137290 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f5714e1e640 10 monclient(hunting): authenticate will time out at 909.900671s 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f570e59c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f571009aa20 legacy=0x7f571013f3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f570dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f57101367a0 0x7f57101d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51212/0 (socket says 172.21.15.171:51212) 2024-09-16T04:08:54.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f570dd9b640 1 -- 172.21.15.171:0/3889328132 learned_addr learned my addr 172.21.15.171:0/3889328132 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500623071 0 0) 0x7f57101cba00 con 0x7f57101367a0 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f56feffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f56feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f56feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f56feffd640 10 monclient(hunting): my global_id is 5017 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.395+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f0003520 con 0x7f57101367a0 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214527954 0 0) 0x7f5710134eb0 con 0x7f5710137290 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): my global_id is 5048 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57101cba00 con 0x7f5710137290 2024-09-16T04:08:54.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1193590530 0 0) 0x7f56f0003520 con 0x7f57101367a0 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5710134eb0 con 0x7f57101367a0 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5704004180 con 0x7f57101367a0 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): dump: 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2798479684 0 0) 0x7f5710134eb0 con 0x7f57101367a0 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 >> v1:172.21.15.148:6790/0 conn(0x7f5710137290 legacy=0x7f57101d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 >> v1:172.21.15.148:6789/0 conn(0x7f571009aa20 legacy=0x7f571013f3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 monclient: found mon.b 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5710135300 con 0x7f57101367a0 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 5 monclient: authenticate success, global_id 5017 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 10 monclient: _renew_subs 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f571013fad0 con 0x7f57101367a0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 10 monclient: _renew_subs 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57101cb160 con 0x7f57101367a0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f57040044a0 con 0x7f57101367a0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5704004870 con 0x7f57101367a0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f570cd99640 4 set_mon_vals no callback set 2024-09-16T04:08:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.396+0000 7f56feffd640 10 monclient: dump: 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:54.417 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5704004ad0 con 0x7f57101367a0 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f5714e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f5714e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f5714e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f4005080 con 0x7f57101367a0 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(66..66 src has 1..66) ==== 4294+0+0 (unknown 988967192 0 0) 0x7f5704047000 con 0x7f57101367a0 2024-09-16T04:08:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.397+0000 7f570e59c640 10 monclient: get_auth_request con 0x7f56f0038ef0 auth_method 0 2024-09-16T04:08:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.402+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5704015d80 con 0x7f57101367a0 2024-09-16T04:08:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.402+0000 7f56feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.402+0000 7f56feffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:54.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.578+0000 7f5714e1e640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:08:54.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.578+0000 7f5714e1e640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T04:08:54.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.578+0000 7f5714e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:54.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:54.578+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f56f4005370 con 0x7f57101367a0 2024-09-16T04:08:55.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.202+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v67) ==== 58+0+2 (unknown 832193543 0 1113811849) 0x7f5704015680 con 0x7f57101367a0 2024-09-16T04:08:55.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.202+0000 7f56feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T04:08:55.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.202+0000 7f56feffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:55.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.313+0000 7f5714e1e640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:08:55.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.313+0000 7f5714e1e640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T04:08:55.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.313+0000 7f5714e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:55.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:55.313+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f56f4000bf0 con 0x7f57101367a0 2024-09-16T04:08:56.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.210+0000 7f56feffd640 1 -- 172.21.15.171:0/3889328132 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v68) ==== 58+0+2 (unknown 904382424 0 1371665406) 0x7f570401e230 con 0x7f57101367a0 2024-09-16T04:08:56.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.210+0000 7f56feffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T04:08:56.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.210+0000 7f56feffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.213+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 >> v1:172.21.15.171:6812/7327273 conn(0x7f56f0038ef0 legacy=0x7f56f003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.213+0000 7f5714e1e640 10 monclient: shutdown 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.213+0000 7f5714e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.213+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 >> v1:172.21.15.171:6789/0 conn(0x7f57101367a0 legacy=0x7f57101d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.214+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 shutdown_connections 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.214+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 >> 172.21.15.171:0/3889328132 conn(0x7f57100a28d0 msgr2=0x7f571013ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.214+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 shutdown_connections 2024-09-16T04:08:56.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.214+0000 7f5714e1e640 1 -- 172.21.15.171:0/3889328132 wait complete. 2024-09-16T04:08:56.226 INFO:tasks.workunit.client.0.smithi171.stderr:+ o1=7 2024-09-16T04:08:56.226 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 2024-09-16T04:08:56.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.303+0000 7f1694bfd640 10 monclient: get_monmap_and_config 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.303+0000 7f1694bfd640 10 monclient: build_initial_monmap 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.303+0000 7f1694bfd640 10 monclient: monmap: 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:56.303891+0000 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:56.303891+0000 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:56.305 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:56.306 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:56.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.303+0000 7f1694bfd640 1 Processor -- start 2024-09-16T04:08:56.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.303+0000 7f1694bfd640 1 -- start start 2024-09-16T04:08:56.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.303+0000 7f1694bfd640 10 monclient: init 2024-09-16T04:08:56.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient(hunting): picked mon.noname-b con 0x7f1690136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient(hunting): picked mon.noname-c con 0x7f16900a1cf0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient(hunting): picked mon.noname-a con 0x7f169013d1e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16900a0550 con 0x7f169013d1e0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16900a0a70 con 0x7f16900a1cf0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16900a0d50 con 0x7f1690136d60 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f1694bfd640 10 monclient(hunting): authenticate will time out at 911.809425s 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f16900a1cf0 0x7f169013bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51226/0 (socket says 172.21.15.171:51226) 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168dd9b640 1 -- 172.21.15.171:0/3022700902 learned_addr learned my addr 172.21.15.171:0/3022700902 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168ed9d640 1 -- 172.21.15.171:0/3022700902 >> v1:172.21.15.148:6789/0 conn(0x7f169013d1e0 legacy=0x7f169013f5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813332928 0 0) 0x7f16900a0a70 con 0x7f16900a1cf0 2024-09-16T04:08:56.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): my global_id is 5020 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f167c003520 con 0x7f16900a1cf0 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708176573 0 0) 0x7f16900a0d50 con 0x7f1690136d60 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 10 monclient(hunting): my global_id is 5051 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16900a0a70 con 0x7f1690136d60 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1795194625 0 0) 0x7f167c003520 con 0x7f16900a1cf0 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.304+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 >> v1:172.21.15.148:6790/0 conn(0x7f1690136d60 legacy=0x7f1690137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 >> v1:172.21.15.148:6789/0 conn(0x7f169013d1e0 legacy=0x7f169013f5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 1 monclient: found mon.noname-c 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16900a4a60 con 0x7f16900a1cf0 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1678001610 con 0x7f16900a1cf0 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 5 monclient: authenticate success, global_id 5020 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: dump: 2024-09-16T04:08:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1678001870 con 0x7f16900a1cf0 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 1 -- 172.21.15.171:0/3022700902 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1678001bc0 con 0x7f16900a1cf0 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 4 set_mon_vals no callback set 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f168d59a640 10 monclient: dump: 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:56.317 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 10 monclient: shutdown 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 1 -- 172.21.15.171:0/3022700902 >> v1:172.21.15.171:6789/0 conn(0x7f16900a1cf0 legacy=0x7f169013bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 1 -- 172.21.15.171:0/3022700902 shutdown_connections 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 1 -- 172.21.15.171:0/3022700902 >> 172.21.15.171:0/3022700902 conn(0x7f16900a98d0 msgr2=0x7f16900a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 1 -- 172.21.15.171:0/3022700902 shutdown_connections 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.305+0000 7f1694bfd640 1 -- 172.21.15.171:0/3022700902 wait complete. 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.306+0000 7f1694bfd640 10 monclient: build_initial_monmap 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.306+0000 7f1694bfd640 10 monclient: monmap: 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:56.318 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.306+0000 7f1694bfd640 1 Processor -- start 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.306+0000 7f1694bfd640 1 -- start start 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.306+0000 7f1694bfd640 10 monclient: init 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient(hunting): picked mon.noname-a con 0x7f16900a1cf0 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:56.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient(hunting): picked mon.noname-b con 0x7f1690136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient(hunting): picked mon.noname-c con 0x7f169013d1e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16901380d0 con 0x7f16900a1cf0 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1690139250 con 0x7f1690136d60 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169009eef0 con 0x7f169013d1e0 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f168e59c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f16900a1cf0 legacy=0x7f169009e0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient(hunting): authenticate will time out at 911.812348s 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f168dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1690136d60 0x7f169009e7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:51234/0 (socket says 172.21.15.171:51234) 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f168dd9b640 1 -- 172.21.15.171:0/4150025613 learned_addr learned my addr 172.21.15.171:0/4150025613 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3621057450 0 0) 0x7f1690139250 con 0x7f1690136d60 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): my global_id is 5023 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1670003520 con 0x7f1690136d60 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538989463 0 0) 0x7f169009eef0 con 0x7f169013d1e0 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): my global_id is 5054 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1690139250 con 0x7f169013d1e0 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2275987766 0 0) 0x7f1670003520 con 0x7f1690136d60 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f169009eef0 con 0x7f1690136d60 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16780015d0 con 0x7f1690136d60 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:56.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): dump: 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 203135860 0 0) 0x7f169009eef0 con 0x7f1690136d60 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 >> v1:172.21.15.148:6790/0 conn(0x7f169013d1e0 legacy=0x7f16901d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 >> v1:172.21.15.148:6789/0 conn(0x7f16900a1cf0 legacy=0x7f169009e0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 monclient: found mon.b 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16900a0240 con 0x7f1690136d60 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1686ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 5 monclient: authenticate success, global_id 5023 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient: _renew_subs 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.307+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f16901352c0 con 0x7f1690136d60 2024-09-16T04:08:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1694bfd640 10 monclient: _renew_subs 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1694bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1690138f70 con 0x7f1690136d60 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1678004a90 con 0x7f1690136d60 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1678004e60 con 0x7f1690136d60 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f168cd99640 4 set_mon_vals no callback set 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 10 monclient: dump: 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1678005000 con 0x7f1690136d60 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.308+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(68..68 src has 1..68) ==== 4524+0+0 (unknown 1544480042 0 0) 0x7f1678047130 con 0x7f1690136d60 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.309+0000 7f168e59c640 10 monclient: get_auth_request con 0x7f1670038bc0 auth_method 0 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.309+0000 7f1694bfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.309+0000 7f1694bfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.309+0000 7f1694bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.309+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1674005080 con 0x7f1690136d60 2024-09-16T04:08:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.313+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1678015c90 con 0x7f1690136d60 2024-09-16T04:08:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.313+0000 7f1686ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.313+0000 7f1686ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:56.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.488+0000 7f1694bfd640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:08:56.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.488+0000 7f1694bfd640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T04:08:56.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.488+0000 7f1694bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:56.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:56.488+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f1674005370 con 0x7f1690136d60 2024-09-16T04:08:57.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.215+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v69) ==== 58+0+2 (unknown 2157058348 0 3065575683) 0x7f167801e140 con 0x7f1690136d60 2024-09-16T04:08:57.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.215+0000 7f1686ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T04:08:57.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.215+0000 7f1686ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.331+0000 7f1694bfd640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:08:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.331+0000 7f1694bfd640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T04:08:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.331+0000 7f1694bfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:57.331+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f1674000bf0 con 0x7f1690136d60 2024-09-16T04:08:58.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.219+0000 7f1686ffd640 1 -- 172.21.15.171:0/4150025613 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v70) ==== 58+0+2 (unknown 1524632769 0 2769969524) 0x7f1678001830 con 0x7f1690136d60 2024-09-16T04:08:58.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.219+0000 7f1686ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T04:08:58.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.219+0000 7f1686ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:08:58.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.222+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 >> v1:172.21.15.171:6812/7327273 conn(0x7f1670038bc0 legacy=0x7f167003b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.222+0000 7f1694bfd640 10 monclient: shutdown 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.222+0000 7f1694bfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.222+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 >> v1:172.21.15.171:6789/0 conn(0x7f1690136d60 legacy=0x7f169009e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.223+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 shutdown_connections 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.223+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 >> 172.21.15.171:0/4150025613 conn(0x7f16900a98d0 msgr2=0x7f16900a6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.223+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 shutdown_connections 2024-09-16T04:08:58.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.223+0000 7f1694bfd640 1 -- 172.21.15.171:0/4150025613 wait complete. 2024-09-16T04:08:58.236 INFO:tasks.workunit.client.0.smithi171.stderr:+ o2=9 2024-09-16T04:08:58.237 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T04:08:58.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.323+0000 7fefa57bd640 10 monclient: get_monmap_and_config 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.323+0000 7fefa57bd640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.323+0000 7fefa57bd640 10 monclient: monmap: 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:58.323890+0000 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:58.323890+0000 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:58.325 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:58.326 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.323+0000 7fefa57bd640 1 Processor -- start 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.323+0000 7fefa57bd640 1 -- start start 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.323+0000 7fefa57bd640 10 monclient: init 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient(hunting): picked mon.noname-a con 0x7fefa0136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient(hunting): picked mon.noname-b con 0x7fefa0137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient(hunting): picked mon.noname-c con 0x7fefa01375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa009b2c0 con 0x7fefa0136d60 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa009b5a0 con 0x7fefa01375a0 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa00a6640 con 0x7fefa0137140 2024-09-16T04:08:58.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 10 monclient(hunting): authenticate will time out at 913.829323s 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9effd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fefa0136d60 legacy=0x7fefa009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fefa01375a0 0x7fefa013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59924/0 (socket says 172.21.15.171:59924) 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9f7fe640 1 -- 172.21.15.171:0/623589177 learned_addr learned my addr 172.21.15.171:0/623589177 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138057558 0 0) 0x7fefa009b5a0 con 0x7fefa01375a0 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient(hunting): my global_id is 5026 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef8c003520 con 0x7fefa01375a0 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3850197084 0 0) 0x7fef8c003520 con 0x7fefa01375a0 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 >> v1:172.21.15.148:6790/0 conn(0x7fefa0137140 legacy=0x7fefa013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 >> v1:172.21.15.148:6789/0 conn(0x7fefa0136d60 legacy=0x7fefa009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 monclient: found mon.noname-c 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa0143b20 con 0x7fefa01375a0 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 5 monclient: authenticate success, global_id 5026 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef98001610 con 0x7fefa01375a0 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fef9dffb640 10 monclient: dump: 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.324+0000 7fefa57bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef98001900 con 0x7fefa01375a0 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fef9dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fef9dffb640 1 -- 172.21.15.171:0/623589177 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef98001cd0 con 0x7fefa01375a0 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 4 set_mon_vals no callback set 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fef9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fef9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fef9dffb640 10 monclient: dump: 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.337 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 10 monclient: shutdown 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 1 -- 172.21.15.171:0/623589177 >> v1:172.21.15.171:6789/0 conn(0x7fefa01375a0 legacy=0x7fefa013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 1 -- 172.21.15.171:0/623589177 shutdown_connections 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 1 -- 172.21.15.171:0/623589177 >> 172.21.15.171:0/623589177 conn(0x7fefa00a9ae0 msgr2=0x7fefa00a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 1 -- 172.21.15.171:0/623589177 shutdown_connections 2024-09-16T04:08:58.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.325+0000 7fefa57bd640 1 -- 172.21.15.171:0/623589177 wait complete. 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.326+0000 7fefa57bd640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.326+0000 7fefa57bd640 10 monclient: monmap: 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.326+0000 7fefa57bd640 1 Processor -- start 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.326+0000 7fefa57bd640 1 -- start start 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.326+0000 7fefa57bd640 10 monclient: init 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient(hunting): picked mon.noname-a con 0x7fefa0136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient(hunting): picked mon.noname-b con 0x7fefa0137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient(hunting): picked mon.noname-c con 0x7fefa01375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa0139470 con 0x7fefa0136d60 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa0139750 con 0x7fefa0137140 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa009f170 con 0x7fefa01375a0 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient(hunting): authenticate will time out at 913.832275s 2024-09-16T04:08:58.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef9effd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fefa0136d60 legacy=0x7fefa009e350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef9e7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fefa0137140 0x7fefa009ea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59936/0 (socket says 172.21.15.171:59936) 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef9e7fc640 1 -- 172.21.15.171:0/3828417553 learned_addr learned my addr 172.21.15.171:0/3828417553 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4195213787 0 0) 0x7fefa0139750 con 0x7fefa0137140 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): my global_id is 5029 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef80003520 con 0x7fefa0137140 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2597909595 0 0) 0x7fefa009f170 con 0x7fefa01375a0 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _negotiate 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): my global_id is 5060 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefa0139750 con 0x7fefa01375a0 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1034445573 0 0) 0x7fef80003520 con 0x7fefa0137140 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefa009f170 con 0x7fefa0137140 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef880019c0 con 0x7fefa0137140 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): dump: 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.342 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2514771010 0 0) 0x7fefa0139750 con 0x7fefa01375a0 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef80003520 con 0x7fefa01375a0 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef98001430 con 0x7fefa01375a0 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient(hunting): dump: 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1555628132 0 0) 0x7fefa009f170 con 0x7fefa0137140 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 >> v1:172.21.15.148:6790/0 conn(0x7fefa01375a0 legacy=0x7fefa01d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 >> v1:172.21.15.148:6789/0 conn(0x7fefa0136d60 legacy=0x7fefa009e350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:58.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 monclient: found mon.b 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa00a8160 con 0x7fefa0137140 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fef977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 5 monclient: authenticate success, global_id 5029 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient: _renew_subs 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.327+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fefa013f6d0 con 0x7fefa0137140 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 10 monclient: _renew_subs 2024-09-16T04:08:58.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fefa0139190 con 0x7fefa0137140 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef880038c0 con 0x7fefa0137140 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef88003c90 con 0x7fefa0137140 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef9d7fa640 4 set_mon_vals no callback set 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 10 monclient: dump: 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.346 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fef88003eb0 con 0x7fefa0137140 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.328+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef84005080 con 0x7fefa0137140 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.329+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 4754+0+0 (unknown 3373874907 0 0) 0x7fef88047280 con 0x7fefa0137140 2024-09-16T04:08:58.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.329+0000 7fef9effd640 10 monclient: get_auth_request con 0x7fef80038ea0 auth_method 0 2024-09-16T04:08:58.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.333+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fef88015df0 con 0x7fefa0137140 2024-09-16T04:08:58.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.333+0000 7fef977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.333+0000 7fef977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:58.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.506+0000 7fefa57bd640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:58.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.506+0000 7fefa57bd640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:58.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.506+0000 7fefa57bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:58.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:58.506+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7fef84005370 con 0x7fefa0137140 2024-09-16T04:08:59.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.221+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map v71) ==== 203+0+0 (unknown 1120037864 0 0) 0x7fef880156f0 con 0x7fefa0137140 2024-09-16T04:08:59.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.221+0000 7fef977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.221+0000 7fef977fe640 10 monclient: _finish_command 2 = system:0 add item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo} to crush map 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fefa57bd640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fefa57bd640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fefa57bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7fef84005bd0 con 0x7fefa0137140 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fef977fe640 1 -- 172.21.15.171:0/3828417553 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v71) ==== 201+0+0 (unknown 3918607151 0 0) 0x7fef88019a40 con 0x7fefa0137140 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fef977fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.328+0000 7fef977fe640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T04:08:59.331 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T04:08:59.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 >> v1:172.21.15.171:6812/7327273 conn(0x7fef80038ea0 legacy=0x7fef8003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 10 monclient: shutdown 2024-09-16T04:08:59.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:59.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 >> v1:172.21.15.171:6789/0 conn(0x7fefa0137140 legacy=0x7fefa009ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 shutdown_connections 2024-09-16T04:08:59.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 >> 172.21.15.171:0/3828417553 conn(0x7fefa00a9ae0 msgr2=0x7fefa00a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:59.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 shutdown_connections 2024-09-16T04:08:59.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.330+0000 7fefa57bd640 1 -- 172.21.15.171:0/3828417553 wait complete. 2024-09-16T04:08:59.341 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 7 1 host=host1 root=foo 2024-09-16T04:08:59.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.421+0000 7f7d480d0640 10 monclient: get_monmap_and_config 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.421+0000 7f7d480d0640 10 monclient: build_initial_monmap 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.421+0000 7f7d480d0640 10 monclient: monmap: 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:59.422460+0000 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:59.422460+0000 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:59.424 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.425 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:59.425 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:59.425 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:59.425 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 1 Processor -- start 2024-09-16T04:08:59.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 1 -- start start 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient: init 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient(hunting): picked mon.noname-b con 0x7f7d4009ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient(hunting): picked mon.noname-a con 0x7f7d400a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient(hunting): picked mon.noname-c con 0x7f7d400a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d400a0f20 con 0x7f7d400a0370 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d400a1200 con 0x7f7d400a0800 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d4009bca0 con 0x7f7d4009ff90 2024-09-16T04:08:59.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.422+0000 7f7d480d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d480d0640 10 monclient(hunting): authenticate will time out at 914.928012s 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d4566b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7d400a0370 legacy=0x7f7d40139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d4666d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7d400a0800 0x7f7d4013d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59952/0 (socket says 172.21.15.171:59952) 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d4666d640 1 -- 172.21.15.171:0/1958381839 learned_addr learned my addr 172.21.15.171:0/1958381839 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 366898462 0 0) 0x7f7d400a1200 con 0x7f7d400a0800 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): my global_id is 5032 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d2c003520 con 0x7f7d400a0800 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181761385 0 0) 0x7f7d4009bca0 con 0x7f7d4009ff90 2024-09-16T04:08:59.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient(hunting): my global_id is 5063 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d400a1200 con 0x7f7d4009ff90 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 953698813 0 0) 0x7f7d2c003520 con 0x7f7d400a0800 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 >> v1:172.21.15.148:6790/0 conn(0x7f7d4009ff90 legacy=0x7f7d4009f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 >> v1:172.21.15.148:6789/0 conn(0x7f7d400a0370 legacy=0x7f7d40139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: _finish_hunting 0 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 monclient: found mon.noname-c 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:59.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4009e830 con 0x7f7d400a0800 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: _finish_auth 0 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: _check_auth_tickets 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d480d0640 5 monclient: authenticate success, global_id 5032 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d38001910 con 0x7f7d400a0800 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: dump: 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.436 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d480d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d38001b70 con 0x7f7d400a0800 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 1 -- 172.21.15.171:0/1958381839 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d38001f40 con 0x7f7d400a0800 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d480d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:59.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d480d0640 4 set_mon_vals no callback set 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.423+0000 7f7d44e6a640 10 monclient: dump: 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.438 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 10 monclient: shutdown 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 1 -- 172.21.15.171:0/1958381839 >> v1:172.21.15.171:6789/0 conn(0x7f7d400a0800 legacy=0x7f7d4013d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 1 -- 172.21.15.171:0/1958381839 shutdown_connections 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 1 -- 172.21.15.171:0/1958381839 >> 172.21.15.171:0/1958381839 conn(0x7f7d400a5450 msgr2=0x7f7d400a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 1 -- 172.21.15.171:0/1958381839 shutdown_connections 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 1 -- 172.21.15.171:0/1958381839 wait complete. 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 10 monclient: build_initial_monmap 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.424+0000 7f7d480d0640 10 monclient: monmap: 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:59.439 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 1 Processor -- start 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 1 -- start start 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient: init 2024-09-16T04:08:59.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient(hunting): picked mon.noname-c con 0x7f7d4009ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient(hunting): picked mon.noname-a con 0x7f7d400a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient(hunting): picked mon.noname-b con 0x7f7d400a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d4013dca0 con 0x7f7d400a0370 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d4009d110 con 0x7f7d400a0800 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d40071630 con 0x7f7d4009ff90 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d480d0640 10 monclient(hunting): authenticate will time out at 914.930629s 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d4666d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7d400a0800 0x7f7d401d3960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59958/0 (socket says 172.21.15.171:59958) 2024-09-16T04:08:59.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d4666d640 1 -- 172.21.15.171:0/1991331630 learned_addr learned my addr 172.21.15.171:0/1991331630 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d4566b640 1 -- 172.21.15.171:0/1991331630 >> v1:172.21.15.148:6789/0 conn(0x7f7d400a0370 legacy=0x7f7d4009ca00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 962990658 0 0) 0x7f7d4009d110 con 0x7f7d400a0800 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d367fc640 10 monclient(hunting): my global_id is 5035 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.425+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d24003520 con 0x7f7d400a0800 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 484651758 0 0) 0x7f7d24003520 con 0x7f7d400a0800 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d4009d110 con 0x7f7d400a0800 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d38001f80 con 0x7f7d400a0800 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:59.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): dump: 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3749818361 0 0) 0x7f7d40071630 con 0x7f7d4009ff90 2024-09-16T04:08:59.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient(hunting): my global_id is 5066 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d24003520 con 0x7f7d4009ff90 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 22311657 0 0) 0x7f7d4009d110 con 0x7f7d400a0800 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 >> v1:172.21.15.148:6790/0 conn(0x7f7d4009ff90 legacy=0x7f7d4009c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 >> v1:172.21.15.148:6789/0 conn(0x7f7d400a0370 legacy=0x7f7d4009ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 monclient: found mon.b 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4013d2c0 con 0x7f7d400a0800 2024-09-16T04:08:59.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 5 monclient: authenticate success, global_id 5035 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 10 monclient: _renew_subs 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4013d5a0 con 0x7f7d400a0800 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 10 monclient: _renew_subs 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d40135360 con 0x7f7d400a0800 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d38004f10 con 0x7f7d400a0800 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:59.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d38005260 con 0x7f7d400a0800 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d367fc640 10 monclient: dump: 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.426+0000 7f7d37fff640 4 set_mon_vals no callback set 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7d38006500 con 0x7f7d400a0800 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d480d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d480d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d480d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4009c0b0 con 0x7f7d400a0800 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 5036+0+0 (unknown 118981832 0 0) 0x7f7d38047820 con 0x7f7d400a0800 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.427+0000 7f7d45e6c640 10 monclient: get_auth_request con 0x7f7d24038fb0 auth_method 0 2024-09-16T04:08:59.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.431+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7d380161e0 con 0x7f7d400a0800 2024-09-16T04:08:59.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.431+0000 7f7d367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:59.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.431+0000 7f7d367fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:59.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.600+0000 7f7d480d0640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.600+0000 7f7d480d0640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.600+0000 7f7d480d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.600+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7f7d4006bd60 con 0x7f7d400a0800 2024-09-16T04:08:59.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.601+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v71) ==== 201+0+0 (unknown 3918607151 0 0) 0x7f7d3801e630 con 0x7f7d400a0800 2024-09-16T04:08:59.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.601+0000 7f7d367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.601+0000 7f7d367fc640 10 monclient: _finish_command 2 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T04:08:59.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d480d0640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d480d0640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d480d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) -- 0x7f7d4013e880 con 0x7f7d400a0800 2024-09-16T04:08:59.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d367fc640 1 -- 172.21.15.171:0/1991331630 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change v71) ==== 201+0+0 (unknown 3918607151 0 0) 0x7f7d38006920 con 0x7f7d400a0800 2024-09-16T04:08:59.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d367fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 7, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-16T04:08:59.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.701+0000 7f7d367fc640 10 monclient: _finish_command 3 = system:0 set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T04:08:59.704 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 7 name 'osd.7' weight 1 at location {host=host1,root=foo}: no change 2024-09-16T04:08:59.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 >> v1:172.21.15.171:6812/7327273 conn(0x7f7d24038fb0 legacy=0x7f7d2403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 10 monclient: shutdown 2024-09-16T04:08:59.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:59.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 >> v1:172.21.15.171:6789/0 conn(0x7f7d400a0800 legacy=0x7f7d401d3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 shutdown_connections 2024-09-16T04:08:59.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 >> 172.21.15.171:0/1991331630 conn(0x7f7d400a5450 msgr2=0x7f7d400a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:59.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 shutdown_connections 2024-09-16T04:08:59.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.703+0000 7f7d480d0640 1 -- 172.21.15.171:0/1991331630 wait complete. 2024-09-16T04:08:59.714 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T04:08:59.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.792+0000 7fe6050af640 10 monclient: get_monmap_and_config 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.792+0000 7fe6050af640 10 monclient: build_initial_monmap 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.792+0000 7fe6050af640 10 monclient: monmap: 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:08:59.792838+0000 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:08:59.792838+0000 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.794 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:59.795 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:08:59.795 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:08:59.795 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.792+0000 7fe6050af640 1 Processor -- start 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.792+0000 7fe6050af640 1 -- start start 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.792+0000 7fe6050af640 10 monclient: init 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient(hunting): picked mon.noname-a con 0x7fe600136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient(hunting): picked mon.noname-b con 0x7fe600137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient(hunting): picked mon.noname-c con 0x7fe6001375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60009b2c0 con 0x7fe600136d60 2024-09-16T04:08:59.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60009b5a0 con 0x7fe6001375a0 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6000a6640 con 0x7fe600137140 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient(hunting): authenticate will time out at 915.298219s 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe600136d60 legacy=0x7fe60009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5ff59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe6001375a0 0x7fe60013f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59960/0 (socket says 172.21.15.171:59960) 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5ff59e640 1 -- 172.21.15.171:0/3259865656 learned_addr learned my addr 172.21.15.171:0/3259865656 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 66090919 0 0) 0x7fe60009b5a0 con 0x7fe6001375a0 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient(hunting): my global_id is 5038 2024-09-16T04:08:59.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5ec003520 con 0x7fe6001375a0 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2023809782 0 0) 0x7fe5ec003520 con 0x7fe6001375a0 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 >> v1:172.21.15.148:6790/0 conn(0x7fe600137140 legacy=0x7fe60013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 >> v1:172.21.15.148:6789/0 conn(0x7fe600136d60 legacy=0x7fe60009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: _finish_hunting 0 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 monclient: found mon.noname-c 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe600143b20 con 0x7fe6001375a0 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: _finish_auth 0 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: _check_auth_tickets 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:59.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5f8001990 con 0x7fe6001375a0 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: dump: 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 5 monclient: authenticate success, global_id 5038 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5f8001c80 con 0x7fe6001375a0 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 1 -- 172.21.15.171:0/3259865656 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5f8002090 con 0x7fe6001375a0 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe5fe59c640 10 monclient: dump: 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.807 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.793+0000 7fe6050af640 4 set_mon_vals no callback set 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 10 monclient: shutdown 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 1 -- 172.21.15.171:0/3259865656 >> v1:172.21.15.171:6789/0 conn(0x7fe6001375a0 legacy=0x7fe60013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 1 -- 172.21.15.171:0/3259865656 shutdown_connections 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 1 -- 172.21.15.171:0/3259865656 >> 172.21.15.171:0/3259865656 conn(0x7fe6000a9ae0 msgr2=0x7fe6000a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:59.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 1 -- 172.21.15.171:0/3259865656 shutdown_connections 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.794+0000 7fe6050af640 1 -- 172.21.15.171:0/3259865656 wait complete. 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.795+0000 7fe6050af640 10 monclient: build_initial_monmap 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.795+0000 7fe6050af640 10 monclient: monmap: 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:59.809 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.795+0000 7fe6050af640 1 Processor -- start 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.795+0000 7fe6050af640 1 -- start start 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.795+0000 7fe6050af640 10 monclient: init 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:59.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient(hunting): picked mon.noname-a con 0x7fe600136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient(hunting): picked mon.noname-c con 0x7fe600137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient(hunting): picked mon.noname-b con 0x7fe6001375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6000a8900 con 0x7fe600136d60 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6000a83b0 con 0x7fe6001375a0 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6000a03b0 con 0x7fe600137140 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:59.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient(hunting): authenticate will time out at 915.301158s 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5fed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe600136d60 legacy=0x7fe60009d560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5ff59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe6001375a0 0x7fe6001d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59970/0 (socket says 172.21.15.171:59970) 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5ff59e640 1 -- 172.21.15.171:0/4252712374 learned_addr learned my addr 172.21.15.171:0/4252712374 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469512578 0 0) 0x7fe6000a83b0 con 0x7fe6001375a0 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): my global_id is 5041 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5e0003520 con 0x7fe6001375a0 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627568313 0 0) 0x7fe6000a03b0 con 0x7fe600137140 2024-09-16T04:08:59.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): my global_id is 5072 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6000a83b0 con 0x7fe600137140 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2360423237 0 0) 0x7fe5e0003520 con 0x7fe6001375a0 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6000a03b0 con 0x7fe6001375a0 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5f80017b0 con 0x7fe6001375a0 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): dump: 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.813 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2736110135 0 0) 0x7fe6000a83b0 con 0x7fe600137140 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5e0003520 con 0x7fe600137140 2024-09-16T04:08:59.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5e8001830 con 0x7fe600137140 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient(hunting): dump: 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.815 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3988622413 0 0) 0x7fe6000a03b0 con 0x7fe6001375a0 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 >> v1:172.21.15.148:6790/0 conn(0x7fe600137140 legacy=0x7fe60009dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 >> v1:172.21.15.148:6789/0 conn(0x7fe600136d60 legacy=0x7fe60009d560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 monclient: found mon.b 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60012de00 con 0x7fe6001375a0 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:59.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 5 monclient: authenticate success, global_id 5041 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient: _renew_subs 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe60012e7d0 con 0x7fe6001375a0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient: _renew_subs 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe600136570 con 0x7fe6001375a0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5f8004ff0 con 0x7fe6001375a0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.796+0000 7fe5f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5f8005340 con 0x7fe6001375a0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe5f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe5f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:59.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe5fdd9b640 4 set_mon_vals no callback set 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe5f7fff640 10 monclient: dump: 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:08:59.818 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe5f8010770 con 0x7fe6001375a0 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe6050af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe6050af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe6050af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.797+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e4005080 con 0x7fe6001375a0 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.798+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(71..71 src has 1..71) ==== 5036+0+0 (unknown 118981832 0 0) 0x7fe5f8047690 con 0x7fe6001375a0 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.798+0000 7fe5fed9d640 10 monclient: get_auth_request con 0x7fe5e0038f10 auth_method 0 2024-09-16T04:08:59.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.802+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe5f8016050 con 0x7fe6001375a0 2024-09-16T04:08:59.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.802+0000 7fe5f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:59.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.802+0000 7fe5f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:59.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.970+0000 7fe6050af640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:08:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.971+0000 7fe6050af640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:08:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.971+0000 7fe6050af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:08:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:08:59.971+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7fe5e4005370 con 0x7fe6001375a0 2024-09-16T04:09:00.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.231+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map v72) ==== 203+0+0 (unknown 1316347800 0 0) 0x7fe5f8013d20 con 0x7fe6001375a0 2024-09-16T04:09:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.231+0000 7fe5f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.231+0000 7fe5f7fff640 10 monclient: _finish_command 2 = system:0 add item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo} to crush map 2024-09-16T04:09:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe6050af640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe6050af640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe6050af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7fe5e4005ab0 con 0x7fe6001375a0 2024-09-16T04:09:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe5f7fff640 1 -- 172.21.15.171:0/4252712374 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v72) ==== 201+0+0 (unknown 416732182 0 0) 0x7fe5f8044f30 con 0x7fe6001375a0 2024-09-16T04:09:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe5f7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.334+0000 7fe5f7fff640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T04:09:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T04:09:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.336+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 >> v1:172.21.15.171:6812/7327273 conn(0x7fe5e0038f10 legacy=0x7fe5e003b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.336+0000 7fe6050af640 10 monclient: shutdown 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.336+0000 7fe6050af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.336+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 >> v1:172.21.15.171:6789/0 conn(0x7fe6001375a0 legacy=0x7fe6001d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.336+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 shutdown_connections 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.336+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 >> 172.21.15.171:0/4252712374 conn(0x7fe6000a9ae0 msgr2=0x7fe600072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.337+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 shutdown_connections 2024-09-16T04:09:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.337+0000 7fe6050af640 1 -- 172.21.15.171:0/4252712374 wait complete. 2024-09-16T04:09:00.347 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 9 1 host=host2 root=foo 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.428+0000 7f21b1827640 10 monclient: get_monmap_and_config 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.428+0000 7f21b1827640 10 monclient: build_initial_monmap 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.428+0000 7f21b1827640 10 monclient: monmap: 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:00.429053+0000 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:00.429053+0000 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:00.430 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:00.431 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:00.431 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.428+0000 7f21b1827640 1 Processor -- start 2024-09-16T04:09:00.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.428+0000 7f21b1827640 1 -- start start 2024-09-16T04:09:00.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.428+0000 7f21b1827640 10 monclient: init 2024-09-16T04:09:00.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:00.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient(hunting): picked mon.noname-b con 0x7f21ac0a3b50 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient(hunting): picked mon.noname-a con 0x7f21ac0a3f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient(hunting): picked mon.noname-c con 0x7f21ac13cec0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac0a4cf0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac09aa90 con 0x7f21ac13cec0 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac09af70 con 0x7f21ac0a3b50 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21b1827640 10 monclient(hunting): authenticate will time out at 915.934529s 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21ab7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f21ac13cec0 0x7f21ac13f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59982/0 (socket says 172.21.15.171:59982) 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21aa7fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f21ac0a3f30 legacy=0x7f21ac0a4340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21ab7fe640 1 -- 172.21.15.171:0/1801909524 learned_addr learned my addr 172.21.15.171:0/1801909524 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673153163 0 0) 0x7f21ac09aa90 con 0x7f21ac13cec0 2024-09-16T04:09:00.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21a9ffb640 10 monclient(hunting): my global_id is 5044 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.429+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2194003520 con 0x7f21ac13cec0 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481139394 0 0) 0x7f21ac09af70 con 0x7f21ac0a3b50 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient(hunting): my global_id is 5075 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21ac09aa90 con 0x7f21ac0a3b50 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2231289125 0 0) 0x7f2194003520 con 0x7f21ac13cec0 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 >> v1:172.21.15.148:6790/0 conn(0x7f21ac0a3b50 legacy=0x7f21ac09bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 >> v1:172.21.15.148:6789/0 conn(0x7f21ac0a3f30 legacy=0x7f21ac0a4340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 monclient: found mon.noname-c 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac09b250 con 0x7f21ac13cec0 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f21a4001990 con 0x7f21ac13cec0 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 5 monclient: authenticate success, global_id 5044 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: dump: 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:00.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21a4001c40 con 0x7f21ac13cec0 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 4 set_mon_vals no callback set 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 1 -- 172.21.15.171:0/1801909524 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f21a4002010 con 0x7f21ac13cec0 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21a9ffb640 10 monclient: dump: 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.445 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 10 monclient: shutdown 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 1 -- 172.21.15.171:0/1801909524 >> v1:172.21.15.171:6789/0 conn(0x7f21ac13cec0 legacy=0x7f21ac13f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 1 -- 172.21.15.171:0/1801909524 shutdown_connections 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 1 -- 172.21.15.171:0/1801909524 >> 172.21.15.171:0/1801909524 conn(0x7f21ac0a9940 msgr2=0x7f21ac0a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 1 -- 172.21.15.171:0/1801909524 shutdown_connections 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.430+0000 7f21b1827640 1 -- 172.21.15.171:0/1801909524 wait complete. 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.431+0000 7f21b1827640 10 monclient: build_initial_monmap 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.431+0000 7f21b1827640 10 monclient: monmap: 2024-09-16T04:09:00.446 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.431+0000 7f21b1827640 1 Processor -- start 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.431+0000 7f21b1827640 1 -- start start 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.431+0000 7f21b1827640 10 monclient: init 2024-09-16T04:09:00.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient(hunting): picked mon.noname-a con 0x7f21ac0a3b50 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient(hunting): picked mon.noname-b con 0x7f21ac0a3f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient(hunting): picked mon.noname-c con 0x7f21ac13cec0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac13c000 con 0x7f21ac0a3b50 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac13c2e0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac0a06c0 con 0x7f21ac13cec0 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21b1827640 10 monclient(hunting): authenticate will time out at 915.937341s 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21aaffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f21ac0a3b50 legacy=0x7f21ac1386b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21aa7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f21ac0a3f30 0x7f21ac138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59988/0 (socket says 172.21.15.171:59988) 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f21aa7fc640 1 -- 172.21.15.171:0/2392810314 learned_addr learned my addr 172.21.15.171:0/2392810314 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:00.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1086375293 0 0) 0x7f21ac13c2e0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): my global_id is 5047 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2198003520 con 0x7f21ac0a3f30 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487875351 0 0) 0x7f21ac0a06c0 con 0x7f21ac13cec0 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): my global_id is 5078 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21ac13c2e0 con 0x7f21ac13cec0 2024-09-16T04:09:00.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 981694122 0 0) 0x7f2198003520 con 0x7f21ac0a3f30 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21ac0a06c0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f21a0004180 con 0x7f21ac0a3f30 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): dump: 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.450 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.432+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2918183945 0 0) 0x7f21ac13c2e0 con 0x7f21ac13cec0 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003520 con 0x7f21ac13cec0 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f21a4001df0 con 0x7f21ac13cec0 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient(hunting): dump: 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.451 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1007139401 0 0) 0x7f21ac0a06c0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 >> v1:172.21.15.148:6790/0 conn(0x7f21ac13cec0 legacy=0x7f21ac1394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 >> v1:172.21.15.148:6789/0 conn(0x7f21ac0a3b50 legacy=0x7f21ac1386b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 monclient: found mon.b 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac0a8c00 con 0x7f21ac0a3f30 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 5 monclient: authenticate success, global_id 5047 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 10 monclient: _renew_subs 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac13c5c0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 10 monclient: _renew_subs 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f21ac1d4bc0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21a00044a0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:00.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f21a00047f0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f21a97fa640 4 set_mon_vals no callback set 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 10 monclient: dump: 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.454 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.433+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f21a0004990 con 0x7f21ac0a3f30 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.434+0000 7f21b1827640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.434+0000 7f21b1827640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.434+0000 7f21b1827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.434+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f218c005080 con 0x7f21ac0a3f30 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.434+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 5280+0+0 (unknown 1880471802 0 0) 0x7f21a00473d0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.434+0000 7f21aaffd640 10 monclient: get_auth_request con 0x7f2198038f40 auth_method 0 2024-09-16T04:09:00.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.438+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f21a0015c40 con 0x7f21ac0a3f30 2024-09-16T04:09:00.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.438+0000 7f2193fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:00.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.438+0000 7f2193fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:00.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f21b1827640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f21b1827640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f21b1827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7f218c005370 con 0x7f21ac0a3f30 2024-09-16T04:09:00.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v72) ==== 201+0+0 (unknown 416732182 0 0) 0x7f21a0015540 con 0x7f21ac0a3f30 2024-09-16T04:09:00.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f2193fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.606+0000 7f2193fff640 10 monclient: _finish_command 2 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T04:09:00.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.706+0000 7f21b1827640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.706+0000 7f21b1827640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.706+0000 7f21b1827640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.706+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) -- 0x7f218c005bd0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.707+0000 7f2193fff640 1 -- 172.21.15.171:0/2392810314 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change v72) ==== 201+0+0 (unknown 416732182 0 0) 0x7f21a001e0f0 con 0x7f21ac0a3f30 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.707+0000 7f2193fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 9, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.707+0000 7f2193fff640 10 monclient: _finish_command 3 = system:0 set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T04:09:00.709 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 9 name 'osd.9' weight 1 at location {host=host2,root=foo}: no change 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 >> v1:172.21.15.171:6812/7327273 conn(0x7f2198038f40 legacy=0x7f219803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 10 monclient: shutdown 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 >> v1:172.21.15.171:6789/0 conn(0x7f21ac0a3f30 legacy=0x7f21ac138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 shutdown_connections 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 >> 172.21.15.171:0/2392810314 conn(0x7f21ac0a9940 msgr2=0x7f21ac0a7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 shutdown_connections 2024-09-16T04:09:00.711 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.709+0000 7f21b1827640 1 -- 172.21.15.171:0/2392810314 wait complete. 2024-09-16T04:09:00.720 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T04:09:00.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.797+0000 7f48d314b640 10 monclient: get_monmap_and_config 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.797+0000 7f48d314b640 10 monclient: build_initial_monmap 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.797+0000 7f48d314b640 10 monclient: monmap: 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:00.798698+0000 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:00.798698+0000 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:00.800 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.798+0000 7f48d314b640 1 Processor -- start 2024-09-16T04:09:00.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.798+0000 7f48d314b640 1 -- start start 2024-09-16T04:09:00.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.798+0000 7f48d314b640 10 monclient: init 2024-09-16T04:09:00.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:00.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient(hunting): picked mon.noname-c con 0x7f48cc09b660 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient(hunting): picked mon.noname-a con 0x7f48cc0734b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient(hunting): picked mon.noname-b con 0x7f48cc136d70 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc09c150 con 0x7f48cc0734b0 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc0998b0 con 0x7f48cc09b660 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc099b90 con 0x7f48cc136d70 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient(hunting): authenticate will time out at 916.304277s 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d0ee7640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f48cc09b660 0x7f48cc09ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60004/0 (socket says 172.21.15.171:60004) 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d0ee7640 1 -- 172.21.15.171:0/1056817252 learned_addr learned my addr 172.21.15.171:0/1056817252 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cbfff640 1 -- 172.21.15.171:0/1056817252 >> v1:172.21.15.148:6789/0 conn(0x7f48cc0734b0 legacy=0x7f48cc13bbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:00.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121447394 0 0) 0x7f48cc0998b0 con 0x7f48cc09b660 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient(hunting): my global_id is 5050 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48b8003520 con 0x7f48cc09b660 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3790166818 0 0) 0x7f48b8003520 con 0x7f48cc09b660 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 >> v1:172.21.15.148:6790/0 conn(0x7f48cc136d70 legacy=0x7f48cc13f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 >> v1:172.21.15.148:6789/0 conn(0x7f48cc0734b0 legacy=0x7f48cc13bbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc099e70 con 0x7f48cc09b660 2024-09-16T04:09:00.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48bc001610 con 0x7f48cc09b660 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 5 monclient: authenticate success, global_id 5050 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: dump: 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.811 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48bc001900 con 0x7f48cc09b660 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 1 -- 172.21.15.171:0/1056817252 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48bc001d10 con 0x7f48cc09b660 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48d314b640 4 set_mon_vals no callback set 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.799+0000 7f48cb7fe640 10 monclient: dump: 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 10 monclient: shutdown 2024-09-16T04:09:00.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 1 -- 172.21.15.171:0/1056817252 >> v1:172.21.15.171:6789/0 conn(0x7f48cc09b660 legacy=0x7f48cc09ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 1 -- 172.21.15.171:0/1056817252 shutdown_connections 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 1 -- 172.21.15.171:0/1056817252 >> 172.21.15.171:0/1056817252 conn(0x7f48cc0a1e90 msgr2=0x7f48cc0a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 1 -- 172.21.15.171:0/1056817252 shutdown_connections 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 1 -- 172.21.15.171:0/1056817252 wait complete. 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 10 monclient: build_initial_monmap 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.800+0000 7f48d314b640 10 monclient: monmap: 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.814 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 1 Processor -- start 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 1 -- start start 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 10 monclient: init 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 10 monclient(hunting): picked mon.noname-b con 0x7f48cc0734b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:00.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 10 monclient(hunting): picked mon.noname-c con 0x7f48cc09b660 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 10 monclient(hunting): picked mon.noname-a con 0x7f48cc136d70 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc090330 con 0x7f48cc136d70 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.801+0000 7f48d314b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc097340 con 0x7f48cc0734b0 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48cc0923d0 con 0x7f48cc09b660 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 10 monclient(hunting): authenticate will time out at 916.307018s 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d0ee7640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f48cc0734b0 0x7f48cc08e940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60018/0 (socket says 172.21.15.171:60018) 2024-09-16T04:09:00.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d0ee7640 1 -- 172.21.15.171:0/1476435770 learned_addr learned my addr 172.21.15.171:0/1476435770 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d16e8640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f48cc136d70 legacy=0x7f48cc1d39d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624370668 0 0) 0x7f48cc097340 con 0x7f48cc0734b0 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): my global_id is 5053 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48ac003520 con 0x7f48cc0734b0 2024-09-16T04:09:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875687325 0 0) 0x7f48cc0923d0 con 0x7f48cc09b660 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): my global_id is 5084 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48cc097340 con 0x7f48cc09b660 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4036915900 0 0) 0x7f48ac003520 con 0x7f48cc0734b0 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48cc0923d0 con 0x7f48cc0734b0 2024-09-16T04:09:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48bc0015d0 con 0x7f48cc0734b0 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): dump: 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.819 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1372452701 0 0) 0x7f48cc0923d0 con 0x7f48cc0734b0 2024-09-16T04:09:00.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.148:6790/0 conn(0x7f48cc09b660 legacy=0x7f48cc08f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.148:6789/0 conn(0x7f48cc136d70 legacy=0x7f48cc1d39d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 monclient: found mon.b 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48cc0926b0 con 0x7f48cc0734b0 2024-09-16T04:09:00.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 5 monclient: authenticate success, global_id 5053 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 10 monclient: _renew_subs 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc097ce0 con 0x7f48cc0734b0 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 10 monclient: _renew_subs 2024-09-16T04:09:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48cc08f760 con 0x7f48cc0734b0 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48bc004cc0 con 0x7f48cc0734b0 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48bc005090 con 0x7f48cc0734b0 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48caffd640 4 set_mon_vals no callback set 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.802+0000 7f48c97fa640 10 monclient: dump: 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.824 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f48bc0052b0 con 0x7f48cc0734b0 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48d314b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48d314b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48d314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48cc134680 con 0x7f48cc0734b0 2024-09-16T04:09:00.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 5280+0+0 (unknown 1880471802 0 0) 0x7f48bc0475b0 con 0x7f48cc0734b0 2024-09-16T04:09:00.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.803+0000 7f48cbfff640 10 monclient: get_auth_request con 0x7f48ac038f60 auth_method 0 2024-09-16T04:09:00.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.808+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f48bc019a70 con 0x7f48cc0734b0 2024-09-16T04:09:00.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.808+0000 7f48c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:00.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.808+0000 7f48c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.974+0000 7f48d314b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.974+0000 7f48d314b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.974+0000 7f48d314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.974+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f48cc09c580 con 0x7f48cc0734b0 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48d0ee7640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6789/0 conn(0x7f48cc0734b0 legacy=0x7f48cc08e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48d0ee7640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6789/0 conn(0x7f48cc0734b0 legacy=0x7f48cc08e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48d0ee7640 1 --1- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6789/0 conn(0x7f48cc0734b0 0x7f48cc08e940 :-1 s=OPENED pgs=373 cs=1 l=1).handle_message read tag failed 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48d0ee7640 1 --1- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6789/0 conn(0x7f48cc0734b0 0x7f48cc08e940 :-1 s=OPENED pgs=373 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:09:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.171:6789/0 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6789/0 conn(0x7f48cc0734b0 legacy=0x7f48cc08e940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): picked mon.c con 0x7f48cc09b660 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): picked mon.a con 0x7f48cc136d70 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): picked mon.b con 0x7f48ac048f10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48cc0923d0 con 0x7f48cc136d70 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48cc090330 con 0x7f48ac048f10 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f48ac003520 con 0x7f48cc09b660 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48d16e8640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.148:6789/0 conn(0x7f48cc136d70 legacy=0x7f48ac047380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:00.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910654267 0 0) 0x7f48cc090330 con 0x7f48ac048f10 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f48ac04e030 con 0x7f48ac048f10 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 126188061 0 0) 0x7f48ac04e030 con 0x7f48ac048f10 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.148:6790/0 conn(0x7f48cc09b660 legacy=0x7f48ac043920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.148:6789/0 conn(0x7f48cc136d70 legacy=0x7f48ac047380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 monclient: found mon.b 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f48ac050150 con 0x7f48ac048f10 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:00.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f48cc0923d0 con 0x7f48ac048f10 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48bc0139a0 con 0x7f48ac048f10 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: dump: 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:00.982 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48bc044d60 con 0x7f48ac048f10 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48c97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.975+0000 7f48caffd640 4 set_mon_vals no callback set 2024-09-16T04:09:00.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.976+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f48bc0450e0 con 0x7f48ac048f10 2024-09-16T04:09:00.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.984+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v73) ==== 138+0+0 (unknown 2632298601 0 0) 0x7f48bc033d30 con 0x7f48ac048f10 2024-09-16T04:09:00.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.984+0000 7f48c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:00.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:00.984+0000 7f48c97fa640 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-16T04:09:01.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.086+0000 7f48d314b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.086+0000 7f48d314b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.086+0000 7f48d314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.086+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7f48cc0008d0 con 0x7f48ac048f10 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.087+0000 7f48c97fa640 1 -- 172.21.15.171:0/1476435770 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v73) ==== 126+0+0 (unknown 3177338128 0 0) 0x7f48bc0263f0 con 0x7f48ac048f10 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.087+0000 7f48c97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.087+0000 7f48c97fa640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T04:09:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'bar' already exists 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6812/7327273 conn(0x7f48ac038f60 legacy=0x7f48ac03b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 10 monclient: shutdown 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 >> v1:172.21.15.171:6789/0 conn(0x7f48ac048f10 legacy=0x7f48ac04b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 shutdown_connections 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 >> 172.21.15.171:0/1476435770 conn(0x7f48cc0a1e90 msgr2=0x7f48cc098590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 shutdown_connections 2024-09-16T04:09:01.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.089+0000 7f48d314b640 1 -- 172.21.15.171:0/1476435770 wait complete. 2024-09-16T04:09:01.099 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket bar root 2024-09-16T04:09:01.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.177+0000 7fe9406c9640 10 monclient: get_monmap_and_config 2024-09-16T04:09:01.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.177+0000 7fe9406c9640 10 monclient: build_initial_monmap 2024-09-16T04:09:01.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.177+0000 7fe9406c9640 10 monclient: monmap: 2024-09-16T04:09:01.179 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:01.179 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:01.178437+0000 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:01.178437+0000 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:01.180 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 1 Processor -- start 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 1 -- start start 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient: init 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe9381327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe93809eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe938139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe93809ef30 con 0x7fe9381327c0 2024-09-16T04:09:01.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe93809d880 con 0x7fe93809eb20 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe93809db60 con 0x7fe938139ea0 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe9406c9640 10 monclient(hunting): authenticate will time out at 916.683896s 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe93dc64640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe93809eb20 0x7fe938138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60032/0 (socket says 172.21.15.171:60032) 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe93dc64640 1 -- 172.21.15.171:0/2072641632 learned_addr learned my addr 172.21.15.171:0/2072641632 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.178+0000 7fe93e465640 1 -- 172.21.15.171:0/2072641632 >> v1:172.21.15.148:6789/0 conn(0x7fe9381327c0 legacy=0x7fe938132ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692741320 0 0) 0x7fe93809d880 con 0x7fe93809eb20 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): my global_id is 5056 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe924003520 con 0x7fe93809eb20 2024-09-16T04:09:01.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2178314764 0 0) 0x7fe93809db60 con 0x7fe938139ea0 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient(hunting): my global_id is 5087 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe93809d880 con 0x7fe938139ea0 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3519647561 0 0) 0x7fe924003520 con 0x7fe93809eb20 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 >> v1:172.21.15.148:6790/0 conn(0x7fe938139ea0 legacy=0x7fe93813c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 >> v1:172.21.15.148:6789/0 conn(0x7fe9381327c0 legacy=0x7fe938132ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: _finish_hunting 0 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 monclient: found mon.noname-c 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93809de40 con 0x7fe93809eb20 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: _finish_auth 0 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: _check_auth_tickets 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe934001590 con 0x7fe93809eb20 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: dump: 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.190 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe9406c9640 5 monclient: authenticate success, global_id 5056 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe9406c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9340017f0 con 0x7fe93809eb20 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 1 -- 172.21.15.171:0/2072641632 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe934001b40 con 0x7fe93809eb20 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe9406c9640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe9406c9640 4 set_mon_vals no callback set 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:01.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.179+0000 7fe93d463640 10 monclient: dump: 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 10 monclient: shutdown 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:01.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 1 -- 172.21.15.171:0/2072641632 >> v1:172.21.15.171:6789/0 conn(0x7fe93809eb20 legacy=0x7fe938138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 1 -- 172.21.15.171:0/2072641632 shutdown_connections 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 1 -- 172.21.15.171:0/2072641632 >> 172.21.15.171:0/2072641632 conn(0x7fe9380a5370 msgr2=0x7fe9380a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 1 -- 172.21.15.171:0/2072641632 shutdown_connections 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 1 -- 172.21.15.171:0/2072641632 wait complete. 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 10 monclient: build_initial_monmap 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 10 monclient: monmap: 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.180+0000 7fe9406c9640 1 Processor -- start 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 1 -- start start 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient: init 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe93809eb20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe9381327c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe938139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe938071c00 con 0x7fe93809eb20 2024-09-16T04:09:01.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe938073040 con 0x7fe9381327c0 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe938072760 con 0x7fe938139ea0 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe9406c9640 10 monclient(hunting): authenticate will time out at 916.686484s 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe93e465640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe93809eb20 legacy=0x7fe9381cbbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe93dc64640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe9381327c0 0x7fe9381d0780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60034/0 (socket says 172.21.15.171:60034) 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe93dc64640 1 -- 172.21.15.171:0/3358073837 learned_addr learned my addr 172.21.15.171:0/3358073837 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546992259 0 0) 0x7fe938073040 con 0x7fe9381327c0 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): my global_id is 5059 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe91c003520 con 0x7fe9381327c0 2024-09-16T04:09:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719718232 0 0) 0x7fe938072760 con 0x7fe938139ea0 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): my global_id is 5090 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe938073040 con 0x7fe938139ea0 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 591654087 0 0) 0x7fe91c003520 con 0x7fe9381327c0 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe938072760 con 0x7fe9381327c0 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9340015d0 con 0x7fe9381327c0 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): dump: 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.181+0000 7fe92effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4089777564 0 0) 0x7fe938072760 con 0x7fe9381327c0 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 >> v1:172.21.15.148:6790/0 conn(0x7fe938139ea0 legacy=0x7fe9381d3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 >> v1:172.21.15.148:6789/0 conn(0x7fe93809eb20 legacy=0x7fe9381cbbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 monclient: found mon.b 2024-09-16T04:09:01.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93811ff90 con 0x7fe9381327c0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 5 monclient: authenticate success, global_id 5059 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 10 monclient: _renew_subs 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe9381cc2d0 con 0x7fe9381327c0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 10 monclient: _renew_subs 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9381cb6b0 con 0x7fe9381327c0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9340017f0 con 0x7fe9381327c0 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:01.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe934004c80 con 0x7fe9381327c0 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe93cc62640 4 set_mon_vals no callback set 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 10 monclient: dump: 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.199 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe934004ee0 con 0x7fe9381327c0 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.182+0000 7fe9406c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.183+0000 7fe9406c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.183+0000 7fe9406c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.183+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe920005080 con 0x7fe9381327c0 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.183+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 5307+0+0 (unknown 1283167528 0 0) 0x7fe934047410 con 0x7fe9381327c0 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.183+0000 7fe93e465640 10 monclient: get_auth_request con 0x7fe91c038f10 auth_method 0 2024-09-16T04:09:01.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.187+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe934015d40 con 0x7fe9381327c0 2024-09-16T04:09:01.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.187+0000 7fe92effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:01.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.187+0000 7fe92effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:01.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe9406c9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe9406c9640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe9406c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7fe920005b80 con 0x7fe9381327c0 2024-09-16T04:09:01.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v73) ==== 126+0+0 (unknown 3177338128 0 0) 0x7fe934019990 con 0x7fe9381327c0 2024-09-16T04:09:01.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe92effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.359+0000 7fe92effd640 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-16T04:09:01.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.460+0000 7fe9406c9640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.460+0000 7fe9406c9640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.460+0000 7fe9406c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.460+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) -- 0x7fe920001090 con 0x7fe9381327c0 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.461+0000 7fe92effd640 1 -- 172.21.15.171:0/3358073837 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v73) ==== 126+0+0 (unknown 3177338128 0 0) 0x7fe93401e250 con 0x7fe9381327c0 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.461+0000 7fe92effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.461+0000 7fe92effd640 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-16T04:09:01.463 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'bar' already exists 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.462+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 >> v1:172.21.15.171:6812/7327273 conn(0x7fe91c038f10 legacy=0x7fe91c03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.462+0000 7fe9406c9640 10 monclient: shutdown 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.462+0000 7fe9406c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.462+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 >> v1:172.21.15.171:6789/0 conn(0x7fe9381327c0 legacy=0x7fe9381d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.462+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 shutdown_connections 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.462+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 >> 172.21.15.171:0/3358073837 conn(0x7fe9380a5370 msgr2=0x7fe9380a35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.463+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 shutdown_connections 2024-09-16T04:09:01.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.463+0000 7fe9406c9640 1 -- 172.21.15.171:0/3358073837 wait complete. 2024-09-16T04:09:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T04:09:01.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.550+0000 7fa452442640 10 monclient: get_monmap_and_config 2024-09-16T04:09:01.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.550+0000 7fa452442640 10 monclient: build_initial_monmap 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.550+0000 7fa452442640 10 monclient: monmap: 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:01.551614+0000 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:01.551614+0000 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:01.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.551+0000 7fa452442640 1 Processor -- start 2024-09-16T04:09:01.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.551+0000 7fa452442640 1 -- start start 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.551+0000 7fa452442640 10 monclient: init 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient(hunting): picked mon.noname-b con 0x7fa44c132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient(hunting): picked mon.noname-a con 0x7fa44c09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient(hunting): picked mon.noname-c con 0x7fa44c132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c1332b0 con 0x7fa44c09ec20 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c134310 con 0x7fa44c132e60 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c09d980 con 0x7fa44c132a80 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:01.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient(hunting): authenticate will time out at 917.057222s 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa4509df640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa44c132e60 0x7fa44c13c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60046/0 (socket says 172.21.15.171:60046) 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa4509df640 1 -- 172.21.15.171:0/1211051553 learned_addr learned my addr 172.21.15.171:0/1211051553 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44b7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa44c09ec20 legacy=0x7fa44c133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2301071834 0 0) 0x7fa44c134310 con 0x7fa44c132e60 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient(hunting): my global_id is 5062 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa434003520 con 0x7fa44c132e60 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 202986523 0 0) 0x7fa434003520 con 0x7fa44c132e60 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 >> v1:172.21.15.148:6790/0 conn(0x7fa44c132a80 legacy=0x7fa44c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 >> v1:172.21.15.148:6789/0 conn(0x7fa44c09ec20 legacy=0x7fa44c133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:01.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 monclient: found mon.noname-c 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c09dc60 con 0x7fa44c132e60 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: _finish_auth 0 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa444001990 con 0x7fa44c132e60 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 5 monclient: authenticate success, global_id 5062 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:01.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: dump: 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa444001c80 con 0x7fa44c132e60 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 1 -- 172.21.15.171:0/1211051553 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa444002090 con 0x7fa44c132e60 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa452442640 4 set_mon_vals no callback set 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.552+0000 7fa44affd640 10 monclient: dump: 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 10 monclient: shutdown 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 1 -- 172.21.15.171:0/1211051553 >> v1:172.21.15.171:6789/0 conn(0x7fa44c132e60 legacy=0x7fa44c13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 1 -- 172.21.15.171:0/1211051553 shutdown_connections 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 1 -- 172.21.15.171:0/1211051553 >> 172.21.15.171:0/1211051553 conn(0x7fa44c0a52a0 msgr2=0x7fa44c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 1 -- 172.21.15.171:0/1211051553 shutdown_connections 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 1 -- 172.21.15.171:0/1211051553 wait complete. 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.553+0000 7fa452442640 10 monclient: build_initial_monmap 2024-09-16T04:09:01.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient: monmap: 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 1 Processor -- start 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 1 -- start start 2024-09-16T04:09:01.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient: init 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient(hunting): picked mon.noname-a con 0x7fa44c09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient(hunting): picked mon.noname-b con 0x7fa44c132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 10 monclient(hunting): picked mon.noname-c con 0x7fa44c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.554+0000 7fa452442640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c0a3120 con 0x7fa44c09ec20 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c06ceb0 con 0x7fa44c132a80 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c11f680 con 0x7fa44c132e60 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 10 monclient(hunting): authenticate will time out at 917.060044s 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa44bfff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa44c09ec20 legacy=0x7fa44c130cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa44b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa44c132a80 0x7fa44c1ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60060/0 (socket says 172.21.15.171:60060) 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa44b7fe640 1 -- 172.21.15.171:0/2408045901 learned_addr learned my addr 172.21.15.171:0/2408045901 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:01.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530635968 0 0) 0x7fa44c06ceb0 con 0x7fa44c132a80 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): my global_id is 5065 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa438003520 con 0x7fa44c132a80 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3127480290 0 0) 0x7fa44c11f680 con 0x7fa44c132e60 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): my global_id is 5096 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa44c06ceb0 con 0x7fa44c132e60 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3956694545 0 0) 0x7fa438003520 con 0x7fa44c132a80 2024-09-16T04:09:01.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa44c11f680 con 0x7fa44c132a80 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa440004150 con 0x7fa44c132a80 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.569 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2348422370 0 0) 0x7fa44c11f680 con 0x7fa44c132a80 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 >> v1:172.21.15.148:6790/0 conn(0x7fa44c132e60 legacy=0x7fa44c1d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 >> v1:172.21.15.148:6789/0 conn(0x7fa44c09ec20 legacy=0x7fa44c130cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 monclient: found mon.b 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c11ede0 con 0x7fa44c132a80 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:01.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 5 monclient: authenticate success, global_id 5065 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 10 monclient: _renew_subs 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa44c11f0c0 con 0x7fa44c132a80 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 10 monclient: _renew_subs 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa44c0726d0 con 0x7fa44c132a80 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4400043b0 con 0x7fa44c132a80 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa440004700 con 0x7fa44c132a80 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:01.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa44a7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.555+0000 7fa448ff9640 10 monclient: dump: 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa4400048a0 con 0x7fa44c132a80 2024-09-16T04:09:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa452442640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa452442640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa452442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c005080 con 0x7fa44c132a80 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 5307+0+0 (unknown 1283167528 0 0) 0x7fa440047300 con 0x7fa44c132a80 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.556+0000 7fa44bfff640 10 monclient: get_auth_request con 0x7fa438039000 auth_method 0 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.561+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa4400197c0 con 0x7fa44c132a80 2024-09-16T04:09:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.561+0000 7fa448ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.561+0000 7fa448ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:01.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.729+0000 7fa452442640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:01.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.729+0000 7fa452442640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:01.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.729+0000 7fa452442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:01.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:01.729+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7fa42c005b80 con 0x7fa44c132a80 2024-09-16T04:09:02.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.236+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v74) ==== 168+0+0 (unknown 1982334279 0 0) 0x7fa44001e080 con 0x7fa44c132a80 2024-09-16T04:09:02.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.236+0000 7fa448ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.236+0000 7fa448ff9640 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T04:09:02.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa452442640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa452442640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa452442640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7fa42c001090 con 0x7fa44c132a80 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa448ff9640 1 -- 172.21.15.171:0/2408045901 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v74) ==== 177+0+0 (unknown 4068446194 0 0) 0x7fa4400137f0 con 0x7fa44c132a80 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa448ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.340+0000 7fa448ff9640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T04:09:02.343 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.342+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 >> v1:172.21.15.171:6812/7327273 conn(0x7fa438039000 legacy=0x7fa43803b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.342+0000 7fa452442640 10 monclient: shutdown 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.342+0000 7fa452442640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.342+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 >> v1:172.21.15.171:6789/0 conn(0x7fa44c132a80 legacy=0x7fa44c1ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.343+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 shutdown_connections 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.343+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 >> 172.21.15.171:0/2408045901 conn(0x7fa44c0a52a0 msgr2=0x7fa44c131c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.343+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 shutdown_connections 2024-09-16T04:09:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.343+0000 7fa452442640 1 -- 172.21.15.171:0/2408045901 wait complete. 2024-09-16T04:09:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush link host1 root=bar 2024-09-16T04:09:02.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.432+0000 7f412d978640 10 monclient: get_monmap_and_config 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.432+0000 7f412d978640 10 monclient: build_initial_monmap 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.432+0000 7f412d978640 10 monclient: monmap: 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:02.433013+0000 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:02.433013+0000 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:02.434 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.435 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:02.435 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:02.435 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:02.435 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.432+0000 7f412d978640 1 Processor -- start 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.432+0000 7f412d978640 1 -- start start 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.432+0000 7f412d978640 10 monclient: init 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient(hunting): picked mon.noname-c con 0x7f4128136dd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient(hunting): picked mon.noname-a con 0x7f41281371b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:02.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient(hunting): picked mon.noname-b con 0x7f41281375b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f412809af40 con 0x7f41281371b0 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f412809b220 con 0x7f4128136dd0 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f412809aa20 con 0x7f41281375b0 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 10 monclient(hunting): authenticate will time out at 917.938454s 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4126ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4128136dd0 0x7f412809bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60074/0 (socket says 172.21.15.171:60074) 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4126ffd640 1 -- 172.21.15.171:0/2987193964 learned_addr learned my addr 172.21.15.171:0/2987193964 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f41267fc640 1 -- 172.21.15.171:0/2987193964 >> v1:172.21.15.148:6789/0 conn(0x7f41281371b0 legacy=0x7f412813bbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2050458905 0 0) 0x7f412809b220 con 0x7f4128136dd0 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient(hunting): my global_id is 5068 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4110003520 con 0x7f4128136dd0 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4244621349 0 0) 0x7f4110003520 con 0x7f4128136dd0 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 >> v1:172.21.15.148:6790/0 conn(0x7f41281375b0 legacy=0x7f412813f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 >> v1:172.21.15.148:6789/0 conn(0x7f41281371b0 legacy=0x7f412813bbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 monclient: found mon.noname-c 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41280a4d30 con 0x7f4128136dd0 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 5 monclient: authenticate success, global_id 5068 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4118001990 con 0x7f4128136dd0 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f412d978640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.433+0000 7f4125ffb640 10 monclient: dump: 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.445 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4118001c70 con 0x7f4128136dd0 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f4125ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:02.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f4125ffb640 1 -- 172.21.15.171:0/2987193964 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4118002040 con 0x7f4128136dd0 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 4 set_mon_vals no callback set 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f4125ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f4125ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f4125ffb640 10 monclient: dump: 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.447 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 10 monclient: shutdown 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 1 -- 172.21.15.171:0/2987193964 >> v1:172.21.15.171:6789/0 conn(0x7f4128136dd0 legacy=0x7f412809bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 1 -- 172.21.15.171:0/2987193964 shutdown_connections 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 1 -- 172.21.15.171:0/2987193964 >> 172.21.15.171:0/2987193964 conn(0x7f41280a9860 msgr2=0x7f41280a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 1 -- 172.21.15.171:0/2987193964 shutdown_connections 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.434+0000 7f412d978640 1 -- 172.21.15.171:0/2987193964 wait complete. 2024-09-16T04:09:02.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.435+0000 7f412d978640 10 monclient: build_initial_monmap 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.435+0000 7f412d978640 10 monclient: monmap: 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.435+0000 7f412d978640 1 Processor -- start 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.435+0000 7f412d978640 1 -- start start 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.435+0000 7f412d978640 10 monclient: init 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient(hunting): picked mon.noname-a con 0x7f4128136dd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient(hunting): picked mon.noname-b con 0x7f41281371b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient(hunting): picked mon.noname-c con 0x7f41281375b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4128130390 con 0x7f4128136dd0 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41280a0340 con 0x7f41281371b0 2024-09-16T04:09:02.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f4126ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4128136dd0 legacy=0x7f41281cb430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f41267fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f41281371b0 0x7f41281d0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60078/0 (socket says 172.21.15.171:60078) 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4128135b50 con 0x7f41281375b0 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f41267fc640 1 -- 172.21.15.171:0/2873421668 learned_addr learned my addr 172.21.15.171:0/2873421668 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient(hunting): authenticate will time out at 917.941253s 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688401121 0 0) 0x7f41280a0340 con 0x7f41281371b0 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): my global_id is 5071 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4114003520 con 0x7f41281371b0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 282206228 0 0) 0x7f4128135b50 con 0x7f41281375b0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): my global_id is 5102 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41280a0340 con 0x7f41281375b0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3630030437 0 0) 0x7f4114003520 con 0x7f41281371b0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4128135b50 con 0x7f41281371b0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f411c0041d0 con 0x7f41281371b0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:02.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): dump: 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:02.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1835665606 0 0) 0x7f4128135b50 con 0x7f41281371b0 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 >> v1:172.21.15.148:6790/0 conn(0x7f41281375b0 legacy=0x7f41281d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 >> v1:172.21.15.148:6789/0 conn(0x7f4128136dd0 legacy=0x7f41281cb430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 monclient: found mon.b 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4128131280 con 0x7f41281371b0 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: _finish_auth 0 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 5 monclient: authenticate success, global_id 5071 2024-09-16T04:09:02.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient: _renew_subs 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41281305d0 con 0x7f41281371b0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient: _renew_subs 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f41281cbb40 con 0x7f41281371b0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f411c004480 con 0x7f41281371b0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f411c004850 con 0x7f41281371b0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f41257fa640 4 set_mon_vals no callback set 2024-09-16T04:09:02.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.436+0000 7f410ffff640 10 monclient: dump: 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.456 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f411c0049f0 con 0x7f41281371b0 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f412d978640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f412d978640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f412d978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4108005080 con 0x7f41281371b0 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(74..74 src has 1..74) ==== 5433+0+0 (unknown 1656608742 0 0) 0x7f411c0474e0 con 0x7f41281371b0 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.437+0000 7f4126ffd640 10 monclient: get_auth_request con 0x7f4114038fb0 auth_method 0 2024-09-16T04:09:02.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.442+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f411c015cb0 con 0x7f41281371b0 2024-09-16T04:09:02.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.442+0000 7f410ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:02.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.442+0000 7f410ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:02.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.613+0000 7f412d978640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.613+0000 7f412d978640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.613+0000 7f412d978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.613+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7f4108005b80 con 0x7f41281371b0 2024-09-16T04:09:02.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.614+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v74) ==== 177+0+0 (unknown 4068446194 0 0) 0x7f411c0155b0 con 0x7f41281371b0 2024-09-16T04:09:02.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.614+0000 7f410ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.614+0000 7f410ffff640 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.716+0000 7f412d978640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.717+0000 7f412d978640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.717+0000 7f412d978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.717+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) -- 0x7f4108005760 con 0x7f41281371b0 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.717+0000 7f410ffff640 1 -- 172.21.15.171:0/2873421668 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v74) ==== 177+0+0 (unknown 4068446194 0 0) 0x7f411c019900 con 0x7f41281371b0 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.717+0000 7f410ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.717+0000 7f410ffff640 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T04:09:02.719 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 >> v1:172.21.15.171:6812/7327273 conn(0x7f4114038fb0 legacy=0x7f411403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 10 monclient: shutdown 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 >> v1:172.21.15.171:6789/0 conn(0x7f41281371b0 legacy=0x7f41281d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 shutdown_connections 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 >> 172.21.15.171:0/2873421668 conn(0x7f41280a9860 msgr2=0x7f41280a7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 shutdown_connections 2024-09-16T04:09:02.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.719+0000 7f412d978640 1 -- 172.21.15.171:0/2873421668 wait complete. 2024-09-16T04:09:02.730 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T04:09:02.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.808+0000 7f1495c94640 10 monclient: get_monmap_and_config 2024-09-16T04:09:02.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.808+0000 7f1495c94640 10 monclient: build_initial_monmap 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.808+0000 7f1495c94640 10 monclient: monmap: 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:02.809559+0000 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:02.809559+0000 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:02.811 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.809+0000 7f1495c94640 1 Processor -- start 2024-09-16T04:09:02.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.809+0000 7f1495c94640 1 -- start start 2024-09-16T04:09:02.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.809+0000 7f1495c94640 10 monclient: init 2024-09-16T04:09:02.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:02.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient(hunting): picked mon.noname-b con 0x7f149009f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient(hunting): picked mon.noname-a con 0x7f14900a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient(hunting): picked mon.noname-c con 0x7f149009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14900a1260 con 0x7f14900a0130 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149009d0c0 con 0x7f149009e1d0 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149009d3a0 con 0x7f149009f640 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient(hunting): authenticate will time out at 918.315192s 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148effd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f14900a0130 legacy=0x7f1490138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f149009e1d0 0x7f149013bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60094/0 (socket says 172.21.15.171:60094) 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148ffff640 1 -- 172.21.15.171:0/3209622790 learned_addr learned my addr 172.21.15.171:0/3209622790 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781510389 0 0) 0x7f149009d0c0 con 0x7f149009e1d0 2024-09-16T04:09:02.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): my global_id is 5074 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1478003520 con 0x7f149009e1d0 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4076059968 0 0) 0x7f149009d3a0 con 0x7f149009f640 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient(hunting): my global_id is 5105 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f149009d0c0 con 0x7f149009f640 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2285664450 0 0) 0x7f1478003520 con 0x7f149009e1d0 2024-09-16T04:09:02.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 >> v1:172.21.15.148:6790/0 conn(0x7f149009f640 legacy=0x7f149009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 >> v1:172.21.15.148:6789/0 conn(0x7f14900a0130 legacy=0x7f1490138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149009d680 con 0x7f149009e1d0 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1488001990 con 0x7f149009e1d0 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 5 monclient: authenticate success, global_id 5074 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:02.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: dump: 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1488001c40 con 0x7f149009e1d0 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:02.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 1 -- 172.21.15.171:0/3209622790 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1488002010 con 0x7f149009e1d0 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f1495c94640 4 set_mon_vals no callback set 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.810+0000 7f148e7fc640 10 monclient: dump: 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.824 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 10 monclient: shutdown 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 1 -- 172.21.15.171:0/3209622790 >> v1:172.21.15.171:6789/0 conn(0x7f149009e1d0 legacy=0x7f149013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 1 -- 172.21.15.171:0/3209622790 shutdown_connections 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 1 -- 172.21.15.171:0/3209622790 >> 172.21.15.171:0/3209622790 conn(0x7f14900a52a0 msgr2=0x7f14900a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 1 -- 172.21.15.171:0/3209622790 shutdown_connections 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.811+0000 7f1495c94640 1 -- 172.21.15.171:0/3209622790 wait complete. 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 10 monclient: build_initial_monmap 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 10 monclient: monmap: 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:02.825 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 1 Processor -- start 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 1 -- start start 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 10 monclient: init 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.812+0000 7f1495c94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient(hunting): picked mon.noname-b con 0x7f149009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient(hunting): picked mon.noname-c con 0x7f149009f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient(hunting): picked mon.noname-a con 0x7f14900a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14901a7620 con 0x7f14900a0130 2024-09-16T04:09:02.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14901a94b0 con 0x7f149009e1d0 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1490072130 con 0x7f149009f640 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient(hunting): authenticate will time out at 918.318124s 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f148ffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f14900a0130 legacy=0x7f14901d41a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f148f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f149009e1d0 0x7f14901cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60100/0 (socket says 172.21.15.171:60100) 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f148f7fe640 1 -- 172.21.15.171:0/205418645 learned_addr learned my addr 172.21.15.171:0/205418645 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581093033 0 0) 0x7f14901a94b0 con 0x7f149009e1d0 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): my global_id is 5077 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f147c003520 con 0x7f149009e1d0 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420702580 0 0) 0x7f1490072130 con 0x7f149009f640 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): my global_id is 5108 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14901a94b0 con 0x7f149009f640 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 250202380 0 0) 0x7f147c003520 con 0x7f149009e1d0 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1490072130 con 0x7f149009e1d0 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1480001970 con 0x7f149009e1d0 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): dump: 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 993219027 0 0) 0x7f14901a94b0 con 0x7f149009f640 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f147c003520 con 0x7f149009f640 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1484004140 con 0x7f149009f640 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient(hunting): dump: 2024-09-16T04:09:02.829 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 860632852 0 0) 0x7f1490072130 con 0x7f149009e1d0 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 >> v1:172.21.15.148:6790/0 conn(0x7f149009f640 legacy=0x7f14901d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:02.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 >> v1:172.21.15.148:6789/0 conn(0x7f14900a0130 legacy=0x7f14901d41a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 monclient: found mon.b 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14900a4080 con 0x7f149009e1d0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 5 monclient: authenticate success, global_id 5077 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient: _renew_subs 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f149006ceb0 con 0x7f149009e1d0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient: _renew_subs 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f14901a91d0 con 0x7f149009e1d0 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1480001d20 con 0x7f149009e1d0 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1480003a60 con 0x7f149009e1d0 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f148dffb640 4 set_mon_vals no callback set 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.813+0000 7f1477fff640 10 monclient: dump: 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:02.832 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.814+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1480003c00 con 0x7f149009e1d0 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.814+0000 7f1495c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.814+0000 7f1495c94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.814+0000 7f1495c94640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.814+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1470005080 con 0x7f149009e1d0 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.815+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(74..74 src has 1..74) ==== 5433+0+0 (unknown 1656608742 0 0) 0x7f1480047460 con 0x7f149009e1d0 2024-09-16T04:09:02.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.815+0000 7f148effd640 10 monclient: get_auth_request con 0x7f147c038f40 auth_method 0 2024-09-16T04:09:02.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.819+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1480015c50 con 0x7f149009e1d0 2024-09-16T04:09:02.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.819+0000 7f1477fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:02.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.819+0000 7f1477fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:02.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.989+0000 7f1495c94640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:02.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.989+0000 7f1495c94640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:02.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.989+0000 7f1495c94640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:02.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:02.989+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7f1470005b80 con 0x7f149009e1d0 2024-09-16T04:09:03.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.237+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v75) ==== 168+0+0 (unknown 2209502296 0 0) 0x7f1480015550 con 0x7f149009e1d0 2024-09-16T04:09:03.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.237+0000 7f1477fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.237+0000 7f1477fff640 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1495c94640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1495c94640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1495c94640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7f1470001090 con 0x7f149009e1d0 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1477fff640 1 -- 172.21.15.171:0/205418645 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v75) ==== 177+0+0 (unknown 547720402 0 0) 0x7f14800198a0 con 0x7f149009e1d0 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1477fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.344+0000 7f1477fff640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T04:09:03.346 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 >> v1:172.21.15.171:6812/7327273 conn(0x7f147c038f40 legacy=0x7f147c03b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 10 monclient: shutdown 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 >> v1:172.21.15.171:6789/0 conn(0x7f149009e1d0 legacy=0x7f14901cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 shutdown_connections 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 >> 172.21.15.171:0/205418645 conn(0x7f14900a52a0 msgr2=0x7f14900a2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 shutdown_connections 2024-09-16T04:09:03.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.346+0000 7f1495c94640 1 -- 172.21.15.171:0/205418645 wait complete. 2024-09-16T04:09:03.357 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush link host2 root=bar 2024-09-16T04:09:03.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.434+0000 7fd2446e6640 10 monclient: get_monmap_and_config 2024-09-16T04:09:03.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.434+0000 7fd2446e6640 10 monclient: build_initial_monmap 2024-09-16T04:09:03.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.434+0000 7fd2446e6640 10 monclient: monmap: 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:03.435611+0000 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:03.435611+0000 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:03.437 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:03.438 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.435+0000 7fd2446e6640 1 Processor -- start 2024-09-16T04:09:03.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.435+0000 7fd2446e6640 1 -- start start 2024-09-16T04:09:03.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.435+0000 7fd2446e6640 10 monclient: init 2024-09-16T04:09:03.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:03.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:03.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd23c09f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd23c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd23c09e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c0a1260 con 0x7fd23c09e1d0 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c09d0c0 con 0x7fd23c09f640 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c09d3a0 con 0x7fd23c0a0130 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient(hunting): authenticate will time out at 918.941190s 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd242c83640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd23c09e1d0 legacy=0x7fd23c13bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd242482640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd23c09f640 0x7fd23c09fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60104/0 (socket says 172.21.15.171:60104) 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd242482640 1 -- 172.21.15.171:0/443168532 learned_addr learned my addr 172.21.15.171:0/443168532 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641973291 0 0) 0x7fd23c09d0c0 con 0x7fd23c09f640 2024-09-16T04:09:03.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): my global_id is 5080 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd228003440 con 0x7fd23c09f640 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875782738 0 0) 0x7fd23c09d3a0 con 0x7fd23c0a0130 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient(hunting): my global_id is 5111 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd23c09d0c0 con 0x7fd23c0a0130 2024-09-16T04:09:03.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 195107840 0 0) 0x7fd228003440 con 0x7fd23c09f640 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 >> v1:172.21.15.148:6790/0 conn(0x7fd23c0a0130 legacy=0x7fd23c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 >> v1:172.21.15.148:6789/0 conn(0x7fd23c09e1d0 legacy=0x7fd23c13bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: _finish_hunting 0 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 monclient: found mon.noname-c 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c09d680 con 0x7fd23c09f640 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: _finish_auth 0 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: _check_auth_tickets 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd238001610 con 0x7fd23c09f640 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 5 monclient: authenticate success, global_id 5080 2024-09-16T04:09:03.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: dump: 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.451 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2380018c0 con 0x7fd23c09f640 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 1 -- 172.21.15.171:0/443168532 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd238001c90 con 0x7fd23c09f640 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd2446e6640 4 set_mon_vals no callback set 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.436+0000 7fd241480640 10 monclient: dump: 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.452 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 10 monclient: shutdown 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 1 -- 172.21.15.171:0/443168532 >> v1:172.21.15.171:6789/0 conn(0x7fd23c09f640 legacy=0x7fd23c09fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 1 -- 172.21.15.171:0/443168532 shutdown_connections 2024-09-16T04:09:03.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 1 -- 172.21.15.171:0/443168532 >> 172.21.15.171:0/443168532 conn(0x7fd23c0a52a0 msgr2=0x7fd23c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 1 -- 172.21.15.171:0/443168532 shutdown_connections 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.437+0000 7fd2446e6640 1 -- 172.21.15.171:0/443168532 wait complete. 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient: build_initial_monmap 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient: monmap: 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:03.454 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 1 Processor -- start 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 1 -- start start 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient: init 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd23c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd23c09f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd23c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c1c7cf0 con 0x7fd23c09f640 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c1a4d60 con 0x7fd23c09e1d0 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd23c072150 con 0x7fd23c0a0130 2024-09-16T04:09:03.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd2446e6640 10 monclient(hunting): authenticate will time out at 918.943946s 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd241c81640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd23c09f640 legacy=0x7fd23c1cc110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd242482640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd23c09e1d0 0x7fd23c1b7680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60108/0 (socket says 172.21.15.171:60108) 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.438+0000 7fd242482640 1 -- 172.21.15.171:0/2403727071 learned_addr learned my addr 172.21.15.171:0/2403727071 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966062408 0 0) 0x7fd23c1a4d60 con 0x7fd23c09e1d0 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): my global_id is 5083 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd220003520 con 0x7fd23c09e1d0 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3080652285 0 0) 0x7fd220003520 con 0x7fd23c09e1d0 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd23c1a4d60 con 0x7fd23c09e1d0 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd238001c10 con 0x7fd23c09e1d0 2024-09-16T04:09:03.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:03.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845471820 0 0) 0x7fd23c072150 con 0x7fd23c0a0130 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient(hunting): my global_id is 5114 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd220003520 con 0x7fd23c0a0130 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2738714382 0 0) 0x7fd23c1a4d60 con 0x7fd23c09e1d0 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 >> v1:172.21.15.148:6790/0 conn(0x7fd23c0a0130 legacy=0x7fd23c1cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 >> v1:172.21.15.148:6789/0 conn(0x7fd23c09f640 legacy=0x7fd23c1cc110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 monclient: found mon.b 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c0a4120 con 0x7fd23c09e1d0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 5 monclient: authenticate success, global_id 5083 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 10 monclient: _renew_subs 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd23c06ceb0 con 0x7fd23c09e1d0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 10 monclient: _renew_subs 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd23c1a3630 con 0x7fd23c09e1d0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd2380018c0 con 0x7fd23c09e1d0 2024-09-16T04:09:03.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd238004eb0 con 0x7fd23c09e1d0 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd240c7f640 4 set_mon_vals no callback set 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.439+0000 7fd232ffd640 10 monclient: dump: 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.460 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd2380050d0 con 0x7fd23c09e1d0 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd2446e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd2446e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd2446e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd224005080 con 0x7fd23c09e1d0 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7fd238047670 con 0x7fd23c09e1d0 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.440+0000 7fd241c81640 10 monclient: get_auth_request con 0x7fd220038f20 auth_method 0 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.445+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd238015fc0 con 0x7fd23c09e1d0 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.445+0000 7fd232ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:03.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.445+0000 7fd232ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:03.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd2446e6640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd2446e6640 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd2446e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7fd224005b80 con 0x7fd23c09e1d0 2024-09-16T04:09:03.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v75) ==== 177+0+0 (unknown 547720402 0 0) 0x7fd2380158c0 con 0x7fd23c09e1d0 2024-09-16T04:09:03.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd232ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.614+0000 7fd232ffd640 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T04:09:03.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd2446e6640 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd2446e6640 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd2446e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) -- 0x7fd224001090 con 0x7fd23c09e1d0 2024-09-16T04:09:03.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd232ffd640 1 -- 172.21.15.171:0/2403727071 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v75) ==== 177+0+0 (unknown 547720402 0 0) 0x7fd238013c20 con 0x7fd23c09e1d0 2024-09-16T04:09:03.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd232ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-16T04:09:03.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.716+0000 7fd232ffd640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T04:09:03.719 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-16T04:09:03.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 >> v1:172.21.15.171:6812/7327273 conn(0x7fd220038f20 legacy=0x7fd22003b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 10 monclient: shutdown 2024-09-16T04:09:03.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:03.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 >> v1:172.21.15.171:6789/0 conn(0x7fd23c09e1d0 legacy=0x7fd23c1b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 shutdown_connections 2024-09-16T04:09:03.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 >> 172.21.15.171:0/2403727071 conn(0x7fd23c0a52a0 msgr2=0x7fd23c0a2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:03.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 shutdown_connections 2024-09-16T04:09:03.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.718+0000 7fd2446e6640 1 -- 172.21.15.171:0/2403727071 wait complete. 2024-09-16T04:09:03.729 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:03.729 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c osd.7 2024-09-16T04:09:03.730 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 2 2024-09-16T04:09:03.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.803+0000 7f2d7e8b7640 10 monclient: get_monmap_and_config 2024-09-16T04:09:03.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.803+0000 7f2d7e8b7640 10 monclient: build_initial_monmap 2024-09-16T04:09:03.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.803+0000 7f2d7e8b7640 10 monclient: monmap: 2024-09-16T04:09:03.806 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:03.806 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:03.804650+0000 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:03.804650+0000 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:03.807 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:03.808 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.804+0000 7f2d7e8b7640 1 Processor -- start 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.804+0000 7f2d7e8b7640 1 -- start start 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.804+0000 7f2d7e8b7640 10 monclient: init 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient(hunting): picked mon.noname-c con 0x7f2d7809ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:03.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient(hunting): picked mon.noname-a con 0x7f2d78073570 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient(hunting): picked mon.noname-b con 0x7f2d7809dcb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d780a0460 con 0x7f2d78073570 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d780a0db0 con 0x7f2d7809ba30 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d780a11b0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient(hunting): authenticate will time out at 919.310253s 2024-09-16T04:09:03.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d77fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2d7809ba30 0x7f2d7809f640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60112/0 (socket says 172.21.15.171:60112) 2024-09-16T04:09:03.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d77fff640 1 -- 172.21.15.171:0/2585290301 learned_addr learned my addr 172.21.15.171:0/2585290301 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:03.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d777fe640 1 -- 172.21.15.171:0/2585290301 >> v1:172.21.15.148:6789/0 conn(0x7f2d78073570 legacy=0x7f2d7809fd50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246153442 0 0) 0x7f2d780a0db0 con 0x7f2d7809ba30 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 5086 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d60003520 con 0x7f2d7809ba30 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220513228 0 0) 0x7f2d780a11b0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient(hunting): my global_id is 5117 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d780a0db0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3225456377 0 0) 0x7f2d60003520 con 0x7f2d7809ba30 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 >> v1:172.21.15.148:6790/0 conn(0x7f2d7809dcb0 legacy=0x7f2d7813b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 >> v1:172.21.15.148:6789/0 conn(0x7f2d78073570 legacy=0x7f2d7809fd50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7809cff0 con 0x7f2d7809ba30 2024-09-16T04:09:03.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 5 monclient: authenticate success, global_id 5086 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2d68001610 con 0x7f2d7809ba30 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: dump: 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.821 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d68001930 con 0x7f2d7809ba30 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 1 -- 172.21.15.171:0/2585290301 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2d68001d00 con 0x7f2d7809ba30 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d7e8b7640 4 set_mon_vals no callback set 2024-09-16T04:09:03.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.805+0000 7f2d76ffd640 10 monclient: dump: 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 10 monclient: shutdown 2024-09-16T04:09:03.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2585290301 >> v1:172.21.15.171:6789/0 conn(0x7f2d7809ba30 legacy=0x7f2d7809f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2585290301 shutdown_connections 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2585290301 >> 172.21.15.171:0/2585290301 conn(0x7f2d780a52a0 msgr2=0x7f2d780a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2585290301 shutdown_connections 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.806+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2585290301 wait complete. 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 10 monclient: build_initial_monmap 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 10 monclient: monmap: 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:03.824 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 1 Processor -- start 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 1 -- start start 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 10 monclient: init 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.807+0000 7f2d7e8b7640 10 monclient(hunting): picked mon.noname-a con 0x7f2d78073570 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient(hunting): picked mon.noname-c con 0x7f2d7809ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:03.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient(hunting): picked mon.noname-b con 0x7f2d7809dcb0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d781a51d0 con 0x7f2d78073570 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d781b7f20 con 0x7f2d7809dcb0 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d780a4330 con 0x7f2d7809ba30 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient(hunting): authenticate will time out at 919.313059s 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d77fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2d78073570 legacy=0x7f2d781b7810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7ce54640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2d7809dcb0 0x7f2d781cf7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60114/0 (socket says 172.21.15.171:60114) 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7ce54640 1 -- 172.21.15.171:0/2885334965 learned_addr learned my addr 172.21.15.171:0/2885334965 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:03.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767402443 0 0) 0x7f2d781b7f20 con 0x7f2d7809dcb0 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): my global_id is 5089 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d64003520 con 0x7f2d7809dcb0 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3192164309 0 0) 0x7f2d780a4330 con 0x7f2d7809ba30 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): my global_id is 5120 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d781b7f20 con 0x7f2d7809ba30 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 906541408 0 0) 0x7f2d64003520 con 0x7f2d7809dcb0 2024-09-16T04:09:03.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d780a4330 con 0x7f2d7809dcb0 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2d70001c30 con 0x7f2d7809dcb0 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.828 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:03.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2829014220 0 0) 0x7f2d781b7f20 con 0x7f2d7809ba30 2024-09-16T04:09:03.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d64003520 con 0x7f2d7809ba30 2024-09-16T04:09:03.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2d6c004130 con 0x7f2d7809ba30 2024-09-16T04:09:03.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:03.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.830 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1931643491 0 0) 0x7f2d780a4330 con 0x7f2d7809dcb0 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 >> v1:172.21.15.148:6790/0 conn(0x7f2d7809ba30 legacy=0x7f2d781cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 >> v1:172.21.15.148:6789/0 conn(0x7f2d78073570 legacy=0x7f2d781b7810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 monclient: found mon.b 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d781a2ee0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 5 monclient: authenticate success, global_id 5089 2024-09-16T04:09:03.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient: _renew_subs 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d781a4ef0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient: _renew_subs 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d781a3a40 con 0x7f2d7809dcb0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d70003990 con 0x7f2d7809dcb0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2d70003d60 con 0x7f2d7809dcb0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d767fc640 4 set_mon_vals no callback set 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:03.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.808+0000 7f2d74ff9640 10 monclient: dump: 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2d70003fc0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d7e8b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:03.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d7e8b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d7e8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d58005080 con 0x7f2d7809dcb0 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7f2d70047860 con 0x7f2d7809dcb0 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.809+0000 7f2d77fff640 10 monclient: get_auth_request con 0x7f2d64038ea0 auth_method 0 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.814+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2d70015fd0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.814+0000 7f2d74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:03.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.814+0000 7f2d74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:03.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.979+0000 7f2d7e8b7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:03.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.979+0000 7f2d7e8b7640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:03.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.979+0000 7f2d7e8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:03.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.979+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f2d58005b80 con 0x7f2d7809dcb0 2024-09-16T04:09:03.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.980+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7f2d700158d0 con 0x7f2d7809dcb0 2024-09-16T04:09:03.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.980+0000 7f2d74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:03.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:03.980+0000 7f2d74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:04.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.079+0000 7f2d7e8b7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:04.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.079+0000 7f2d7e8b7640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.079+0000 7f2d7e8b7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.079+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f2d58006730 con 0x7f2d7809dcb0 2024-09-16T04:09:04.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.080+0000 7f2d74ff9640 1 -- 172.21.15.171:0/2885334965 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7f2d70019c20 con 0x7f2d7809dcb0 2024-09-16T04:09:04.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.080+0000 7f2d74ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.080+0000 7f2d74ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 >> v1:172.21.15.171:6812/7327273 conn(0x7f2d64038ea0 legacy=0x7f2d6403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 10 monclient: shutdown 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 >> v1:172.21.15.171:6789/0 conn(0x7f2d7809dcb0 legacy=0x7f2d781cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 shutdown_connections 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 >> 172.21.15.171:0/2885334965 conn(0x7f2d780a52a0 msgr2=0x7f2d78072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 shutdown_connections 2024-09-16T04:09:04.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.082+0000 7f2d7e8b7640 1 -- 172.21.15.171:0/2885334965 wait complete. 2024-09-16T04:09:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c host1 2024-09-16T04:09:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 2 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.167+0000 7ff2e62d7640 10 monclient: get_monmap_and_config 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.167+0000 7ff2e62d7640 10 monclient: build_initial_monmap 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.167+0000 7ff2e62d7640 10 monclient: monmap: 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:04.168294+0000 2024-09-16T04:09:04.169 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:04.168294+0000 2024-09-16T04:09:04.170 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:04.170 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.170 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:04.170 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:04.170 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:04.170 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 1 Processor -- start 2024-09-16T04:09:04.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 1 -- start start 2024-09-16T04:09:04.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient: init 2024-09-16T04:09:04.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient(hunting): picked mon.noname-a con 0x7ff2e0130930 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient(hunting): picked mon.noname-c con 0x7ff2e009eb60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e0130d10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e01343a0 con 0x7ff2e0130930 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e009c1b0 con 0x7ff2e009eb60 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e009d8e0 con 0x7ff2e0130d10 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:04.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2e62d7640 10 monclient(hunting): authenticate will time out at 919.673778s 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2df7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff2e009eb60 0x7ff2e0133c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60116/0 (socket says 172.21.15.171:60116) 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2df7fe640 1 -- 172.21.15.171:0/1474829896 learned_addr learned my addr 172.21.15.171:0/1474829896 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.168+0000 7ff2dffff640 1 -- 172.21.15.171:0/1474829896 >> v1:172.21.15.148:6789/0 conn(0x7ff2e0130930 legacy=0x7ff2e0133580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352771119 0 0) 0x7ff2e009c1b0 con 0x7ff2e009eb60 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient(hunting): my global_id is 5092 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2cc003520 con 0x7ff2e009eb60 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2867602248 0 0) 0x7ff2cc003520 con 0x7ff2e009eb60 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 >> v1:172.21.15.148:6790/0 conn(0x7ff2e0130d10 legacy=0x7ff2e013c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 >> v1:172.21.15.148:6789/0 conn(0x7ff2e0130930 legacy=0x7ff2e0133580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:04.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e009dbc0 con 0x7ff2e009eb60 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 5 monclient: authenticate success, global_id 5092 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d0001610 con 0x7ff2e009eb60 2024-09-16T04:09:04.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: dump: 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2d0001900 con 0x7ff2e009eb60 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 1 -- 172.21.15.171:0/1474829896 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d0001d10 con 0x7ff2e009eb60 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 4 set_mon_vals no callback set 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2deffd640 10 monclient: dump: 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.181 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 10 monclient: shutdown 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 1 -- 172.21.15.171:0/1474829896 >> v1:172.21.15.171:6789/0 conn(0x7ff2e009eb60 legacy=0x7ff2e0133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 1 -- 172.21.15.171:0/1474829896 shutdown_connections 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.169+0000 7ff2e62d7640 1 -- 172.21.15.171:0/1474829896 >> 172.21.15.171:0/1474829896 conn(0x7ff2e00a52a0 msgr2=0x7ff2e00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.170+0000 7ff2e62d7640 1 -- 172.21.15.171:0/1474829896 shutdown_connections 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.170+0000 7ff2e62d7640 1 -- 172.21.15.171:0/1474829896 wait complete. 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.170+0000 7ff2e62d7640 10 monclient: build_initial_monmap 2024-09-16T04:09:04.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.170+0000 7ff2e62d7640 10 monclient: monmap: 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.170+0000 7ff2e62d7640 1 Processor -- start 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 1 -- start start 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient: init 2024-09-16T04:09:04.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e009eb60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient(hunting): picked mon.noname-c con 0x7ff2e0130930 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient(hunting): picked mon.noname-a con 0x7ff2e0130d10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e012f3e0 con 0x7ff2e0130d10 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e0122eb0 con 0x7ff2e009eb60 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e0123470 con 0x7ff2e0130930 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e62d7640 10 monclient(hunting): authenticate will time out at 919.676466s 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2e4874640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff2e0130d10 legacy=0x7ff2e01d4020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff2e009eb60 0x7ff2e01cb710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60130/0 (socket says 172.21.15.171:60130) 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dffff640 1 -- 172.21.15.171:0/2515718772 learned_addr learned my addr 172.21.15.171:0/2515718772 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:04.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214863747 0 0) 0x7ff2e0122eb0 con 0x7ff2e009eb60 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): my global_id is 5095 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c0003520 con 0x7ff2e009eb60 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3849521749 0 0) 0x7ff2e0123470 con 0x7ff2e0130930 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): my global_id is 5126 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2e0122eb0 con 0x7ff2e0130930 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2793261690 0 0) 0x7ff2c0003520 con 0x7ff2e009eb60 2024-09-16T04:09:04.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2e0123470 con 0x7ff2e009eb60 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2c8004180 con 0x7ff2e009eb60 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): dump: 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.186 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.171+0000 7ff2dcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3710321723 0 0) 0x7ff2e0123470 con 0x7ff2e009eb60 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 >> v1:172.21.15.148:6790/0 conn(0x7ff2e0130930 legacy=0x7ff2e01d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 >> v1:172.21.15.148:6789/0 conn(0x7ff2e0130d10 legacy=0x7ff2e01d4020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 monclient: found mon.b 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e01206b0 con 0x7ff2e009eb60 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:04.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 5 monclient: authenticate success, global_id 5095 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 10 monclient: _renew_subs 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e01209a0 con 0x7ff2e009eb60 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 10 monclient: _renew_subs 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2e01cc000 con 0x7ff2e009eb60 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2c80044a0 con 0x7ff2e009eb60 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2c8004870 con 0x7ff2e009eb60 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 10 monclient: dump: 2024-09-16T04:09:04.188 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2de7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.172+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff2c8004ad0 con 0x7ff2e009eb60 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.173+0000 7ff2e62d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.173+0000 7ff2e62d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.173+0000 7ff2e62d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.173+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7ff2c8047490 con 0x7ff2e009eb60 2024-09-16T04:09:04.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.173+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e012fdf0 con 0x7ff2e009eb60 2024-09-16T04:09:04.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.173+0000 7ff2df7fe640 10 monclient: get_auth_request con 0x7ff2c0038f60 auth_method 0 2024-09-16T04:09:04.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.177+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff2c8015d80 con 0x7ff2e009eb60 2024-09-16T04:09:04.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.177+0000 7ff2dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.177+0000 7ff2dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:04.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2e62d7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:04.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2e62d7640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2e62d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7ff2e009f110 con 0x7ff2e009eb60 2024-09-16T04:09:04.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7ff2c8015680 con 0x7ff2e009eb60 2024-09-16T04:09:04.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.346+0000 7ff2dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:04.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.444+0000 7ff2e62d7640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:04.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.444+0000 7ff2e62d7640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.444+0000 7ff2e62d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.444+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7ff2e0121110 con 0x7ff2e009eb60 2024-09-16T04:09:04.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.445+0000 7ff2dcff9640 1 -- 172.21.15.171:0/2515718772 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7ff2c801e230 con 0x7ff2e009eb60 2024-09-16T04:09:04.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.445+0000 7ff2dcff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.445+0000 7ff2dcff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:04.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 >> v1:172.21.15.171:6812/7327273 conn(0x7ff2c0038f60 legacy=0x7ff2c003b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 10 monclient: shutdown 2024-09-16T04:09:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 >> v1:172.21.15.171:6789/0 conn(0x7ff2e009eb60 legacy=0x7ff2e01cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 shutdown_connections 2024-09-16T04:09:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 >> 172.21.15.171:0/2515718772 conn(0x7ff2e00a52a0 msgr2=0x7ff2e009cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 shutdown_connections 2024-09-16T04:09:04.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.447+0000 7ff2e62d7640 1 -- 172.21.15.171:0/2515718772 wait complete. 2024-09-16T04:09:04.458 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:04.459 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c osd.9 2024-09-16T04:09:04.459 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 2 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.533+0000 7f96e7017640 10 monclient: get_monmap_and_config 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.533+0000 7f96e7017640 10 monclient: build_initial_monmap 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.533+0000 7f96e7017640 10 monclient: monmap: 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:04.534008+0000 2024-09-16T04:09:04.535 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:04.534008+0000 2024-09-16T04:09:04.536 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:04.536 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.536 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:04.536 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:04.536 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:04.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.533+0000 7f96e7017640 1 Processor -- start 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 1 -- start start 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient: init 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient(hunting): picked mon.noname-c con 0x7f96e0136cf0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient(hunting): picked mon.noname-a con 0x7f96e0139ad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient(hunting): picked mon.noname-b con 0x7f96e013d5e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e00a4f90 con 0x7f96e0139ad0 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e00a0a30 con 0x7f96e0136cf0 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e00a0d10 con 0x7f96e013d5e0 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e7017640 10 monclient(hunting): authenticate will time out at 920.039495s 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96dffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f96e0139ad0 legacy=0x7f96e013bec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e4db3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f96e0136cf0 0x7f96e01370d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60140/0 (socket says 172.21.15.171:60140) 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96e4db3640 1 -- 172.21.15.171:0/2428319472 learned_addr learned my addr 172.21.15.171:0/2428319472 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955603498 0 0) 0x7f96e00a0a30 con 0x7f96e0136cf0 2024-09-16T04:09:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96df7fe640 10 monclient(hunting): my global_id is 5098 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.534+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96c8003520 con 0x7f96e0136cf0 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2819695340 0 0) 0x7f96c8003520 con 0x7f96e0136cf0 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 >> v1:172.21.15.148:6790/0 conn(0x7f96e013d5e0 legacy=0x7f96e013f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 >> v1:172.21.15.148:6789/0 conn(0x7f96e0139ad0 legacy=0x7f96e013bec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e00a0ff0 con 0x7f96e0136cf0 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96d0001990 con 0x7f96e0136cf0 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 5 monclient: authenticate success, global_id 5098 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: dump: 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f96d0001c00 con 0x7f96e0136cf0 2024-09-16T04:09:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 1 -- 172.21.15.171:0/2428319472 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96d0002010 con 0x7f96e0136cf0 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 4 set_mon_vals no callback set 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96df7fe640 10 monclient: dump: 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 10 monclient: shutdown 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 1 -- 172.21.15.171:0/2428319472 >> v1:172.21.15.171:6789/0 conn(0x7f96e0136cf0 legacy=0x7f96e01370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 1 -- 172.21.15.171:0/2428319472 shutdown_connections 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 1 -- 172.21.15.171:0/2428319472 >> 172.21.15.171:0/2428319472 conn(0x7f96e00a9830 msgr2=0x7f96e00a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 1 -- 172.21.15.171:0/2428319472 shutdown_connections 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.535+0000 7f96e7017640 1 -- 172.21.15.171:0/2428319472 wait complete. 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.536+0000 7f96e7017640 10 monclient: build_initial_monmap 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.536+0000 7f96e7017640 10 monclient: monmap: 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:04.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.536+0000 7f96e7017640 1 Processor -- start 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.536+0000 7f96e7017640 1 -- start start 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.536+0000 7f96e7017640 10 monclient: init 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient(hunting): picked mon.noname-b con 0x7f96e0136cf0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient(hunting): picked mon.noname-c con 0x7f96e0139ad0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient(hunting): picked mon.noname-a con 0x7f96e013d5e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e009ce70 con 0x7f96e013d5e0 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e00a01d0 con 0x7f96e0136cf0 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e009cb90 con 0x7f96e0139ad0 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient(hunting): authenticate will time out at 920.042334s 2024-09-16T04:09:04.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e4db3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f96e0136cf0 0x7f96e013b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60142/0 (socket says 172.21.15.171:60142) 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e4db3640 1 -- 172.21.15.171:0/3792922130 learned_addr learned my addr 172.21.15.171:0/3792922130 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e55b4640 1 -- 172.21.15.171:0/3792922130 >> v1:172.21.15.148:6789/0 conn(0x7f96e013d5e0 legacy=0x7f96e013c210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 441742099 0 0) 0x7f96e00a01d0 con 0x7f96e0136cf0 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): my global_id is 5101 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96cc003520 con 0x7f96e0136cf0 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692315519 0 0) 0x7f96e009cb90 con 0x7f96e0139ad0 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): my global_id is 5132 2024-09-16T04:09:04.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96e00a01d0 con 0x7f96e0139ad0 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 177779585 0 0) 0x7f96cc003520 con 0x7f96e0136cf0 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96e009cb90 con 0x7f96e0136cf0 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96d0002010 con 0x7f96e0136cf0 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.556 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3170334775 0 0) 0x7f96e009cb90 con 0x7f96e0136cf0 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 >> v1:172.21.15.148:6790/0 conn(0x7f96e0139ad0 legacy=0x7f96e013bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 >> v1:172.21.15.148:6789/0 conn(0x7f96e013d5e0 legacy=0x7f96e013c210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 monclient: found mon.b 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e009b460 con 0x7f96e0136cf0 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 5 monclient: authenticate success, global_id 5101 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient: _renew_subs 2024-09-16T04:09:04.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.537+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f96e013ce50 con 0x7f96e0136cf0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96e7017640 10 monclient: _renew_subs 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96e7017640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f96e01d49a0 con 0x7f96e0136cf0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f96d0004f70 con 0x7f96e0136cf0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f96d0001c00 con 0x7f96e0136cf0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 10 monclient: dump: 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.559 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96deffd640 4 set_mon_vals no callback set 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.538+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f96d0005340 con 0x7f96e0136cf0 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.539+0000 7f96e7017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.539+0000 7f96e7017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.539+0000 7f96e7017640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.539+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c0005080 con 0x7f96e0136cf0 2024-09-16T04:09:04.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.539+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7f96d0047920 con 0x7f96e0136cf0 2024-09-16T04:09:04.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.539+0000 7f96dffff640 10 monclient: get_auth_request con 0x7f96cc039100 auth_method 0 2024-09-16T04:09:04.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.543+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f96d001e5e0 con 0x7f96e0136cf0 2024-09-16T04:09:04.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.543+0000 7f96dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.543+0000 7f96dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.711+0000 7f96e7017640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.711+0000 7f96e7017640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.711+0000 7f96e7017640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.711+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f96c0005b80 con 0x7f96e0136cf0 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.712+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7f96d0045010 con 0x7f96e0136cf0 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.712+0000 7f96dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.712+0000 7f96dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96e7017640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96e7017640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96e7017640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f96c0005d50 con 0x7f96e0136cf0 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96dd7fa640 1 -- 172.21.15.171:0/3792922130 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7f96d0045390 con 0x7f96e0136cf0 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96dd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:04.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.811+0000 7f96dd7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 >> v1:172.21.15.171:6812/7327273 conn(0x7f96cc039100 legacy=0x7f96cc03b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 10 monclient: shutdown 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 >> v1:172.21.15.171:6789/0 conn(0x7f96e0136cf0 legacy=0x7f96e013b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 shutdown_connections 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 >> 172.21.15.171:0/3792922130 conn(0x7f96e00a9830 msgr2=0x7f96e00a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 shutdown_connections 2024-09-16T04:09:04.815 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.813+0000 7f96e7017640 1 -- 172.21.15.171:0/3792922130 wait complete. 2024-09-16T04:09:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c host2 2024-09-16T04:09:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 2 2024-09-16T04:09:04.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.899+0000 7fc6d3b91640 10 monclient: get_monmap_and_config 2024-09-16T04:09:04.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.899+0000 7fc6d3b91640 10 monclient: build_initial_monmap 2024-09-16T04:09:04.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.899+0000 7fc6d3b91640 10 monclient: monmap: 2024-09-16T04:09:04.901 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:04.900449+0000 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:04.900449+0000 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:04.902 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 1 Processor -- start 2024-09-16T04:09:04.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 1 -- start start 2024-09-16T04:09:04.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient: init 2024-09-16T04:09:04.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:04.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:04.909 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient(hunting): picked mon.noname-a con 0x7fc6cc133970 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient(hunting): picked mon.noname-c con 0x7fc6cc133d50 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient(hunting): picked mon.noname-b con 0x7fc6cc134160 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc09c670 con 0x7fc6cc133970 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc09c950 con 0x7fc6cc133d50 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc09cc30 con 0x7fc6cc134160 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.900+0000 7fc6d3b91640 10 monclient(hunting): authenticate will time out at 920.405911s 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d112c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc6cc133d50 0x7fc6cc138840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60146/0 (socket says 172.21.15.171:60146) 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d112c640 1 -- 172.21.15.171:0/4232518818 learned_addr learned my addr 172.21.15.171:0/4232518818 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d192d640 1 -- 172.21.15.171:0/4232518818 >> v1:172.21.15.148:6789/0 conn(0x7fc6cc133970 legacy=0x7fc6cc09db50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775167515 0 0) 0x7fc6cc09c950 con 0x7fc6cc133d50 2024-09-16T04:09:04.910 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): my global_id is 5104 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b8003520 con 0x7fc6cc133d50 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114709325 0 0) 0x7fc6cc09cc30 con 0x7fc6cc134160 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient(hunting): my global_id is 5135 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6cc09c950 con 0x7fc6cc134160 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4142704495 0 0) 0x7fc6b8003520 con 0x7fc6cc133d50 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 >> v1:172.21.15.148:6790/0 conn(0x7fc6cc134160 legacy=0x7fc6cc13bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 >> v1:172.21.15.148:6789/0 conn(0x7fc6cc133970 legacy=0x7fc6cc09db50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: _finish_hunting 0 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 monclient: found mon.noname-c 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc09b310 con 0x7fc6cc133d50 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: _finish_auth 0 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: _check_auth_tickets 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6bc001610 con 0x7fc6cc133d50 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d3b91640 5 monclient: authenticate success, global_id 5104 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d3b91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.912 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: dump: 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d3b91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:04.913 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6bc001930 con 0x7fc6cc133d50 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 1 -- 172.21.15.171:0/4232518818 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6bc001d00 con 0x7fc6cc133d50 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d3b91640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d3b91640 4 set_mon_vals no callback set 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.901+0000 7fc6d092b640 10 monclient: dump: 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.914 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 10 monclient: shutdown 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 1 -- 172.21.15.171:0/4232518818 >> v1:172.21.15.171:6789/0 conn(0x7fc6cc133d50 legacy=0x7fc6cc138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 1 -- 172.21.15.171:0/4232518818 shutdown_connections 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 1 -- 172.21.15.171:0/4232518818 >> 172.21.15.171:0/4232518818 conn(0x7fc6cc0a63c0 msgr2=0x7fc6cc0a2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 1 -- 172.21.15.171:0/4232518818 shutdown_connections 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 1 -- 172.21.15.171:0/4232518818 wait complete. 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 10 monclient: build_initial_monmap 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.902+0000 7fc6d3b91640 10 monclient: monmap: 2024-09-16T04:09:04.915 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 1 Processor -- start 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 1 -- start start 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient: init 2024-09-16T04:09:04.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient(hunting): picked mon.noname-a con 0x7fc6cc133970 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient(hunting): picked mon.noname-b con 0x7fc6cc133d50 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient(hunting): picked mon.noname-c con 0x7fc6cc134160 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc11fa60 con 0x7fc6cc133970 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc1218f0 con 0x7fc6cc133d50 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc09f540 con 0x7fc6cc134160 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d3b91640 10 monclient(hunting): authenticate will time out at 920.408682s 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d112c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc6cc133d50 0x7fc6cc1cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60148/0 (socket says 172.21.15.171:60148) 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d112c640 1 -- 172.21.15.171:0/3648020560 learned_addr learned my addr 172.21.15.171:0/3648020560 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6d192d640 1 -- 172.21.15.171:0/3648020560 >> v1:172.21.15.148:6789/0 conn(0x7fc6cc133970 legacy=0x7fc6cc1c7670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344925402 0 0) 0x7fc6cc1218f0 con 0x7fc6cc133d50 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6c27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6c27fc640 10 monclient(hunting): my global_id is 5107 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.903+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b0003520 con 0x7fc6cc133d50 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3775140303 0 0) 0x7fc6b0003520 con 0x7fc6cc133d50 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6cc1218f0 con 0x7fc6cc133d50 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6bc0015d0 con 0x7fc6cc133d50 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:04.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): dump: 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151934056 0 0) 0x7fc6cc09f540 con 0x7fc6cc134160 2024-09-16T04:09:04.919 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient(hunting): my global_id is 5138 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b0003520 con 0x7fc6cc134160 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4138493261 0 0) 0x7fc6cc1218f0 con 0x7fc6cc133d50 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 >> v1:172.21.15.148:6790/0 conn(0x7fc6cc134160 legacy=0x7fc6cc1cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 >> v1:172.21.15.148:6789/0 conn(0x7fc6cc133970 legacy=0x7fc6cc1c7670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 monclient: found mon.b 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc0a1dd0 con 0x7fc6cc133d50 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 5 monclient: authenticate success, global_id 5107 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 10 monclient: _renew_subs 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc6cc09d0c0 con 0x7fc6cc133d50 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 10 monclient: _renew_subs 2024-09-16T04:09:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc6cc121610 con 0x7fc6cc133d50 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6bc004db0 con 0x7fc6cc133d50 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6bc005100 con 0x7fc6cc133d50 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c27fc640 10 monclient: dump: 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:04.922 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.904+0000 7fc6c3fff640 4 set_mon_vals no callback set 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc6bc0052a0 con 0x7fc6cc133d50 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6d3b91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6d3b91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6d3b91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6cc0008d0 con 0x7fc6cc133d50 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7fc6bc047650 con 0x7fc6cc133d50 2024-09-16T04:09:04.923 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.905+0000 7fc6d192d640 10 monclient: get_auth_request con 0x7fc6b0038fe0 auth_method 0 2024-09-16T04:09:04.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.909+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc6bc015f20 con 0x7fc6cc133d50 2024-09-16T04:09:04.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.909+0000 7fc6c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:04.924 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:04.909+0000 7fc6c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:05.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6d3b91640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:05.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6d3b91640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:05.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6d3b91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fc6cc0a31e0 con 0x7fc6cc133d50 2024-09-16T04:09:05.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7fc6bc015820 con 0x7fc6cc133d50 2024-09-16T04:09:05.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:05.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.075+0000 7fc6c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6d3b91640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6d3b91640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6d3b91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fc6cc11e970 con 0x7fc6cc133d50 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6c27fc640 1 -- 172.21.15.171:0/3648020560 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v75) ==== 56+0+1474 (unknown 3476242654 0 171730953) 0x7fc6bc044ce0 con 0x7fc6cc133d50 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6c27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:05.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.175+0000 7fc6c27fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:05.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 >> v1:172.21.15.171:6812/7327273 conn(0x7fc6b0038fe0 legacy=0x7fc6b003b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 10 monclient: shutdown 2024-09-16T04:09:05.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:05.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 >> v1:172.21.15.171:6789/0 conn(0x7fc6cc133d50 legacy=0x7fc6cc1cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 shutdown_connections 2024-09-16T04:09:05.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 >> 172.21.15.171:0/3648020560 conn(0x7fc6cc0a63c0 msgr2=0x7fc6cc0a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:05.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 shutdown_connections 2024-09-16T04:09:05.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.177+0000 7fc6d3b91640 1 -- 172.21.15.171:0/3648020560 wait complete. 2024-09-16T04:09:05.188 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-16T04:09:05.188 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:05.189 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm host1 foo 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.266+0000 7f3ede02e640 10 monclient: get_monmap_and_config 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.266+0000 7f3ede02e640 10 monclient: build_initial_monmap 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.266+0000 7f3ede02e640 10 monclient: monmap: 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:05.267233+0000 2024-09-16T04:09:05.268 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:05.267233+0000 2024-09-16T04:09:05.269 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:05.269 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.269 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:05.269 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:05.269 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:05.269 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 1 Processor -- start 2024-09-16T04:09:05.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 1 -- start start 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient: init 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed80685d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed809bfc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed8136e10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed8099c00 con 0x7f3ed809bfc0 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed81373c0 con 0x7f3ed80685d0 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed809a150 con 0x7f3ed8136e10 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ede02e640 10 monclient(hunting): authenticate will time out at 920.772724s 2024-09-16T04:09:05.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ed77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3ed80685d0 0x7f3ed809b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60154/0 (socket says 172.21.15.171:60154) 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ed77fe640 1 -- 172.21.15.171:0/2516464067 learned_addr learned my addr 172.21.15.171:0/2516464067 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.267+0000 7f3ed6ffd640 1 -- 172.21.15.171:0/2516464067 >> v1:172.21.15.148:6789/0 conn(0x7f3ed809bfc0 legacy=0x7f3ed813bc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367235539 0 0) 0x7f3ed81373c0 con 0x7f3ed80685d0 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient(hunting): my global_id is 5110 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ec4003520 con 0x7f3ed80685d0 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2257264730 0 0) 0x7f3ec4003520 con 0x7f3ed80685d0 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 >> v1:172.21.15.148:6790/0 conn(0x7f3ed8136e10 legacy=0x7f3ed813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 >> v1:172.21.15.148:6789/0 conn(0x7f3ed809bfc0 legacy=0x7f3ed813bc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:05.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed8143aa0 con 0x7f3ed80685d0 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3ec8001610 con 0x7f3ed80685d0 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: dump: 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.279 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 5 monclient: authenticate success, global_id 5110 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ec8001970 con 0x7f3ed80685d0 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 1 -- 172.21.15.171:0/2516464067 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3ec80048a0 con 0x7f3ed80685d0 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:05.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 4 set_mon_vals no callback set 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ed67fc640 10 monclient: dump: 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.281 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 10 monclient: shutdown 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 1 -- 172.21.15.171:0/2516464067 >> v1:172.21.15.171:6789/0 conn(0x7f3ed80685d0 legacy=0x7f3ed809b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 1 -- 172.21.15.171:0/2516464067 shutdown_connections 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 1 -- 172.21.15.171:0/2516464067 >> 172.21.15.171:0/2516464067 conn(0x7f3ed80a1e90 msgr2=0x7f3ed80a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 1 -- 172.21.15.171:0/2516464067 shutdown_connections 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.268+0000 7f3ede02e640 1 -- 172.21.15.171:0/2516464067 wait complete. 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.269+0000 7f3ede02e640 10 monclient: build_initial_monmap 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.269+0000 7f3ede02e640 10 monclient: monmap: 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:05.282 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.269+0000 7f3ede02e640 1 Processor -- start 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.269+0000 7f3ede02e640 1 -- start start 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.269+0000 7f3ede02e640 10 monclient: init 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:05.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed80685d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed809bfc0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed8136e10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed813c210 con 0x7f3ed8136e10 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed813ddc0 con 0x7f3ed80685d0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed80a0450 con 0x7f3ed809bfc0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ede02e640 10 monclient(hunting): authenticate will time out at 920.775458s 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ed77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3ed80685d0 0x7f3ed81cbc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60168/0 (socket says 172.21.15.171:60168) 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ed77fe640 1 -- 172.21.15.171:0/3348537267 learned_addr learned my addr 172.21.15.171:0/3348537267 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ed7fff640 1 -- 172.21.15.171:0/3348537267 >> v1:172.21.15.148:6789/0 conn(0x7f3ed8136e10 legacy=0x7f3ed81d4320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793901805 0 0) 0x7f3ed813ddc0 con 0x7f3ed80685d0 2024-09-16T04:09:05.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): my global_id is 5113 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3eb8003520 con 0x7f3ed80685d0 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2259739359 0 0) 0x7f3eb8003520 con 0x7f3ed80685d0 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ed813ddc0 con 0x7f3ed80685d0 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3ec8001430 con 0x7f3ed80685d0 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): dump: 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.285 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107081078 0 0) 0x7f3ed80a0450 con 0x7f3ed809bfc0 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 10 monclient(hunting): my global_id is 5144 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.270+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3eb8003520 con 0x7f3ed809bfc0 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 542317935 0 0) 0x7f3ed813ddc0 con 0x7f3ed80685d0 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 >> v1:172.21.15.148:6790/0 conn(0x7f3ed809bfc0 legacy=0x7f3ed81d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 >> v1:172.21.15.148:6789/0 conn(0x7f3ed8136e10 legacy=0x7f3ed81d4320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 monclient: found mon.b 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed809feb0 con 0x7f3ed80685d0 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: _finish_auth 0 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:05.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 5 monclient: authenticate success, global_id 5113 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 10 monclient: _renew_subs 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed809c580 con 0x7f3ed80685d0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 10 monclient: _renew_subs 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ede02e640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ed813dae0 con 0x7f3ed80685d0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ec8004dc0 con 0x7f3ed80685d0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3ec8005110 con 0x7f3ed80685d0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 10 monclient: dump: 2024-09-16T04:09:05.288 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ed5ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.271+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3ec8010540 con 0x7f3ed80685d0 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.272+0000 7f3ede02e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:05.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.272+0000 7f3ede02e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.272+0000 7f3ede02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.272+0000 7f3ede02e640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ebc005080 con 0x7f3ed80685d0 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.272+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7f3ec8047510 con 0x7f3ed80685d0 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.272+0000 7f3ed6ffd640 10 monclient: get_auth_request con 0x7f3eb8038fc0 auth_method 0 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.276+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3ec8015d80 con 0x7f3ed80685d0 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.276+0000 7f3ecffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:05.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.276+0000 7f3ecffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:05.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.447+0000 7f3ede02e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:05.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.447+0000 7f3ede02e640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:05.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.447+0000 7f3ede02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.447+0000 7f3ede02e640 1 -- 172.21.15.171:0/3348537267 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) -- 0x7f3ebc005b80 con 0x7f3ed80685d0 2024-09-16T04:09:05.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.448+0000 7f3ecffff640 1 -- 172.21.15.171:0/3348537267 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v75) ==== 120+0+0 (unknown 2543334075 0 0) 0x7f3ec8015680 con 0x7f3ed80685d0 2024-09-16T04:09:05.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.448+0000 7f3ecffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:05.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.448+0000 7f3ecffff640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T04:09:05.450 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T04:09:05.454 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:05.454 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T04:09:05.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.531+0000 7ff4b87f8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:05.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.531+0000 7ff4b87f8640 10 monclient: build_initial_monmap 2024-09-16T04:09:05.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.531+0000 7ff4b87f8640 10 monclient: monmap: 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:05.532568+0000 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:05.532568+0000 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:05.534 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.532+0000 7ff4b87f8640 1 Processor -- start 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.532+0000 7ff4b87f8640 1 -- start start 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.532+0000 7ff4b87f8640 10 monclient: init 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.532+0000 7ff4b87f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.532+0000 7ff4b87f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:05.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b01332b0 con 0x7ff4b0132a80 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b0134310 con 0x7ff4b009ec20 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b009d980 con 0x7ff4b0132e60 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 10 monclient(hunting): authenticate will time out at 921.038085s 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b6594640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff4b0132a80 legacy=0x7ff4b01334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5d93640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff4b009ec20 0x7ff4b0133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60174/0 (socket says 172.21.15.171:60174) 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5d93640 1 -- 172.21.15.171:0/933387347 learned_addr learned my addr 172.21.15.171:0/933387347 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465110307 0 0) 0x7ff4b0134310 con 0x7ff4b009ec20 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): my global_id is 5116 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff49c003520 con 0x7ff4b009ec20 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3695429881 0 0) 0x7ff4b009d980 con 0x7ff4b0132e60 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient(hunting): my global_id is 5147 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4b0134310 con 0x7ff4b0132e60 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1627999845 0 0) 0x7ff49c003520 con 0x7ff4b009ec20 2024-09-16T04:09:05.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 >> v1:172.21.15.148:6790/0 conn(0x7ff4b0132e60 legacy=0x7ff4b013c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 >> v1:172.21.15.148:6789/0 conn(0x7ff4b0132a80 legacy=0x7ff4b01334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: _finish_hunting 0 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 monclient: found mon.noname-c 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b009dc60 con 0x7ff4b009ec20 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: _finish_auth 0 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: _check_auth_tickets 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:05.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff4ac001610 con 0x7ff4b009ec20 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 5 monclient: authenticate success, global_id 5116 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: dump: 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.547 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4ac0018c0 con 0x7ff4b009ec20 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 1 -- 172.21.15.171:0/933387347 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff4ac001c90 con 0x7ff4b009ec20 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b87f8640 4 set_mon_vals no callback set 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:05.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.533+0000 7ff4b5592640 10 monclient: dump: 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 10 monclient: shutdown 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:05.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 1 -- 172.21.15.171:0/933387347 >> v1:172.21.15.171:6789/0 conn(0x7ff4b009ec20 legacy=0x7ff4b0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 1 -- 172.21.15.171:0/933387347 shutdown_connections 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 1 -- 172.21.15.171:0/933387347 >> 172.21.15.171:0/933387347 conn(0x7ff4b00a52a0 msgr2=0x7ff4b00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 1 -- 172.21.15.171:0/933387347 shutdown_connections 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 1 -- 172.21.15.171:0/933387347 wait complete. 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 10 monclient: build_initial_monmap 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.534+0000 7ff4b87f8640 10 monclient: monmap: 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:05.550 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:05.551 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:05.551 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:05.551 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:05.551 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.551 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:05.551 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:05.552 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:05.552 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 1 Processor -- start 2024-09-16T04:09:05.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 1 -- start start 2024-09-16T04:09:05.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient: init 2024-09-16T04:09:05.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b0132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b00a0460 con 0x7ff4b009ec20 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b011bf80 con 0x7ff4b0132a80 2024-09-16T04:09:05.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b0121b00 con 0x7ff4b0132e60 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b87f8640 10 monclient(hunting): authenticate will time out at 921.040663s 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b6594640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff4b009ec20 legacy=0x7ff4b009f640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b5d93640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff4b0132a80 0x7ff4b009fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60178/0 (socket says 172.21.15.171:60178) 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4b5d93640 1 -- 172.21.15.171:0/3946908423 learned_addr learned my addr 172.21.15.171:0/3946908423 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331254860 0 0) 0x7ff4b011bf80 con 0x7ff4b0132a80 2024-09-16T04:09:05.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4a6ffd640 10 monclient(hunting): my global_id is 5119 2024-09-16T04:09:05.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.535+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff494003520 con 0x7ff4b0132a80 2024-09-16T04:09:05.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 531798609 0 0) 0x7ff494003520 con 0x7ff4b0132a80 2024-09-16T04:09:05.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff4b011bf80 con 0x7ff4b0132a80 2024-09-16T04:09:05.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff4ac001c10 con 0x7ff4b0132a80 2024-09-16T04:09:05.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:05.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:05.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:05.556 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.556 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.557 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.557 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.557 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.559 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3394046588 0 0) 0x7ff4b0121b00 con 0x7ff4b0132e60 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:05.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient(hunting): my global_id is 5150 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff494003520 con 0x7ff4b0132e60 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 942618287 0 0) 0x7ff4b011bf80 con 0x7ff4b0132a80 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 >> v1:172.21.15.148:6790/0 conn(0x7ff4b0132e60 legacy=0x7ff4b01d3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 >> v1:172.21.15.148:6789/0 conn(0x7ff4b009ec20 legacy=0x7ff4b009f640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 monclient: found mon.b 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b011d150 con 0x7ff4b0132a80 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:05.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 5 monclient: authenticate success, global_id 5119 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 10 monclient: _renew_subs 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b011ce70 con 0x7ff4b0132a80 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 10 monclient: _renew_subs 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff4b011bce0 con 0x7ff4b0132a80 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff4ac0018c0 con 0x7ff4b0132a80 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff4ac004e60 con 0x7ff4b0132a80 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4b4d91640 4 set_mon_vals no callback set 2024-09-16T04:09:05.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.536+0000 7ff4a6ffd640 10 monclient: dump: 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:05.563 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:05.564 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:05.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff4ac005000 con 0x7ff4b0132a80 2024-09-16T04:09:05.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4b87f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:05.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4b87f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:05.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4b87f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff498005080 con 0x7ff4b0132a80 2024-09-16T04:09:05.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 5457+0+0 (unknown 2215622868 0 0) 0x7ff4ac047640 con 0x7ff4b0132a80 2024-09-16T04:09:05.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.537+0000 7ff4b6594640 10 monclient: get_auth_request con 0x7ff494038fc0 auth_method 0 2024-09-16T04:09:05.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.541+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff4ac019a40 con 0x7ff4b0132a80 2024-09-16T04:09:05.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.541+0000 7ff4a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:05.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.541+0000 7ff4a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.714+0000 7ff4b87f8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.714+0000 7ff4b87f8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.714+0000 7ff4b87f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:05.714+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7ff498005b80 con 0x7ff4b0132a80 2024-09-16T04:09:06.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.253+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v76) ==== 147+0+0 (unknown 3830613184 0 0) 0x7ff4ac01e300 con 0x7ff4b0132a80 2024-09-16T04:09:06.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.254+0000 7ff4a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.254+0000 7ff4a6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T04:09:06.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4b87f8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4b87f8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4b87f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7ff498001090 con 0x7ff4b0132a80 2024-09-16T04:09:06.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4a6ffd640 1 -- 172.21.15.171:0/3946908423 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v76) ==== 141+0+0 (unknown 1907081458 0 0) 0x7ff4ac044d30 con 0x7ff4b0132a80 2024-09-16T04:09:06.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4a6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.360+0000 7ff4a6ffd640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T04:09:06.363 INFO:tasks.workunit.client.0.smithi171.stderr:item -11 does not appear in that position 2024-09-16T04:09:06.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 >> v1:172.21.15.171:6812/7327273 conn(0x7ff494038fc0 legacy=0x7ff49403b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 10 monclient: shutdown 2024-09-16T04:09:06.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:06.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 >> v1:172.21.15.171:6789/0 conn(0x7ff4b0132a80 legacy=0x7ff4b009fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 shutdown_connections 2024-09-16T04:09:06.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 >> 172.21.15.171:0/3946908423 conn(0x7ff4b00a52a0 msgr2=0x7ff4b0133b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:06.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 shutdown_connections 2024-09-16T04:09:06.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.362+0000 7ff4b87f8640 1 -- 172.21.15.171:0/3946908423 wait complete. 2024-09-16T04:09:06.373 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush unlink host1 foo 2024-09-16T04:09:06.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.451+0000 7f40607d8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:06.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.451+0000 7f40607d8640 10 monclient: build_initial_monmap 2024-09-16T04:09:06.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.451+0000 7f40607d8640 10 monclient: monmap: 2024-09-16T04:09:06.453 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:06.453 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:06.452208+0000 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:06.452208+0000 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:06.454 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 1 Processor -- start 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 1 -- start start 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient: init 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient(hunting): picked mon.noname-c con 0x7f4058136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient(hunting): picked mon.noname-b con 0x7f4058139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient(hunting): picked mon.noname-a con 0x7f405813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40580a1340 con 0x7f405813d620 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4058137650 con 0x7f4058136b60 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40580a0aa0 con 0x7f4058139b10 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f40607d8640 10 monclient(hunting): authenticate will time out at 921.957824s 2024-09-16T04:09:06.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f405e574640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4058136b60 0x7f4058136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60186/0 (socket says 172.21.15.171:60186) 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f405e574640 1 -- 172.21.15.171:0/633485633 learned_addr learned my addr 172.21.15.171:0/633485633 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.452+0000 7f405ed75640 1 -- 172.21.15.171:0/633485633 >> v1:172.21.15.148:6789/0 conn(0x7f405813d620 legacy=0x7f405813fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951749792 0 0) 0x7f4058137650 con 0x7f4058136b60 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): my global_id is 5122 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4044003520 con 0x7f4058136b60 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439390879 0 0) 0x7f40580a0aa0 con 0x7f4058139b10 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient(hunting): my global_id is 5153 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4058137650 con 0x7f4058139b10 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3313176597 0 0) 0x7f4044003520 con 0x7f4058136b60 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 >> v1:172.21.15.148:6790/0 conn(0x7f4058139b10 legacy=0x7f405813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 >> v1:172.21.15.148:6789/0 conn(0x7f405813d620 legacy=0x7f405813fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: _finish_hunting 0 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 monclient: found mon.noname-c 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40580a0550 con 0x7f4058136b60 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: _finish_auth 0 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: _check_auth_tickets 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 5 monclient: authenticate success, global_id 5122 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4054001610 con 0x7f4058136b60 2024-09-16T04:09:06.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: dump: 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.465 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4054001930 con 0x7f4058136b60 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 1 -- 172.21.15.171:0/633485633 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4054001d00 con 0x7f4058136b60 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 4 set_mon_vals no callback set 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f405d572640 10 monclient: dump: 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.466 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 10 monclient: shutdown 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.453+0000 7f40607d8640 1 -- 172.21.15.171:0/633485633 >> v1:172.21.15.171:6789/0 conn(0x7f4058136b60 legacy=0x7f4058136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 1 -- 172.21.15.171:0/633485633 shutdown_connections 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 1 -- 172.21.15.171:0/633485633 >> 172.21.15.171:0/633485633 conn(0x7f40580a98b0 msgr2=0x7f40580a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 1 -- 172.21.15.171:0/633485633 shutdown_connections 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 1 -- 172.21.15.171:0/633485633 wait complete. 2024-09-16T04:09:06.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 10 monclient: build_initial_monmap 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 10 monclient: monmap: 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:06.468 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 1 Processor -- start 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.454+0000 7f40607d8640 1 -- start start 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient: init 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient(hunting): picked mon.noname-a con 0x7f4058136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient(hunting): picked mon.noname-c con 0x7f4058139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient(hunting): picked mon.noname-b con 0x7f405813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4058120990 con 0x7f4058136b60 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405809f4d0 con 0x7f405813d620 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40580a6f50 con 0x7f4058139b10 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:06.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f40607d8640 10 monclient(hunting): authenticate will time out at 921.960434s 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f405e574640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4058136b60 legacy=0x7f40581cba10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f405ed75640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f405813d620 0x7f40581d4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60192/0 (socket says 172.21.15.171:60192) 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f405ed75640 1 -- 172.21.15.171:0/1158894534 learned_addr learned my addr 172.21.15.171:0/1158894534 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971531217 0 0) 0x7f405809f4d0 con 0x7f405813d620 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): my global_id is 5125 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f403c003550 con 0x7f405813d620 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575840694 0 0) 0x7f40580a6f50 con 0x7f4058139b10 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): my global_id is 5156 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f405809f4d0 con 0x7f4058139b10 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 155730331 0 0) 0x7f403c003550 con 0x7f405813d620 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40580a6f50 con 0x7f405813d620 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4050004440 con 0x7f405813d620 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): dump: 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.471 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.455+0000 7f404effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1037840348 0 0) 0x7f405809f4d0 con 0x7f4058139b10 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f403c003550 con 0x7f4058139b10 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40480018b0 con 0x7f4058139b10 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient(hunting): dump: 2024-09-16T04:09:06.472 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2684307502 0 0) 0x7f40580a6f50 con 0x7f405813d620 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 >> v1:172.21.15.148:6790/0 conn(0x7f4058139b10 legacy=0x7f40581d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 >> v1:172.21.15.148:6789/0 conn(0x7f4058136b60 legacy=0x7f40581cba10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:06.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 monclient: found mon.b 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40580a8b00 con 0x7f405813d620 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 5 monclient: authenticate success, global_id 5125 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 10 monclient: _renew_subs 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40580a0240 con 0x7f405813d620 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 10 monclient: _renew_subs 2024-09-16T04:09:06.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40581206b0 con 0x7f405813d620 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4050004760 con 0x7f405813d620 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4050004ab0 con 0x7f405813d620 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f405cd71640 4 set_mon_vals no callback set 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 10 monclient: dump: 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.475 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4050004c50 con 0x7f405813d620 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.456+0000 7f40607d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.457+0000 7f40607d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.457+0000 7f40607d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.457+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4040005080 con 0x7f405813d620 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.457+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5449+0+0 (unknown 3719219916 0 0) 0x7f4050047720 con 0x7f405813d620 2024-09-16T04:09:06.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.457+0000 7f405e574640 10 monclient: get_auth_request con 0x7f403c038f40 auth_method 0 2024-09-16T04:09:06.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.461+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4050015f40 con 0x7f405813d620 2024-09-16T04:09:06.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.461+0000 7f404effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:06.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.461+0000 7f404effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:06.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.632+0000 7f40607d8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.632+0000 7f40607d8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.632+0000 7f40607d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.632+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7f4040005b80 con 0x7f405813d620 2024-09-16T04:09:06.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.633+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v76) ==== 141+0+0 (unknown 1907081458 0 0) 0x7f4050015840 con 0x7f405813d620 2024-09-16T04:09:06.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.633+0000 7f404effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.633+0000 7f404effd640 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-16T04:09:06.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.735+0000 7f40607d8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.735+0000 7f40607d8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.735+0000 7f40607d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.735+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) -- 0x7f4040001090 con 0x7f405813d620 2024-09-16T04:09:06.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.735+0000 7f404effd640 1 -- 172.21.15.171:0/1158894534 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v76) ==== 141+0+0 (unknown 1907081458 0 0) 0x7f405001e3f0 con 0x7f405813d620 2024-09-16T04:09:06.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.736+0000 7f404effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-16T04:09:06.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.736+0000 7f404effd640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T04:09:06.738 INFO:tasks.workunit.client.0.smithi171.stderr:item -11 does not appear in that position 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 >> v1:172.21.15.171:6812/7327273 conn(0x7f403c038f40 legacy=0x7f403c03b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 10 monclient: shutdown 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 >> v1:172.21.15.171:6789/0 conn(0x7f405813d620 legacy=0x7f40581d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 shutdown_connections 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 >> 172.21.15.171:0/1158894534 conn(0x7f40580a98b0 msgr2=0x7f40580a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 shutdown_connections 2024-09-16T04:09:06.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.738+0000 7f40607d8640 1 -- 172.21.15.171:0/1158894534 wait complete. 2024-09-16T04:09:06.749 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:06.749 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c host1 2024-09-16T04:09:06.749 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 1 2024-09-16T04:09:06.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.823+0000 7f19abc8b640 10 monclient: get_monmap_and_config 2024-09-16T04:09:06.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.823+0000 7f19abc8b640 10 monclient: build_initial_monmap 2024-09-16T04:09:06.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.823+0000 7f19abc8b640 10 monclient: monmap: 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:06.824291+0000 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:06.824291+0000 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 1 Processor -- start 2024-09-16T04:09:06.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 1 -- start start 2024-09-16T04:09:06.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient: init 2024-09-16T04:09:06.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient(hunting): picked mon.noname-c con 0x7f19a4133970 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient(hunting): picked mon.noname-b con 0x7f19a409aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient(hunting): picked mon.noname-a con 0x7f19a413d210 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a409b700 con 0x7f19a413d210 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a409b140 con 0x7f19a4133970 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a409b420 con 0x7f19a409aa20 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19abc8b640 10 monclient(hunting): authenticate will time out at 922.329871s 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19a9a27640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f19a4133970 0x7f19a4133d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60204/0 (socket says 172.21.15.171:60204) 2024-09-16T04:09:06.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19a9a27640 1 -- 172.21.15.171:0/2458290398 learned_addr learned my addr 172.21.15.171:0/2458290398 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.824+0000 7f19aa228640 1 -- 172.21.15.171:0/2458290398 >> v1:172.21.15.148:6789/0 conn(0x7f19a413d210 legacy=0x7f19a413f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202834794 0 0) 0x7f19a409b140 con 0x7f19a4133970 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient(hunting): my global_id is 5128 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1990003520 con 0x7f19a4133970 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3523538522 0 0) 0x7f1990003520 con 0x7f19a4133970 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 >> v1:172.21.15.148:6790/0 conn(0x7f19a409aa20 legacy=0x7f19a413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 >> v1:172.21.15.148:6789/0 conn(0x7f19a413d210 legacy=0x7f19a413f600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: _finish_hunting 0 2024-09-16T04:09:06.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 monclient: found mon.noname-c 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a4075c40 con 0x7f19a4133970 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: _finish_auth 0 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: _check_auth_tickets 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19a0001610 con 0x7f19a4133970 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 5 monclient: authenticate success, global_id 5128 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:06.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: dump: 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f19a0001970 con 0x7f19a4133970 2024-09-16T04:09:06.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 1 -- 172.21.15.171:0/2458290398 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19a0001d40 con 0x7f19a4133970 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 4 set_mon_vals no callback set 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19a8a25640 10 monclient: dump: 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.838 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 10 monclient: shutdown 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.825+0000 7f19abc8b640 1 -- 172.21.15.171:0/2458290398 >> v1:172.21.15.171:6789/0 conn(0x7f19a4133970 legacy=0x7f19a4133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 1 -- 172.21.15.171:0/2458290398 shutdown_connections 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 1 -- 172.21.15.171:0/2458290398 >> 172.21.15.171:0/2458290398 conn(0x7f19a40a63c0 msgr2=0x7f19a40a2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 1 -- 172.21.15.171:0/2458290398 shutdown_connections 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 1 -- 172.21.15.171:0/2458290398 wait complete. 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 10 monclient: build_initial_monmap 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 10 monclient: monmap: 2024-09-16T04:09:06.839 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 1 Processor -- start 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 1 -- start start 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.826+0000 7f19abc8b640 10 monclient: init 2024-09-16T04:09:06.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient(hunting): picked mon.noname-c con 0x7f19a409aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient(hunting): picked mon.noname-a con 0x7f19a4133970 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient(hunting): picked mon.noname-b con 0x7f19a413d210 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a411d800 con 0x7f19a4133970 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a411dae0 con 0x7f19a413d210 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a41331c0 con 0x7f19a409aa20 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19abc8b640 10 monclient(hunting): authenticate will time out at 922.332409s 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19a9226640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f19a4133970 legacy=0x7f19a41d07e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:06.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19aa228640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f19a413d210 0x7f19a41d3f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60206/0 (socket says 172.21.15.171:60206) 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f19aa228640 1 -- 172.21.15.171:0/1415281329 learned_addr learned my addr 172.21.15.171:0/1415281329 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241617624 0 0) 0x7f19a411dae0 con 0x7f19a413d210 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): my global_id is 5131 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1988003520 con 0x7f19a413d210 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024088566 0 0) 0x7f19a41331c0 con 0x7f19a409aa20 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:06.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): my global_id is 5162 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19a411dae0 con 0x7f19a409aa20 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1937066532 0 0) 0x7f1988003520 con 0x7f19a413d210 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19a41331c0 con 0x7f19a413d210 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f199c004440 con 0x7f19a413d210 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.843 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3758996202 0 0) 0x7f19a411dae0 con 0x7f19a409aa20 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.827+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1988003520 con 0x7f19a409aa20 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19a0001430 con 0x7f19a409aa20 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:06.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2451636115 0 0) 0x7f19a41331c0 con 0x7f19a413d210 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 >> v1:172.21.15.148:6790/0 conn(0x7f19a409aa20 legacy=0x7f19a4131f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:06.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 >> v1:172.21.15.148:6789/0 conn(0x7f19a4133970 legacy=0x7f19a41d07e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 monclient: found mon.b 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a413bfe0 con 0x7f19a413d210 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 5 monclient: authenticate success, global_id 5131 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 10 monclient: _renew_subs 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f19a411d1e0 con 0x7f19a413d210 2024-09-16T04:09:06.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 10 monclient: _renew_subs 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f19a413b530 con 0x7f19a413d210 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f199c0046f0 con 0x7f19a413d210 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f199c004ac0 con 0x7f19a413d210 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199bfff640 4 set_mon_vals no callback set 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 10 monclient: dump: 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.847 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.828+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f199c004d60 con 0x7f19a413d210 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.829+0000 7f19abc8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.829+0000 7f19abc8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.829+0000 7f19abc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.829+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a40008d0 con 0x7f19a413d210 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.829+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5449+0+0 (unknown 3719219916 0 0) 0x7f199c047860 con 0x7f19a413d210 2024-09-16T04:09:06.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.829+0000 7f19a9a27640 10 monclient: get_auth_request con 0x7f1988038ea0 auth_method 0 2024-09-16T04:09:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.833+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f199c016020 con 0x7f19a413d210 2024-09-16T04:09:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.833+0000 7f199a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:06.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.833+0000 7f199a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.998+0000 7f19abc8b640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.998+0000 7f19abc8b640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.998+0000 7f19abc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.998+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f19a413b820 con 0x7f19a413d210 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.999+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) ==== 56+0+1340 (unknown 452184613 0 734843752) 0x7f199c015920 con 0x7f19a413d210 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.999+0000 7f199a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:06.999+0000 7f199a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f19abc8b640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f19abc8b640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f19abc8b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f19a41326a0 con 0x7f19a413d210 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f199a7fc640 1 -- 172.21.15.171:0/1415281329 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) ==== 56+0+1340 (unknown 452184613 0 734843752) 0x7f199c01e4d0 con 0x7f19a413d210 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f199a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:07.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.097+0000 7f199a7fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:07.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 >> v1:172.21.15.171:6812/7327273 conn(0x7f1988038ea0 legacy=0x7f198803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 10 monclient: shutdown 2024-09-16T04:09:07.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:07.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 >> v1:172.21.15.171:6789/0 conn(0x7f19a413d210 legacy=0x7f19a41d3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 shutdown_connections 2024-09-16T04:09:07.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 >> 172.21.15.171:0/1415281329 conn(0x7f19a40a63c0 msgr2=0x7f19a4136540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:07.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 shutdown_connections 2024-09-16T04:09:07.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.099+0000 7f19abc8b640 1 -- 172.21.15.171:0/1415281329 wait complete. 2024-09-16T04:09:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T04:09:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm foo 2024-09-16T04:09:07.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.186+0000 7f78b04ae640 10 monclient: get_monmap_and_config 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.186+0000 7f78b04ae640 10 monclient: build_initial_monmap 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.186+0000 7f78b04ae640 10 monclient: monmap: 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:07.187724+0000 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:07.187724+0000 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:07.189 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:07.190 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.187+0000 7f78b04ae640 1 Processor -- start 2024-09-16T04:09:07.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.187+0000 7f78b04ae640 1 -- start start 2024-09-16T04:09:07.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.187+0000 7f78b04ae640 10 monclient: init 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient(hunting): picked mon.noname-b con 0x7f78a81338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient(hunting): picked mon.noname-c con 0x7f78a809aa20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient(hunting): picked mon.noname-a con 0x7f78a813d240 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a81343d0 con 0x7f78a813d240 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a8075db0 con 0x7f78a809aa20 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a809b1e0 con 0x7f78a81338e0 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 10 monclient(hunting): authenticate will time out at 922.693359s 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78aea4b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f78a813d240 legacy=0x7f78a813f630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ada49640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78a809aa20 0x7f78a813bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60218/0 (socket says 172.21.15.171:60218) 2024-09-16T04:09:07.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ada49640 1 -- 172.21.15.171:0/695427975 learned_addr learned my addr 172.21.15.171:0/695427975 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3472799095 0 0) 0x7f78a8075db0 con 0x7f78a809aa20 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): my global_id is 5134 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7894003520 con 0x7f78a809aa20 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2830767193 0 0) 0x7f78a809b1e0 con 0x7f78a81338e0 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient(hunting): my global_id is 5165 2024-09-16T04:09:07.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a8075db0 con 0x7f78a81338e0 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2467157916 0 0) 0x7f7894003520 con 0x7f78a809aa20 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 >> v1:172.21.15.148:6790/0 conn(0x7f78a81338e0 legacy=0x7f78a8133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 >> v1:172.21.15.148:6789/0 conn(0x7f78a813d240 legacy=0x7f78a813f630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: _finish_hunting 0 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 monclient: found mon.noname-c 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a809b4c0 con 0x7f78a809aa20 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: _finish_auth 0 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: _check_auth_tickets 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a4001990 con 0x7f78a809aa20 2024-09-16T04:09:07.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78ad248640 10 monclient: dump: 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 5 monclient: authenticate success, global_id 5134 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.188+0000 7f78b04ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78a4001c40 con 0x7f78a809aa20 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78ad248640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78ad248640 1 -- 172.21.15.171:0/695427975 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a4002010 con 0x7f78a809aa20 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 4 set_mon_vals no callback set 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78ad248640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78ad248640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78ad248640 10 monclient: dump: 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.202 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 10 monclient: shutdown 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 1 -- 172.21.15.171:0/695427975 >> v1:172.21.15.171:6789/0 conn(0x7f78a809aa20 legacy=0x7f78a813bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 1 -- 172.21.15.171:0/695427975 shutdown_connections 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 1 -- 172.21.15.171:0/695427975 >> 172.21.15.171:0/695427975 conn(0x7f78a80a28d0 msgr2=0x7f78a80a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 1 -- 172.21.15.171:0/695427975 shutdown_connections 2024-09-16T04:09:07.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.189+0000 7f78b04ae640 1 -- 172.21.15.171:0/695427975 wait complete. 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient: build_initial_monmap 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient: monmap: 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:07.204 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 1 Processor -- start 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 1 -- start start 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient: init 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient(hunting): picked mon.noname-a con 0x7f78a809aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient(hunting): picked mon.noname-c con 0x7f78a81338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.190+0000 7f78b04ae640 10 monclient(hunting): picked mon.noname-b con 0x7f78a813d240 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a811dca0 con 0x7f78a809aa20 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a813c210 con 0x7f78a813d240 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78ae24a640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f78a809aa20 legacy=0x7f78a81325a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a80a0010 con 0x7f78a81338e0 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:07.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 10 monclient(hunting): authenticate will time out at 922.696080s 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78aea4b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78a813d240 0x7f78a81d3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:60228/0 (socket says 172.21.15.171:60228) 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78aea4b640 1 -- 172.21.15.171:0/2915658286 learned_addr learned my addr 172.21.15.171:0/2915658286 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850589544 0 0) 0x7f78a813c210 con 0x7f78a813d240 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): my global_id is 5137 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f788c003520 con 0x7f78a813d240 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1192637984 0 0) 0x7f78a80a0010 con 0x7f78a81338e0 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): my global_id is 5168 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a813c210 con 0x7f78a81338e0 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 495508833 0 0) 0x7f788c003520 con 0x7f78a813d240 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78a80a0010 con 0x7f78a813d240 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a00040c0 con 0x7f78a813d240 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): dump: 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.207 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3922042384 0 0) 0x7f78a813c210 con 0x7f78a81338e0 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f788c003520 con 0x7f78a81338e0 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a4001f90 con 0x7f78a81338e0 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient(hunting): dump: 2024-09-16T04:09:07.208 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1895093287 0 0) 0x7f78a80a0010 con 0x7f78a813d240 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 >> v1:172.21.15.148:6790/0 conn(0x7f78a81338e0 legacy=0x7f78a81d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 >> v1:172.21.15.148:6789/0 conn(0x7f78a809aa20 legacy=0x7f78a81325a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 monclient: found mon.b 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a809d0c0 con 0x7f78a813d240 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: _finish_auth 0 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 5 monclient: authenticate success, global_id 5137 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 10 monclient: _renew_subs 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78a8133380 con 0x7f78a813d240 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 10 monclient: _renew_subs 2024-09-16T04:09:07.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78b04ae640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78a8132cb0 con 0x7f78a813d240 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78a00043e0 con 0x7f78a813d240 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a0004730 con 0x7f78a813d240 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f78aca47640 4 set_mon_vals no callback set 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.191+0000 7f789affd640 10 monclient: dump: 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.211 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.192+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f78a00048d0 con 0x7f78a813d240 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.192+0000 7f78b04ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.192+0000 7f78b04ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.192+0000 7f78b04ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.192+0000 7f78b04ae640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7890005080 con 0x7f78a813d240 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.193+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5449+0+0 (unknown 3719219916 0 0) 0x7f78a00473a0 con 0x7f78a813d240 2024-09-16T04:09:07.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.193+0000 7f78ae24a640 10 monclient: get_auth_request con 0x7f788c038ef0 auth_method 0 2024-09-16T04:09:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.197+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f78a0015bc0 con 0x7f78a813d240 2024-09-16T04:09:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.197+0000 7f789affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.197+0000 7f789affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.369+0000 7f78b04ae640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.369+0000 7f78b04ae640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.369+0000 7f78b04ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.369+0000 7f78b04ae640 1 -- 172.21.15.171:0/2915658286 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f7890005370 con 0x7f78a813d240 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.370+0000 7f789affd640 1 -- 172.21.15.171:0/2915658286 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v76) ==== 99+0+0 (unknown 2758449796 0 0) 0x7f78a00154c0 con 0x7f78a813d240 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.370+0000 7f789affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:07.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.370+0000 7f789affd640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T04:09:07.373 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T04:09:07.376 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:07.376 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T04:09:07.376 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:07.376 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm bar 2024-09-16T04:09:07.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.453+0000 7fc3a7223640 10 monclient: get_monmap_and_config 2024-09-16T04:09:07.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.453+0000 7fc3a7223640 10 monclient: build_initial_monmap 2024-09-16T04:09:07.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.453+0000 7fc3a7223640 10 monclient: monmap: 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:07.454463+0000 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:07.454463+0000 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:07.456 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 1 Processor -- start 2024-09-16T04:09:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 1 -- start start 2024-09-16T04:09:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient: init 2024-09-16T04:09:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:07.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a0132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a01332b0 con 0x7fc3a009ec20 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a0134310 con 0x7fc3a0132a80 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a009d980 con 0x7fc3a0132e60 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a7223640 10 monclient(hunting): authenticate will time out at 922.959894s 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc39ffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc3a009ec20 legacy=0x7fc3a0133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.454+0000 7fc3a4fbf640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc3a0132a80 0x7fc3a01334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35922/0 (socket says 172.21.15.171:35922) 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc3a4fbf640 1 -- 172.21.15.171:0/3321843650 learned_addr learned my addr 172.21.15.171:0/3321843650 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:07.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764870045 0 0) 0x7fc3a0134310 con 0x7fc3a0132a80 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient(hunting): my global_id is 5140 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc390003520 con 0x7fc3a0132a80 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 123182809 0 0) 0x7fc390003520 con 0x7fc3a0132a80 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 >> v1:172.21.15.148:6790/0 conn(0x7fc3a0132e60 legacy=0x7fc3a013c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 >> v1:172.21.15.148:6789/0 conn(0x7fc3a009ec20 legacy=0x7fc3a0133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:07.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a009dc60 con 0x7fc3a0132a80 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc3a7223640 5 monclient: authenticate success, global_id 5140 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc394001610 con 0x7fc3a0132a80 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: dump: 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.466 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc3a7223640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc394001900 con 0x7fc3a0132a80 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc3a7223640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc3a7223640 4 set_mon_vals no callback set 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 1 -- 172.21.15.171:0/3321843650 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc394001cd0 con 0x7fc3a0132a80 2024-09-16T04:09:07.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.455+0000 7fc39f7fe640 10 monclient: dump: 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.468 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 10 monclient: shutdown 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 1 -- 172.21.15.171:0/3321843650 >> v1:172.21.15.171:6789/0 conn(0x7fc3a0132a80 legacy=0x7fc3a01334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 1 -- 172.21.15.171:0/3321843650 shutdown_connections 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 1 -- 172.21.15.171:0/3321843650 >> 172.21.15.171:0/3321843650 conn(0x7fc3a00a52a0 msgr2=0x7fc3a00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 1 -- 172.21.15.171:0/3321843650 shutdown_connections 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 1 -- 172.21.15.171:0/3321843650 wait complete. 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 10 monclient: build_initial_monmap 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 10 monclient: monmap: 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:07.469 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.456+0000 7fc3a7223640 1 Processor -- start 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 1 -- start start 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient: init 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:07.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a0132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a0132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a0121a70 con 0x7fc3a009ec20 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a0123200 con 0x7fc3a0132e60 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a009c8f0 con 0x7fc3a0132a80 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a4fbf640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc3a009ec20 legacy=0x7fc3a01cb710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a7223640 10 monclient(hunting): authenticate will time out at 922.962652s 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a57c0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc3a0132e60 0x7fc3a01d40e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35932/0 (socket says 172.21.15.171:35932) 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc3a57c0640 1 -- 172.21.15.171:0/2995594978 learned_addr learned my addr 172.21.15.171:0/2995594978 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167583937 0 0) 0x7fc3a0123200 con 0x7fc3a0132e60 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc39d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc39d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc39d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc39d7fa640 10 monclient(hunting): my global_id is 5143 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.457+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc384003520 con 0x7fc3a0132e60 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839933436 0 0) 0x7fc3a009c8f0 con 0x7fc3a0132a80 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): my global_id is 5174 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3a0123200 con 0x7fc3a0132a80 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2776424775 0 0) 0x7fc384003520 con 0x7fc3a0132e60 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3a009c8f0 con 0x7fc3a0132e60 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc398001c30 con 0x7fc3a0132e60 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:07.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:07.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 186508630 0 0) 0x7fc3a0123200 con 0x7fc3a0132a80 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc384003520 con 0x7fc3a0132a80 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc38c0040f0 con 0x7fc3a0132a80 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.474 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3512991405 0 0) 0x7fc3a009c8f0 con 0x7fc3a0132e60 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 >> v1:172.21.15.148:6790/0 conn(0x7fc3a0132a80 legacy=0x7fc3a01d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 >> v1:172.21.15.148:6789/0 conn(0x7fc3a009ec20 legacy=0x7fc3a01cb710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 monclient: found mon.b 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a0122f20 con 0x7fc3a0132e60 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:07.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 5 monclient: authenticate success, global_id 5143 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 10 monclient: _renew_subs 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a01217f0 con 0x7fc3a0132e60 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 10 monclient: _renew_subs 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc3a7223640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3a0120890 con 0x7fc3a0132e60 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc398002000 con 0x7fc3a0132e60 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc398003de0 con 0x7fc3a0132e60 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39effd640 4 set_mon_vals no callback set 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.458+0000 7fc39d7fa640 10 monclient: dump: 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc398004000 con 0x7fc3a0132e60 2024-09-16T04:09:07.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5449+0+0 (unknown 3719219916 0 0) 0x7fc3980478e0 con 0x7fc3a0132e60 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc3a7223640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc3a7223640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc3a7223640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc3a7223640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc380005080 con 0x7fc3a0132e60 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.459+0000 7fc3a4fbf640 10 monclient: get_auth_request con 0x7fc384038f10 auth_method 0 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.464+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc398016020 con 0x7fc3a0132e60 2024-09-16T04:09:07.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.464+0000 7fc39d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.464+0000 7fc39d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:07.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc3a7223640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc3a7223640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc3a7223640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc3a7223640 1 -- 172.21.15.171:0/2995594978 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7fc380005370 con 0x7fc3a0132e60 2024-09-16T04:09:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc39d7fa640 1 -- 172.21.15.171:0/2995594978 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v76) ==== 99+0+0 (unknown 88219787 0 0) 0x7fc398019cd0 con 0x7fc3a0132e60 2024-09-16T04:09:07.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc39d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:07.636 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.633+0000 7fc39d7fa640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T04:09:07.636 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T04:09:07.639 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:07.639 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush unlink host1 bar 2024-09-16T04:09:07.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.717+0000 7f110805e640 10 monclient: get_monmap_and_config 2024-09-16T04:09:07.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.717+0000 7f110805e640 10 monclient: build_initial_monmap 2024-09-16T04:09:07.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.717+0000 7f110805e640 10 monclient: monmap: 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:07.718032+0000 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:07.718032+0000 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.717+0000 7f110805e640 1 Processor -- start 2024-09-16T04:09:07.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 1 -- start start 2024-09-16T04:09:07.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient: init 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient(hunting): picked mon.noname-b con 0x7f110009f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient(hunting): picked mon.noname-a con 0x7f11000a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient(hunting): picked mon.noname-c con 0x7f110009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11000a1260 con 0x7f11000a0130 2024-09-16T04:09:07.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110009d0c0 con 0x7f110009e1d0 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110009d3a0 con 0x7f110009f640 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f110805e640 10 monclient(hunting): authenticate will time out at 923.223594s 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f11055f9640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f11000a0130 legacy=0x7f1100138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f11065fb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f110009e1d0 0x7f110013bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35948/0 (socket says 172.21.15.171:35948) 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f11065fb640 1 -- 172.21.15.171:0/1639395643 learned_addr learned my addr 172.21.15.171:0/1639395643 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038118794 0 0) 0x7f110009d0c0 con 0x7f110009e1d0 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f1104df8640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f1104df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f1104df8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f1104df8640 10 monclient(hunting): my global_id is 5146 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.718+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10ec003520 con 0x7f110009e1d0 2024-09-16T04:09:07.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102473782 0 0) 0x7f110009d3a0 con 0x7f110009f640 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient(hunting): my global_id is 5177 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f110009d0c0 con 0x7f110009f640 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3267219668 0 0) 0x7f10ec003520 con 0x7f110009e1d0 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 >> v1:172.21.15.148:6790/0 conn(0x7f110009f640 legacy=0x7f110009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 >> v1:172.21.15.148:6789/0 conn(0x7f11000a0130 legacy=0x7f1100138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: _finish_hunting 0 2024-09-16T04:09:07.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 monclient: found mon.noname-c 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110009d680 con 0x7f110009e1d0 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: _finish_auth 0 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: _check_auth_tickets 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10f8001990 con 0x7f110009e1d0 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 5 monclient: authenticate success, global_id 5146 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: dump: 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.730 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10f8001c40 con 0x7f110009e1d0 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 1 -- 172.21.15.171:0/1639395643 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10f8002010 con 0x7f110009e1d0 2024-09-16T04:09:07.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 4 set_mon_vals no callback set 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f1104df8640 10 monclient: dump: 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.732 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 10 monclient: shutdown 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 1 -- 172.21.15.171:0/1639395643 >> v1:172.21.15.171:6789/0 conn(0x7f110009e1d0 legacy=0x7f110013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 1 -- 172.21.15.171:0/1639395643 shutdown_connections 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 1 -- 172.21.15.171:0/1639395643 >> 172.21.15.171:0/1639395643 conn(0x7f11000a52a0 msgr2=0x7f11000a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 1 -- 172.21.15.171:0/1639395643 shutdown_connections 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.719+0000 7f110805e640 1 -- 172.21.15.171:0/1639395643 wait complete. 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.720+0000 7f110805e640 10 monclient: build_initial_monmap 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.720+0000 7f110805e640 10 monclient: monmap: 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:07.733 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.720+0000 7f110805e640 1 Processor -- start 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.720+0000 7f110805e640 1 -- start start 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.720+0000 7f110805e640 10 monclient: init 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:07.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient(hunting): picked mon.noname-c con 0x7f110009e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient(hunting): picked mon.noname-a con 0x7f110009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient(hunting): picked mon.noname-b con 0x7f11000a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1100125c90 con 0x7f110009f640 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110006ceb0 con 0x7f11000a0130 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11000724d0 con 0x7f110009e1d0 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f110805e640 10 monclient(hunting): authenticate will time out at 923.226379s 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f11055f9640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f110009f640 legacy=0x7f11001ce350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f11065fb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f11000a0130 0x7f11001d1a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35958/0 (socket says 172.21.15.171:35958) 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f11065fb640 1 -- 172.21.15.171:0/3795318088 learned_addr learned my addr 172.21.15.171:0/3795318088 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377100866 0 0) 0x7f110006ceb0 con 0x7f11000a0130 2024-09-16T04:09:07.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): my global_id is 5149 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e4003520 con 0x7f11000a0130 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220790601 0 0) 0x7f11000724d0 con 0x7f110009e1d0 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): my global_id is 5180 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f110006ceb0 con 0x7f110009e1d0 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 615306715 0 0) 0x7f10e4003520 con 0x7f11000a0130 2024-09-16T04:09:07.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11000724d0 con 0x7f11000a0130 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10f8001f90 con 0x7f11000a0130 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): dump: 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.737 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 317748023 0 0) 0x7f110006ceb0 con 0x7f110009e1d0 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10e4003520 con 0x7f110009e1d0 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10fc0018b0 con 0x7f110009e1d0 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 10 monclient(hunting): dump: 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.738 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.721+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 267178828 0 0) 0x7f11000724d0 con 0x7f11000a0130 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 >> v1:172.21.15.148:6790/0 conn(0x7f110009e1d0 legacy=0x7f110013d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 >> v1:172.21.15.148:6789/0 conn(0x7f110009f640 legacy=0x7f11001ce350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 monclient: found mon.b 2024-09-16T04:09:07.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11001256d0 con 0x7f11000a0130 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 5 monclient: authenticate success, global_id 5149 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 monclient: _renew_subs 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11001259b0 con 0x7f11000a0130 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 monclient: _renew_subs 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f110013d8f0 con 0x7f11000a0130 2024-09-16T04:09:07.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10f8004fc0 con 0x7f11000a0130 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10f8005350 con 0x7f11000a0130 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f7fff640 4 set_mon_vals no callback set 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 10 monclient: dump: 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:07.741 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f10f80054f0 con 0x7f11000a0130 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.722+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e8005080 con 0x7f11000a0130 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.723+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 5449+0+0 (unknown 3719219916 0 0) 0x7f10f8047810 con 0x7f11000a0130 2024-09-16T04:09:07.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.723+0000 7f1105dfa640 10 monclient: get_auth_request con 0x7f10e4038ec0 auth_method 0 2024-09-16T04:09:07.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.727+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f10f8019d00 con 0x7f11000a0130 2024-09-16T04:09:07.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.727+0000 7f10f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:07.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.727+0000 7f10f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:07.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.897+0000 7f110805e640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T04:09:07.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.897+0000 7f110805e640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T04:09:07.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.897+0000 7f110805e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:07.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:07.897+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) -- 0x7f10e8005b80 con 0x7f11000a0130 2024-09-16T04:09:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.258+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v77) ==== 147+0+0 (unknown 3542451779 0 0) 0x7f10f801e5c0 con 0x7f11000a0130 2024-09-16T04:09:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.258+0000 7f10f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T04:09:08.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.258+0000 7f10f67fc640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T04:09:08.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f110805e640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T04:09:08.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f110805e640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T04:09:08.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f110805e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) -- 0x7f10e8001090 con 0x7f11000a0130 2024-09-16T04:09:08.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f10f67fc640 1 -- 172.21.15.171:0/3795318088 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v77) ==== 141+0+0 (unknown 349735148 0 0) 0x7f10f80157f0 con 0x7f11000a0130 2024-09-16T04:09:08.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f10f67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-16T04:09:08.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.366+0000 7f10f67fc640 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-16T04:09:08.369 INFO:tasks.workunit.client.0.smithi171.stderr:item -11 does not appear in that position 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.368+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 >> v1:172.21.15.171:6812/7327273 conn(0x7f10e4038ec0 legacy=0x7f10e403b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.368+0000 7f110805e640 10 monclient: shutdown 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.368+0000 7f110805e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.368+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 >> v1:172.21.15.171:6789/0 conn(0x7f11000a0130 legacy=0x7f11001d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.368+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 shutdown_connections 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.368+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 >> 172.21.15.171:0/3795318088 conn(0x7f11000a52a0 msgr2=0x7f11000a3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.369+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 shutdown_connections 2024-09-16T04:09:08.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.369+0000 7f110805e640 1 -- 172.21.15.171:0/3795318088 wait complete. 2024-09-16T04:09:08.380 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:08.380 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c host1 2024-09-16T04:09:08.380 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 1 2024-09-16T04:09:08.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.453+0000 7f547a7d8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:08.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.453+0000 7f547a7d8640 10 monclient: build_initial_monmap 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.453+0000 7f547a7d8640 10 monclient: monmap: 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:08.454574+0000 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:08.454574+0000 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:08.456 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.454+0000 7f547a7d8640 1 Processor -- start 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.454+0000 7f547a7d8640 1 -- start start 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.454+0000 7f547a7d8640 10 monclient: init 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.454+0000 7f547a7d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.454+0000 7f547a7d8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 10 monclient(hunting): picked mon.noname-b con 0x7f547409ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 10 monclient(hunting): picked mon.noname-c con 0x7f5474073570 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 10 monclient(hunting): picked mon.noname-a con 0x7f547409dcb0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54740a0460 con 0x7f547409dcb0 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54740a0db0 con 0x7f5474073570 2024-09-16T04:09:08.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54740a11b0 con 0x7f547409ba30 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 10 monclient(hunting): authenticate will time out at 923.960064s 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5478d75640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f547409dcb0 legacy=0x7f547413b460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f54737fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5474073570 0x7f547409fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35960/0 (socket says 172.21.15.171:35960) 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f54737fe640 1 -- 172.21.15.171:0/3365699217 learned_addr learned my addr 172.21.15.171:0/3365699217 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463711793 0 0) 0x7f54740a0db0 con 0x7f5474073570 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): my global_id is 5152 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f545c003440 con 0x7f5474073570 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955446329 0 0) 0x7f54740a11b0 con 0x7f547409ba30 2024-09-16T04:09:08.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient(hunting): my global_id is 5183 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54740a0db0 con 0x7f547409ba30 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 394361181 0 0) 0x7f545c003440 con 0x7f5474073570 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 >> v1:172.21.15.148:6790/0 conn(0x7f547409ba30 legacy=0x7f547409f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 >> v1:172.21.15.148:6789/0 conn(0x7f547409dcb0 legacy=0x7f547413b460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547409cff0 con 0x7f5474073570 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:08.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5468001610 con 0x7f5474073570 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: dump: 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.467 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 5 monclient: authenticate success, global_id 5152 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5468001930 con 0x7f5474073570 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 1 -- 172.21.15.171:0/3365699217 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5468001d00 con 0x7f5474073570 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f547a7d8640 4 set_mon_vals no callback set 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.455+0000 7f5472ffd640 10 monclient: dump: 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.468 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 10 monclient: shutdown 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 1 -- 172.21.15.171:0/3365699217 >> v1:172.21.15.171:6789/0 conn(0x7f5474073570 legacy=0x7f547409fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 1 -- 172.21.15.171:0/3365699217 shutdown_connections 2024-09-16T04:09:08.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 1 -- 172.21.15.171:0/3365699217 >> 172.21.15.171:0/3365699217 conn(0x7f54740a52a0 msgr2=0x7f54740a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 1 -- 172.21.15.171:0/3365699217 shutdown_connections 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 1 -- 172.21.15.171:0/3365699217 wait complete. 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 10 monclient: build_initial_monmap 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.456+0000 7f547a7d8640 10 monclient: monmap: 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:08.470 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 1 Processor -- start 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 1 -- start start 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient: init 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient(hunting): picked mon.noname-b con 0x7f5474073570 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient(hunting): picked mon.noname-a con 0x7f547409ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient(hunting): picked mon.noname-c con 0x7f547409dcb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547406cea0 con 0x7f547409ba30 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54741a48c0 con 0x7f5474073570 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5474072200 con 0x7f547409dcb0 2024-09-16T04:09:08.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f547a7d8640 10 monclient(hunting): authenticate will time out at 923.962823s 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f5473fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5474073570 0x7f54741b74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35976/0 (socket says 172.21.15.171:35976) 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f5473fff640 1 -- 172.21.15.171:0/1587471712 learned_addr learned my addr 172.21.15.171:0/1587471712 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.457+0000 7f54737fe640 1 -- 172.21.15.171:0/1587471712 >> v1:172.21.15.148:6789/0 conn(0x7f547409ba30 legacy=0x7f54741cbfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934137233 0 0) 0x7f54741a48c0 con 0x7f5474073570 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): my global_id is 5155 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5460003520 con 0x7f5474073570 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090728002 0 0) 0x7f5474072200 con 0x7f547409dcb0 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): my global_id is 5186 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54741a48c0 con 0x7f547409dcb0 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1999224649 0 0) 0x7f5460003520 con 0x7f5474073570 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5474072200 con 0x7f5474073570 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5464001910 con 0x7f5474073570 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.473 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1678056480 0 0) 0x7f5474072200 con 0x7f5474073570 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 >> v1:172.21.15.148:6790/0 conn(0x7f547409dcb0 legacy=0x7f54741cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 >> v1:172.21.15.148:6789/0 conn(0x7f547409ba30 legacy=0x7f54741cbfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 monclient: found mon.b 2024-09-16T04:09:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741a4ba0 con 0x7f5474073570 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 5 monclient: authenticate success, global_id 5155 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 10 monclient: _renew_subs 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f54741a3470 con 0x7f5474073570 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 10 monclient: _renew_subs 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f54741b7db0 con 0x7f5474073570 2024-09-16T04:09:08.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5464003610 con 0x7f5474073570 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5464003960 con 0x7f5474073570 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f54727fc640 4 set_mon_vals no callback set 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.458+0000 7f5470ff9640 10 monclient: dump: 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.476 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5464003b00 con 0x7f5474073570 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f547a7d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f547a7d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f547a7d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5454005080 con 0x7f5474073570 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(77..77 src has 1..77) ==== 5409+0+0 (unknown 60355407 0 0) 0x7f54640473b0 con 0x7f5474073570 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.459+0000 7f54737fe640 10 monclient: get_auth_request con 0x7f5460038be0 auth_method 0 2024-09-16T04:09:08.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.464+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5464015b90 con 0x7f5474073570 2024-09-16T04:09:08.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.464+0000 7f5470ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:08.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.464+0000 7f5470ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:08.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.630+0000 7f547a7d8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:08.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.630+0000 7f547a7d8640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:08.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.630+0000 7f547a7d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.630+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f5454005b80 con 0x7f5474073570 2024-09-16T04:09:08.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.631+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1340 (unknown 2226722781 0 258216497) 0x7f5464019840 con 0x7f5474073570 2024-09-16T04:09:08.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.631+0000 7f5470ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:08.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.631+0000 7f5470ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:08.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f547a7d8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:08.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f547a7d8640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:08.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f547a7d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f5454005d50 con 0x7f5474073570 2024-09-16T04:09:08.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f5470ff9640 1 -- 172.21.15.171:0/1587471712 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) ==== 56+0+1340 (unknown 2226722781 0 258216497) 0x7f546401e100 con 0x7f5474073570 2024-09-16T04:09:08.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f5470ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:08.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.730+0000 7f5470ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.732+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 >> v1:172.21.15.171:6812/7327273 conn(0x7f5460038be0 legacy=0x7f546003b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.732+0000 7f547a7d8640 10 monclient: shutdown 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.732+0000 7f547a7d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.732+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 >> v1:172.21.15.171:6789/0 conn(0x7f5474073570 legacy=0x7f54741b74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.732+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 shutdown_connections 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.732+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 >> 172.21.15.171:0/1587471712 conn(0x7f54740a52a0 msgr2=0x7f54740a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.733+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 shutdown_connections 2024-09-16T04:09:08.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.733+0000 7f547a7d8640 1 -- 172.21.15.171:0/1587471712 wait complete. 2024-09-16T04:09:08.743 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.7 host1 2024-09-16T04:09:08.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.822+0000 7f7aeaaf1640 10 monclient: get_monmap_and_config 2024-09-16T04:09:08.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.822+0000 7f7aeaaf1640 10 monclient: build_initial_monmap 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.822+0000 7f7aeaaf1640 10 monclient: monmap: 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:08.823552+0000 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:08.823552+0000 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.825 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:08.826 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:08.826 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:08.826 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 1 Processor -- start 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 1 -- start start 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 10 monclient: init 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 10 monclient(hunting): picked mon.noname-b con 0x7f7ae409f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.823+0000 7f7aeaaf1640 10 monclient(hunting): picked mon.noname-c con 0x7f7ae40a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 10 monclient(hunting): picked mon.noname-a con 0x7f7ae409e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae40a1260 con 0x7f7ae409e1d0 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae409d0c0 con 0x7f7ae40a0130 2024-09-16T04:09:08.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae409d3a0 con 0x7f7ae409f640 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 10 monclient(hunting): authenticate will time out at 924.329062s 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7ae40a0130 0x7f7ae4138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35980/0 (socket says 172.21.15.171:35980) 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae3fff640 1 -- 172.21.15.171:0/1942268052 learned_addr learned my addr 172.21.15.171:0/1942268052 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae908e640 1 -- 172.21.15.171:0/1942268052 >> v1:172.21.15.148:6789/0 conn(0x7f7ae409e1d0 legacy=0x7f7ae413bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17249932 0 0) 0x7f7ae409d0c0 con 0x7f7ae40a0130 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient(hunting): my global_id is 5158 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ad4003520 con 0x7f7ae40a0130 2024-09-16T04:09:08.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2015747489 0 0) 0x7f7ad4003520 con 0x7f7ae40a0130 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 >> v1:172.21.15.148:6790/0 conn(0x7f7ae409f640 legacy=0x7f7ae409fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 >> v1:172.21.15.148:6789/0 conn(0x7f7ae409e1d0 legacy=0x7f7ae413bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae409d680 con 0x7f7ae40a0130 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:08.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ad0001610 con 0x7f7ae40a0130 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 5 monclient: authenticate success, global_id 5158 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: dump: 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.837 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ad0001900 con 0x7f7ae40a0130 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 1 -- 172.21.15.171:0/1942268052 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ad0001cd0 con 0x7f7ae40a0130 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7aeaaf1640 4 set_mon_vals no callback set 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:08.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.824+0000 7f7ae37fe640 10 monclient: dump: 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 10 monclient: shutdown 2024-09-16T04:09:08.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/1942268052 >> v1:172.21.15.171:6789/0 conn(0x7f7ae40a0130 legacy=0x7f7ae4138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/1942268052 shutdown_connections 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/1942268052 >> 172.21.15.171:0/1942268052 conn(0x7f7ae40a52a0 msgr2=0x7f7ae40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/1942268052 shutdown_connections 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/1942268052 wait complete. 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 10 monclient: build_initial_monmap 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.825+0000 7f7aeaaf1640 10 monclient: monmap: 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:08.840 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.841 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:08.848 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 1 Processor -- start 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 1 -- start start 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient: init 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient(hunting): picked mon.noname-c con 0x7f7ae409e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient(hunting): picked mon.noname-b con 0x7f7ae409f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient(hunting): picked mon.noname-a con 0x7f7ae40a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae4070460 con 0x7f7ae40a0130 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae41cbf20 con 0x7f7ae409f640 2024-09-16T04:09:08.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae41ac120 con 0x7f7ae409e1d0 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7aeaaf1640 10 monclient(hunting): authenticate will time out at 924.331909s 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7ae908e640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7ae40a0130 legacy=0x7f7ae41d3ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7ae3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7ae409f640 0x7f7ae41d0380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:35990/0 (socket says 172.21.15.171:35990) 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.826+0000 7f7ae3fff640 1 -- 172.21.15.171:0/3670841775 learned_addr learned my addr 172.21.15.171:0/3670841775 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420139073 0 0) 0x7f7ae41cbf20 con 0x7f7ae409f640 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): my global_id is 5161 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ac8003520 con 0x7f7ae409f640 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1584629495 0 0) 0x7f7ae41ac120 con 0x7f7ae409e1d0 2024-09-16T04:09:08.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): my global_id is 5192 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ae41cbf20 con 0x7f7ae409e1d0 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3186927708 0 0) 0x7f7ac8003520 con 0x7f7ae409f640 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ae41ac120 con 0x7f7ae409f640 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ad00015d0 con 0x7f7ae409f640 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): dump: 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.851 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2042657965 0 0) 0x7f7ae41cbf20 con 0x7f7ae409e1d0 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ac8003520 con 0x7f7ae409e1d0 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ad8001870 con 0x7f7ae409e1d0 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:08.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient(hunting): dump: 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4022091552 0 0) 0x7f7ae41ac120 con 0x7f7ae409f640 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 >> v1:172.21.15.148:6790/0 conn(0x7f7ae409e1d0 legacy=0x7f7ae41cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 >> v1:172.21.15.148:6789/0 conn(0x7f7ae40a0130 legacy=0x7f7ae41d3ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:08.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 monclient: found mon.b 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae41ac400 con 0x7f7ae409f640 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 5 monclient: authenticate success, global_id 5161 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 10 monclient: _renew_subs 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae41c5690 con 0x7f7ae409f640 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 10 monclient: _renew_subs 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.854 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ae41aac30 con 0x7f7ae409f640 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ad0004cc0 con 0x7f7ae409f640 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae2ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ad0001130 con 0x7f7ae409f640 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.827+0000 7f7ae17fa640 10 monclient: dump: 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:08.855 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7ad0005090 con 0x7f7ae409f640 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7aeaaf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7aeaaf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7aeaaf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac4005080 con 0x7f7ae409f640 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(77..77 src has 1..77) ==== 5409+0+0 (unknown 60355407 0 0) 0x7f7ad0047590 con 0x7f7ae409f640 2024-09-16T04:09:08.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.828+0000 7f7ae888d640 10 monclient: get_auth_request con 0x7f7ac8038ea0 auth_method 0 2024-09-16T04:09:08.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.833+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7ad0019aa0 con 0x7f7ae409f640 2024-09-16T04:09:08.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.833+0000 7f7ae17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:08.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:08.833+0000 7f7ae17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.001+0000 7f7aeaaf1640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T04:09:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.001+0000 7f7aeaaf1640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T04:09:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.001+0000 7f7aeaaf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.001+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) -- 0x7f7ac4005b80 con 0x7f7ae409f640 2024-09-16T04:09:09.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.267+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 removed item id 7 name 'osd.7' from crush map v78) ==== 143+0+0 (unknown 365836961 0 0) 0x7f7ad001e360 con 0x7f7ae409f640 2024-09-16T04:09:09.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.267+0000 7f7ae17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T04:09:09.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.267+0000 7f7ae17fa640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7aeaaf1640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7aeaaf1640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7aeaaf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"} v 0) -- 0x7f7ac4001090 con 0x7f7ae409f640 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7ae17fa640 1 -- 172.21.15.171:0/3670841775 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}]=0 device 'osd.7' does not appear in the crush map v78) ==== 145+0+0 (unknown 1963075984 0 0) 0x7f7ad0044d90 con 0x7f7ae409f640 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7ae17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7", "ancestor": "host1"}] 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.371+0000 7f7ae17fa640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T04:09:09.373 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 >> v1:172.21.15.171:6812/7327273 conn(0x7f7ac8038ea0 legacy=0x7f7ac803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 10 monclient: shutdown 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 >> v1:172.21.15.171:6789/0 conn(0x7f7ae409f640 legacy=0x7f7ae41d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 shutdown_connections 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 >> 172.21.15.171:0/3670841775 conn(0x7f7ae40a52a0 msgr2=0x7f7ae40709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 shutdown_connections 2024-09-16T04:09:09.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.373+0000 7f7aeaaf1640 1 -- 172.21.15.171:0/3670841775 wait complete. 2024-09-16T04:09:09.384 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm host1 2024-09-16T04:09:09.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.460+0000 7f48ae542640 10 monclient: get_monmap_and_config 2024-09-16T04:09:09.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.460+0000 7f48ae542640 10 monclient: build_initial_monmap 2024-09-16T04:09:09.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.460+0000 7f48ae542640 10 monclient: monmap: 2024-09-16T04:09:09.462 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:09.461413+0000 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:09.461413+0000 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:09.463 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 1 Processor -- start 2024-09-16T04:09:09.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 1 -- start start 2024-09-16T04:09:09.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient: init 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient(hunting): picked mon.noname-c con 0x7f48a809f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient(hunting): picked mon.noname-a con 0x7f48a80a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient(hunting): picked mon.noname-b con 0x7f48a809e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a80a1260 con 0x7f48a80a0130 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a809d0c0 con 0x7f48a809f640 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a809d3a0 con 0x7f48a809e1d0 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48ae542640 10 monclient(hunting): authenticate will time out at 924.966889s 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.461+0000 7f48a7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f48a809f640 0x7f48a809fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36004/0 (socket says 172.21.15.171:36004) 2024-09-16T04:09:09.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a7fff640 1 -- 172.21.15.171:0/748543103 learned_addr learned my addr 172.21.15.171:0/748543103 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a77fe640 1 -- 172.21.15.171:0/748543103 >> v1:172.21.15.148:6789/0 conn(0x7f48a80a0130 legacy=0x7f48a8138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2018086144 0 0) 0x7f48a809d0c0 con 0x7f48a809f640 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient(hunting): my global_id is 5164 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4890003520 con 0x7f48a809f640 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1555528359 0 0) 0x7f4890003520 con 0x7f48a809f640 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 >> v1:172.21.15.148:6790/0 conn(0x7f48a809e1d0 legacy=0x7f48a813bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 >> v1:172.21.15.148:6789/0 conn(0x7f48a80a0130 legacy=0x7f48a8138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:09.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a809d680 con 0x7f48a809f640 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48ae542640 5 monclient: authenticate success, global_id 5164 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48ae542640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4898001610 con 0x7f48a809f640 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: dump: 2024-09-16T04:09:09.473 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48ae542640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4898001900 con 0x7f48a809f640 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 1 -- 172.21.15.171:0/748543103 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4898001cd0 con 0x7f48a809f640 2024-09-16T04:09:09.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48ae542640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48ae542640 4 set_mon_vals no callback set 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.462+0000 7f48a6ffd640 10 monclient: dump: 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:09.475 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 10 monclient: shutdown 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 1 -- 172.21.15.171:0/748543103 >> v1:172.21.15.171:6789/0 conn(0x7f48a809f640 legacy=0x7f48a809fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 1 -- 172.21.15.171:0/748543103 shutdown_connections 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 1 -- 172.21.15.171:0/748543103 >> 172.21.15.171:0/748543103 conn(0x7f48a80a52a0 msgr2=0x7f48a80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 1 -- 172.21.15.171:0/748543103 shutdown_connections 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 1 -- 172.21.15.171:0/748543103 wait complete. 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 10 monclient: build_initial_monmap 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 10 monclient: monmap: 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:09.476 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.463+0000 7f48ae542640 1 Processor -- start 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 1 -- start start 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient: init 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient(hunting): picked mon.noname-a con 0x7f48a809e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:09.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient(hunting): picked mon.noname-b con 0x7f48a809f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient(hunting): picked mon.noname-c con 0x7f48a80a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a806ceb0 con 0x7f48a809e1d0 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a81a5d50 con 0x7f48a809f640 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a81a77a0 con 0x7f48a80a0130 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48ae542640 10 monclient(hunting): authenticate will time out at 924.969660s 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a7fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f48a809e1d0 legacy=0x7f48a81c68e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f48a809f640 0x7f48a81b6640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36018/0 (socket says 172.21.15.171:36018) 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a77fe640 1 -- 172.21.15.171:0/3739994103 learned_addr learned my addr 172.21.15.171:0/3739994103 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 493204955 0 0) 0x7f48a81a5d50 con 0x7f48a809f640 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:09.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a4ff9640 10 monclient(hunting): my global_id is 5167 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.464+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4894003520 con 0x7f48a809f640 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783824436 0 0) 0x7f48a81a77a0 con 0x7f48a80a0130 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): my global_id is 5198 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48a81a5d50 con 0x7f48a80a0130 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1234132154 0 0) 0x7f4894003520 con 0x7f48a809f640 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48a81a77a0 con 0x7f48a809f640 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f489c0041d0 con 0x7f48a809f640 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:09.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2725485690 0 0) 0x7f48a81a5d50 con 0x7f48a80a0130 2024-09-16T04:09:09.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4894003520 con 0x7f48a80a0130 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48a0001b30 con 0x7f48a80a0130 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:09.481 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3654733876 0 0) 0x7f48a81a77a0 con 0x7f48a809f640 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 >> v1:172.21.15.148:6790/0 conn(0x7f48a80a0130 legacy=0x7f48a81cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 >> v1:172.21.15.148:6789/0 conn(0x7f48a809e1d0 legacy=0x7f48a81c68e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 monclient: found mon.b 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a81a69d0 con 0x7f48a809f640 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 5 monclient: authenticate success, global_id 5167 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 10 monclient: _renew_subs 2024-09-16T04:09:09.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48a81b6d50 con 0x7f48a809f640 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 10 monclient: _renew_subs 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48a81d0520 con 0x7f48a809f640 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f489c004480 con 0x7f48a809f640 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f489c004850 con 0x7f48a809f640 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a67fc640 4 set_mon_vals no callback set 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.465+0000 7f48a4ff9640 10 monclient: dump: 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:09.483 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f489c0049f0 con 0x7f48a809f640 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48ae542640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48ae542640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48ae542640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4888005080 con 0x7f48a809f640 2024-09-16T04:09:09.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(78..78 src has 1..78) ==== 5388+0+0 (unknown 313268142 0 0) 0x7f489c047440 con 0x7f48a809f640 2024-09-16T04:09:09.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.466+0000 7f48a7fff640 10 monclient: get_auth_request con 0x7f4894038f10 auth_method 0 2024-09-16T04:09:09.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.471+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f489c015c40 con 0x7f48a809f640 2024-09-16T04:09:09.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.471+0000 7f48a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:09.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.471+0000 7f48a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:09.642 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.640+0000 7f48ae542640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T04:09:09.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.640+0000 7f48ae542640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T04:09:09.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.640+0000 7f48ae542640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:09.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:09.640+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) -- 0x7f4888005370 con 0x7f48a809f640 2024-09-16T04:09:10.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.272+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v79) ==== 124+0+0 (unknown 968798856 0 0) 0x7f489c015540 con 0x7f48a809f640 2024-09-16T04:09:10.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.272+0000 7f48a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T04:09:10.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.272+0000 7f48a4ff9640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-16T04:09:10.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.377+0000 7f48ae542640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T04:09:10.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.377+0000 7f48ae542640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T04:09:10.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.377+0000 7f48ae542640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.377+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) -- 0x7f4888002210 con 0x7f48a809f640 2024-09-16T04:09:10.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.377+0000 7f48a4ff9640 1 -- 172.21.15.171:0/3739994103 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v79) ==== 124+0+0 (unknown 2715533929 0 0) 0x7f489c019890 con 0x7f48a809f640 2024-09-16T04:09:10.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.378+0000 7f48a4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-16T04:09:10.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.378+0000 7f48a4ff9640 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-16T04:09:10.380 INFO:tasks.workunit.client.0.smithi171.stderr:device 'host1' does not appear in the crush map 2024-09-16T04:09:10.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.379+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 >> v1:172.21.15.171:6812/7327273 conn(0x7f4894038f10 legacy=0x7f489403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.379+0000 7f48ae542640 10 monclient: shutdown 2024-09-16T04:09:10.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.379+0000 7f48ae542640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:10.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.379+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 >> v1:172.21.15.171:6789/0 conn(0x7f48a809f640 legacy=0x7f48a81b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.379+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 shutdown_connections 2024-09-16T04:09:10.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.379+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 >> 172.21.15.171:0/3739994103 conn(0x7f48a80a52a0 msgr2=0x7f48a8070530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:10.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.380+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 shutdown_connections 2024-09-16T04:09:10.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.380+0000 7f48ae542640 1 -- 172.21.15.171:0/3739994103 wait complete. 2024-09-16T04:09:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c host1 2024-09-16T04:09:10.392 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 0 2024-09-16T04:09:10.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.465+0000 7fcdc67af640 10 monclient: get_monmap_and_config 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.465+0000 7fcdc67af640 10 monclient: build_initial_monmap 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.465+0000 7fcdc67af640 10 monclient: monmap: 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:10.466687+0000 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:10.466687+0000 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:10.468 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:10.469 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:10.469 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.466+0000 7fcdc67af640 1 Processor -- start 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.466+0000 7fcdc67af640 1 -- start start 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.466+0000 7fcdc67af640 10 monclient: init 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc00a2880 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc006c2f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc009ef80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc006c070 con 0x7fcdc006c2f0 2024-09-16T04:09:10.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc009e680 con 0x7fcdc009ef80 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc006c770 con 0x7fcdc00a2880 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc67af640 10 monclient(hunting): authenticate will time out at 925.972459s 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdb7fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fcdc006c2f0 legacy=0x7fcdc013be70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc4d4c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcdc009ef80 0x7fcdc013f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36020/0 (socket says 172.21.15.171:36020) 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdc4d4c640 1 -- 172.21.15.171:0/3318423285 learned_addr learned my addr 172.21.15.171:0/3318423285 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425010918 0 0) 0x7fcdc009e680 con 0x7fcdc009ef80 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): my global_id is 5170 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdac003520 con 0x7fcdc009ef80 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006028232 0 0) 0x7fcdc006c770 con 0x7fcdc00a2880 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 10 monclient(hunting): my global_id is 5201 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdc009e680 con 0x7fcdc00a2880 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4011710407 0 0) 0x7fcdac003520 con 0x7fcdc009ef80 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.467+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 >> v1:172.21.15.148:6790/0 conn(0x7fcdc00a2880 legacy=0x7fcdc00a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 >> v1:172.21.15.148:6789/0 conn(0x7fcdc006c2f0 legacy=0x7fcdc013be70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:10.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc00a42f0 con 0x7fcdc009ef80 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcdb8001990 con 0x7fcdc009ef80 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 5 monclient: authenticate success, global_id 5170 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:10.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: dump: 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.484 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcdb8001cb0 con 0x7fcdc009ef80 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 1 -- 172.21.15.171:0/3318423285 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcdb8002080 con 0x7fcdc009ef80 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 4 set_mon_vals no callback set 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdbf7fe640 10 monclient: dump: 2024-09-16T04:09:10.485 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 10 monclient: shutdown 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 1 -- 172.21.15.171:0/3318423285 >> v1:172.21.15.171:6789/0 conn(0x7fcdc009ef80 legacy=0x7fcdc013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 1 -- 172.21.15.171:0/3318423285 shutdown_connections 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 1 -- 172.21.15.171:0/3318423285 >> 172.21.15.171:0/3318423285 conn(0x7fcdc00a74d0 msgr2=0x7fcdc00a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 1 -- 172.21.15.171:0/3318423285 shutdown_connections 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.468+0000 7fcdc67af640 1 -- 172.21.15.171:0/3318423285 wait complete. 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.469+0000 7fcdc67af640 10 monclient: build_initial_monmap 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.469+0000 7fcdc67af640 10 monclient: monmap: 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:10.487 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.469+0000 7fcdc67af640 1 Processor -- start 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.469+0000 7fcdc67af640 1 -- start start 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.469+0000 7fcdc67af640 10 monclient: init 2024-09-16T04:09:10.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc009ef80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc00a2880 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc01caf10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc0125320 con 0x7fcdc01caf10 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc0125600 con 0x7fcdc00a2880 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc009f470 con 0x7fcdc009ef80 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient(hunting): authenticate will time out at 925.975144s 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdb7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcdc00a2880 0x7fcdc01d0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36036/0 (socket says 172.21.15.171:36036) 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc4d4c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fcdc01caf10 legacy=0x7fcdc01d39c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:10.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdb7fff640 1 -- 172.21.15.171:0/2379080620 learned_addr learned my addr 172.21.15.171:0/2379080620 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 9056252 0 0) 0x7fcdc0125600 con 0x7fcdc00a2880 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): my global_id is 5173 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcda0003520 con 0x7fcdc00a2880 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890768651 0 0) 0x7fcdc009f470 con 0x7fcdc009ef80 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): my global_id is 5204 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdc0125600 con 0x7fcdc009ef80 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2013600182 0 0) 0x7fcda0003520 con 0x7fcdc00a2880 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcdc009f470 con 0x7fcdc00a2880 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcda8004180 con 0x7fcdc00a2880 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.491 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2359441378 0 0) 0x7fcdc0125600 con 0x7fcdc009ef80 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcda0003520 con 0x7fcdc009ef80 2024-09-16T04:09:10.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcdb00018b0 con 0x7fcdc009ef80 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.493 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 49083014 0 0) 0x7fcdc009f470 con 0x7fcdc00a2880 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 >> v1:172.21.15.148:6790/0 conn(0x7fcdc009ef80 legacy=0x7fcdc0136f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 >> v1:172.21.15.148:6789/0 conn(0x7fcdc01caf10 legacy=0x7fcdc01d39c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 monclient: found mon.b 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc0124150 con 0x7fcdc00a2880 2024-09-16T04:09:10.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 5 monclient: authenticate success, global_id 5173 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient: _renew_subs 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc0125040 con 0x7fcdc00a2880 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient: _renew_subs 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcdc0137620 con 0x7fcdc00a2880 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcda8004430 con 0x7fcdc00a2880 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:10.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcda8004780 con 0x7fcdc00a2880 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbd7fa640 10 monclient: dump: 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.496 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.470+0000 7fcdbeffd640 4 set_mon_vals no callback set 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.471+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcda8004a20 con 0x7fcdc00a2880 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.471+0000 7fcdc67af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.471+0000 7fcdc67af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.471+0000 7fcdc67af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.471+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc009e0d0 con 0x7fcdc00a2880 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.471+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5253+0+0 (unknown 1284764747 0 0) 0x7fcda80474b0 con 0x7fcdc00a2880 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.472+0000 7fcdbffff640 10 monclient: get_auth_request con 0x7fcda0038ea0 auth_method 0 2024-09-16T04:09:10.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.476+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fcda8015c70 con 0x7fcdc00a2880 2024-09-16T04:09:10.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.476+0000 7fcdbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:10.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.476+0000 7fcdbd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:10.642 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.640+0000 7fcdc67af640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:10.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.640+0000 7fcdc67af640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:10.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.640+0000 7fcdc67af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.640+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fcdc006df00 con 0x7fcdc00a2880 2024-09-16T04:09:10.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.641+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) ==== 56+0+1273 (unknown 3183532764 0 103961603) 0x7fcda8019920 con 0x7fcdc00a2880 2024-09-16T04:09:10.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.641+0000 7fcdbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:10.643 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.641+0000 7fcdbd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.739+0000 7fcdc67af640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.739+0000 7fcdc67af640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.739+0000 7fcdc67af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.739+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fcdc01cbd80 con 0x7fcdc00a2880 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.740+0000 7fcdbd7fa640 1 -- 172.21.15.171:0/2379080620 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v79) ==== 56+0+1273 (unknown 3183532764 0 103961603) 0x7fcda801e1e0 con 0x7fcdc00a2880 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.740+0000 7fcdbd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:10.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.740+0000 7fcdbd7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:10.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 >> v1:172.21.15.171:6812/7327273 conn(0x7fcda0038ea0 legacy=0x7fcda003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 10 monclient: shutdown 2024-09-16T04:09:10.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:10.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 >> v1:172.21.15.171:6789/0 conn(0x7fcdc00a2880 legacy=0x7fcdc01d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 shutdown_connections 2024-09-16T04:09:10.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 >> 172.21.15.171:0/2379080620 conn(0x7fcdc00a74d0 msgr2=0x7fcdc00a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:10.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 shutdown_connections 2024-09-16T04:09:10.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.742+0000 7fcdc67af640 1 -- 172.21.15.171:0/2379080620 wait complete. 2024-09-16T04:09:10.753 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd tree-from host1 2024-09-16T04:09:10.753 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:10.754 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree-from host1 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.828+0000 7f2f39f4b640 10 monclient: get_monmap_and_config 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.828+0000 7f2f39f4b640 10 monclient: build_initial_monmap 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.828+0000 7f2f39f4b640 10 monclient: monmap: 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:10.828910+0000 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:10.828910+0000 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:10.830 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:10.831 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.828+0000 7f2f39f4b640 1 Processor -- start 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.828+0000 7f2f39f4b640 1 -- start start 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.828+0000 7f2f39f4b640 10 monclient: init 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f34136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f2f34139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f340a1340 con 0x7f2f3413d620 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f34137650 con 0x7f2f34139b10 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f340a0aa0 con 0x7f2f34136b60 2024-09-16T04:09:10.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 10 monclient(hunting): authenticate will time out at 926.334457s 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f32ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2f34139b10 0x7f2f3413bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36050/0 (socket says 172.21.15.171:36050) 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f32ffd640 1 -- 172.21.15.171:0/2715290840 learned_addr learned my addr 172.21.15.171:0/2715290840 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f33fff640 1 -- 172.21.15.171:0/2715290840 >> v1:172.21.15.148:6789/0 conn(0x7f2f3413d620 legacy=0x7f2f3413fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927663465 0 0) 0x7f2f34137650 con 0x7f2f34139b10 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient(hunting): my global_id is 5176 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f20003520 con 0x7f2f34139b10 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1405153493 0 0) 0x7f2f20003520 con 0x7f2f34139b10 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 >> v1:172.21.15.148:6790/0 conn(0x7f2f34136b60 legacy=0x7f2f34136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 >> v1:172.21.15.148:6789/0 conn(0x7f2f3413d620 legacy=0x7f2f3413fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f340a0550 con 0x7f2f34139b10 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f39f4b640 5 monclient: authenticate success, global_id 5176 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.829+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f1c001610 con 0x7f2f34139b10 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:10.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: dump: 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:10.841 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f1c001970 con 0x7f2f34139b10 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 1 -- 172.21.15.171:0/2715290840 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f1c0048a0 con 0x7f2f34139b10 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 4 set_mon_vals no callback set 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f327fc640 10 monclient: dump: 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.842 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 10 monclient: shutdown 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 1 -- 172.21.15.171:0/2715290840 >> v1:172.21.15.171:6789/0 conn(0x7f2f34139b10 legacy=0x7f2f3413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 1 -- 172.21.15.171:0/2715290840 shutdown_connections 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 1 -- 172.21.15.171:0/2715290840 >> 172.21.15.171:0/2715290840 conn(0x7f2f340a98b0 msgr2=0x7f2f340a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 1 -- 172.21.15.171:0/2715290840 shutdown_connections 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.830+0000 7f2f39f4b640 1 -- 172.21.15.171:0/2715290840 wait complete. 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.831+0000 7f2f39f4b640 10 monclient: build_initial_monmap 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.831+0000 7f2f39f4b640 10 monclient: monmap: 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:10.843 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.831+0000 7f2f39f4b640 1 Processor -- start 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.831+0000 7f2f39f4b640 1 -- start start 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.831+0000 7f2f39f4b640 10 monclient: init 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:10.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f2f34136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f2f34139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3409f550 con 0x7f2f34139b10 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f340a0240 con 0x7f2f3413d620 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f34120ca0 con 0x7f2f34136b60 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient(hunting): authenticate will time out at 926.337225s 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f32ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2f34139b10 legacy=0x7f2f3409daf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f33fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2f3413d620 0x7f2f341d3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36062/0 (socket says 172.21.15.171:36062) 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f33fff640 1 -- 172.21.15.171:0/1026047249 learned_addr learned my addr 172.21.15.171:0/1026047249 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362791685 0 0) 0x7f2f2c0039f0 con 0x7f2f3413d620 2024-09-16T04:09:10.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): my global_id is 5179 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f14003550 con 0x7f2f3413d620 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794108210 0 0) 0x7f2f34120ca0 con 0x7f2f34136b60 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): my global_id is 5210 2024-09-16T04:09:10.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f2c0039f0 con 0x7f2f34136b60 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3844145329 0 0) 0x7f2f14003550 con 0x7f2f3413d620 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f34120ca0 con 0x7f2f3413d620 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f2c004200 con 0x7f2f3413d620 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): dump: 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.847 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3491563221 0 0) 0x7f2f2c0039f0 con 0x7f2f34136b60 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f14003550 con 0x7f2f34136b60 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f24001bf0 con 0x7f2f34136b60 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:10.848 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient(hunting): dump: 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 360895984 0 0) 0x7f2f34120ca0 con 0x7f2f3413d620 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 >> v1:172.21.15.148:6790/0 conn(0x7f2f34136b60 legacy=0x7f2f3409d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:10.849 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 >> v1:172.21.15.148:6789/0 conn(0x7f2f34139b10 legacy=0x7f2f3409daf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 monclient: found mon.b 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3409e200 con 0x7f2f3413d620 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient: _finish_auth 0 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 5 monclient: authenticate success, global_id 5179 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient: _renew_subs 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f340a2c20 con 0x7f2f3413d620 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient: _renew_subs 2024-09-16T04:09:10.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f39f4b640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f341213a0 con 0x7f2f3413d620 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f2c0044b0 con 0x7f2f3413d620 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.832+0000 7f2f2bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2f2c004880 con 0x7f2f3413d620 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f31ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f2bfff640 10 monclient: dump: 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:10.851 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2f2c004b20 con 0x7f2f3413d620 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f39f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f39f4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f39f4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f39f4b640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f18005080 con 0x7f2f3413d620 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.833+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5253+0+0 (unknown 1284764747 0 0) 0x7f2f2c047410 con 0x7f2f3413d620 2024-09-16T04:09:10.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.834+0000 7f2f337fe640 10 monclient: get_auth_request con 0x7f2f14038ed0 auth_method 0 2024-09-16T04:09:10.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.838+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2f340a0240 con 0x7f2f3413d620 2024-09-16T04:09:10.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.838+0000 7f2f2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:10.853 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:10.838+0000 7f2f2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.005+0000 7f2f39f4b640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.005+0000 7f2f39f4b640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.005+0000 7f2f39f4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.005+0000 7f2f39f4b640 1 -- 172.21.15.171:0/1026047249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) -- 0x7f2f18005370 con 0x7f2f3413d620 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.006+0000 7f2f2bfff640 1 -- 172.21.15.171:0/1026047249 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v79) ==== 109+0+0 (unknown 2784578372 0 0) 0x7f2f2c015dd0 con 0x7f2f3413d620 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.006+0000 7f2f2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.006+0000 7f2f2bfff640 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-16T04:09:11.008 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-16T04:09:11.011 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:11.011 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree-from host2 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.086+0000 7f8f0248d640 10 monclient: get_monmap_and_config 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.086+0000 7f8f0248d640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.086+0000 7f8f0248d640 10 monclient: monmap: 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:11.087678+0000 2024-09-16T04:09:11.089 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:11.087678+0000 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.087+0000 7f8f0248d640 1 Processor -- start 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.087+0000 7f8f0248d640 1 -- start start 2024-09-16T04:09:11.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.087+0000 7f8f0248d640 10 monclient: init 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient(hunting): picked mon.noname-c con 0x7f8efc132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient(hunting): picked mon.noname-a con 0x7f8efc09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient(hunting): picked mon.noname-b con 0x7f8efc132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc1332b0 con 0x7f8efc09ec20 2024-09-16T04:09:11.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc134310 con 0x7f8efc132a80 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc09d980 con 0x7f8efc132e60 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient(hunting): authenticate will time out at 926.593247s 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efb7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8efc09ec20 legacy=0x7f8efc133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efbfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8efc132a80 0x7f8efc1334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36072/0 (socket says 172.21.15.171:36072) 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efbfff640 1 -- 172.21.15.171:0/3109679704 learned_addr learned my addr 172.21.15.171:0/3109679704 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864735766 0 0) 0x7f8efc134310 con 0x7f8efc132a80 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient(hunting): my global_id is 5182 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ee4003520 con 0x7f8efc132a80 2024-09-16T04:09:11.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3756327655 0 0) 0x7f8ee4003520 con 0x7f8efc132a80 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 >> v1:172.21.15.148:6790/0 conn(0x7f8efc132e60 legacy=0x7f8efc13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 >> v1:172.21.15.148:6789/0 conn(0x7f8efc09ec20 legacy=0x7f8efc133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc09dc60 con 0x7f8efc132a80 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8eec001610 con 0x7f8efc132a80 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: dump: 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 5 monclient: authenticate success, global_id 5182 2024-09-16T04:09:11.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8eec001880 con 0x7f8efc132a80 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8f0248d640 4 set_mon_vals no callback set 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 1 -- 172.21.15.171:0/3109679704 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8eec001c90 con 0x7f8efc132a80 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.088+0000 7f8efaffd640 10 monclient: dump: 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.101 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 10 monclient: shutdown 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 1 -- 172.21.15.171:0/3109679704 >> v1:172.21.15.171:6789/0 conn(0x7f8efc132a80 legacy=0x7f8efc1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 1 -- 172.21.15.171:0/3109679704 shutdown_connections 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 1 -- 172.21.15.171:0/3109679704 >> 172.21.15.171:0/3109679704 conn(0x7f8efc0a52a0 msgr2=0x7f8efc0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 1 -- 172.21.15.171:0/3109679704 shutdown_connections 2024-09-16T04:09:11.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.089+0000 7f8f0248d640 1 -- 172.21.15.171:0/3109679704 wait complete. 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient: monmap: 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:11.103 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 1 Processor -- start 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 1 -- start start 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient: init 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient(hunting): picked mon.noname-c con 0x7f8efc09ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient(hunting): picked mon.noname-b con 0x7f8efc132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.090+0000 7f8f0248d640 10 monclient(hunting): picked mon.noname-a con 0x7f8efc132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc124b00 con 0x7f8efc132e60 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc123e40 con 0x7f8efc132a80 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8efc124120 con 0x7f8efc09ec20 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 10 monclient(hunting): authenticate will time out at 926.596034s 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f00a2a640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8efc132e60 legacy=0x7f8efc1d1d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8efb7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8efc132a80 0x7f8efc1ce5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36088/0 (socket says 172.21.15.171:36088) 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8efb7fe640 1 -- 172.21.15.171:0/3666111166 learned_addr learned my addr 172.21.15.171:0/3666111166 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424452708 0 0) 0x7f8efc123e40 con 0x7f8efc132a80 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): my global_id is 5185 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ee8003520 con 0x7f8efc132a80 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1599751662 0 0) 0x7f8ee8003520 con 0x7f8efc132a80 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8efc123e40 con 0x7f8efc132a80 2024-09-16T04:09:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ef0004180 con 0x7f8efc132a80 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.106 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887692889 0 0) 0x7f8efc124120 con 0x7f8efc09ec20 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient(hunting): my global_id is 5216 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ee8003520 con 0x7f8efc09ec20 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1934144785 0 0) 0x7f8efc123e40 con 0x7f8efc132a80 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 >> v1:172.21.15.148:6790/0 conn(0x7f8efc09ec20 legacy=0x7f8efc1c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 >> v1:172.21.15.148:6789/0 conn(0x7f8efc132e60 legacy=0x7f8efc1d1d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 monclient: found mon.b 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc1319a0 con 0x7f8efc132a80 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 5 monclient: authenticate success, global_id 5185 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 10 monclient: _renew_subs 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8efc0a3340 con 0x7f8efc132a80 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 10 monclient: _renew_subs 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8efc122710 con 0x7f8efc132a80 2024-09-16T04:09:11.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ef0004470 con 0x7f8efc132a80 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ef0004840 con 0x7f8efc132a80 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8ef8ff9640 10 monclient: dump: 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.109 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.091+0000 7f8efa7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8ef0004aa0 con 0x7f8efc132a80 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8f0248d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8f0248d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8f0248d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8efc11c9a0 con 0x7f8efc132a80 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5253+0+0 (unknown 1284764747 0 0) 0x7f8ef0047420 con 0x7f8efc132a80 2024-09-16T04:09:11.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.092+0000 7f8efbfff640 10 monclient: get_auth_request con 0x7f8ee8038f90 auth_method 0 2024-09-16T04:09:11.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.097+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8ef0015d80 con 0x7f8efc132a80 2024-09-16T04:09:11.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.097+0000 7f8ef8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.097+0000 7f8ef8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.265+0000 7f8f0248d640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.265+0000 7f8f0248d640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.265+0000 7f8f0248d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.265+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) -- 0x7f8efc06bcf0 con 0x7f8efc132a80 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.266+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v79) ==== 80+0+177 (unknown 3666716430 0 2845383371) 0x7f8ef0015680 con 0x7f8efc132a80 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.266+0000 7f8ef8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T04:09:11.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.266+0000 7f8ef8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:11.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.368+0000 7f8f0248d640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T04:09:11.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.368+0000 7f8f0248d640 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T04:09:11.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.368+0000 7f8f0248d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.368+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) -- 0x7f8efc02d7c0 con 0x7f8efc132a80 2024-09-16T04:09:11.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.369+0000 7f8ef8ff9640 1 -- 172.21.15.171:0/3666111166 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v79) ==== 80+0+177 (unknown 3666716430 0 2845383371) 0x7f8ef00199d0 con 0x7f8efc132a80 2024-09-16T04:09:11.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.369+0000 7f8ef8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-16T04:09:11.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.369+0000 7f8ef8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:11.373 INFO:tasks.workunit.client.0.smithi171.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-16T04:09:11.373 INFO:tasks.workunit.client.0.smithi171.stdout:-16 1.00000 host host2 2024-09-16T04:09:11.373 INFO:tasks.workunit.client.0.smithi171.stdout: 9 1.00000 osd.9 down 1.00000 1.00000 2024-09-16T04:09:11.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 >> v1:172.21.15.171:6812/7327273 conn(0x7f8ee8038f90 legacy=0x7f8ee803b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 10 monclient: shutdown 2024-09-16T04:09:11.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:11.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 >> v1:172.21.15.171:6789/0 conn(0x7f8efc132a80 legacy=0x7f8efc1ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 shutdown_connections 2024-09-16T04:09:11.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 >> 172.21.15.171:0/3666111166 conn(0x7f8efc0a52a0 msgr2=0x7f8efc131ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:11.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 shutdown_connections 2024-09-16T04:09:11.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.371+0000 7f8f0248d640 1 -- 172.21.15.171:0/3666111166 wait complete. 2024-09-16T04:09:11.382 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd tree-from osd.9 2024-09-16T04:09:11.383 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:11.383 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree-from osd.9 2024-09-16T04:09:11.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.458+0000 7f4dc74d0640 10 monclient: get_monmap_and_config 2024-09-16T04:09:11.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.458+0000 7f4dc74d0640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.458+0000 7f4dc74d0640 10 monclient: monmap: 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:11.459552+0000 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:11.459552+0000 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:11.461 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 1 Processor -- start 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 1 -- start start 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 10 monclient: init 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc009f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc00a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc009e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc00a1260 con 0x7f4dc009e1d0 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.459+0000 7f4dc74d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc009d0c0 con 0x7f4dc009f640 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc009d3a0 con 0x7f4dc00a0130 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 10 monclient(hunting): authenticate will time out at 926.965036s 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc526c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4dc009f640 0x7f4dc009fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36094/0 (socket says 172.21.15.171:36094) 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc5a6d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4dc009e1d0 legacy=0x7f4dc013bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc526c640 1 -- 172.21.15.171:0/1689878986 learned_addr learned my addr 172.21.15.171:0/1689878986 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194265438 0 0) 0x7f4dc009d0c0 con 0x7f4dc009f640 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient(hunting): my global_id is 5188 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dac003520 con 0x7f4dc009f640 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3415544219 0 0) 0x7f4dac003520 con 0x7f4dc009f640 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 >> v1:172.21.15.148:6790/0 conn(0x7f4dc00a0130 legacy=0x7f4dc0138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 >> v1:172.21.15.148:6789/0 conn(0x7f4dc009e1d0 legacy=0x7f4dc013bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 monclient: found mon.noname-c 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc009d680 con 0x7f4dc009f640 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 5 monclient: authenticate success, global_id 5188 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4dbc001610 con 0x7f4dc009f640 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: dump: 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.471 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4dbc001900 con 0x7f4dc009f640 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 1 -- 172.21.15.171:0/1689878986 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4dbc001cd0 con 0x7f4dc009f640 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4dc74d0640 4 set_mon_vals no callback set 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.460+0000 7f4db7fff640 10 monclient: dump: 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 10 monclient: shutdown 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:11.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 1 -- 172.21.15.171:0/1689878986 >> v1:172.21.15.171:6789/0 conn(0x7f4dc009f640 legacy=0x7f4dc009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 1 -- 172.21.15.171:0/1689878986 shutdown_connections 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 1 -- 172.21.15.171:0/1689878986 >> 172.21.15.171:0/1689878986 conn(0x7f4dc00a52a0 msgr2=0x7f4dc00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 1 -- 172.21.15.171:0/1689878986 shutdown_connections 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 1 -- 172.21.15.171:0/1689878986 wait complete. 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.461+0000 7f4dc74d0640 10 monclient: monmap: 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.474 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 1 Processor -- start 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 1 -- start start 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient: init 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc009f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc00a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc01a7e50 con 0x7f4dc00a0130 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc01aa910 con 0x7f4dc009e1d0 2024-09-16T04:09:11.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc01c36a0 con 0x7f4dc009f640 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc74d0640 10 monclient(hunting): authenticate will time out at 926.967748s 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc526c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4dc009e1d0 0x7f4dc01c2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36096/0 (socket says 172.21.15.171:36096) 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc526c640 1 -- 172.21.15.171:0/329669057 learned_addr learned my addr 172.21.15.171:0/329669057 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4dc5a6d640 1 -- 172.21.15.171:0/329669057 >> v1:172.21.15.148:6789/0 conn(0x7f4dc00a0130 legacy=0x7f4dc01d1a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4168364342 0 0) 0x7f4dc01aa910 con 0x7f4dc009e1d0 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4db5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4db5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4db5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4db5ffb640 10 monclient(hunting): my global_id is 5191 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.462+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4da4003520 con 0x7f4dc009e1d0 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1259439930 0 0) 0x7f4dc01c36a0 con 0x7f4dc009f640 2024-09-16T04:09:11.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): my global_id is 5222 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dc01aa910 con 0x7f4dc009f640 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1491910365 0 0) 0x7f4da4003520 con 0x7f4dc009e1d0 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dc01c36a0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4dbc0015d0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): dump: 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.477 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3043311138 0 0) 0x7f4dc01c36a0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 >> v1:172.21.15.148:6790/0 conn(0x7f4dc009f640 legacy=0x7f4dc01ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 >> v1:172.21.15.148:6789/0 conn(0x7f4dc00a0130 legacy=0x7f4dc01d1a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 monclient: found mon.b 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc006ceb0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 5 monclient: authenticate success, global_id 5191 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 10 monclient: _renew_subs 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc01a8640 con 0x7f4dc009e1d0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4dbc004cc0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4dbc005090 con 0x7f4dc009e1d0 2024-09-16T04:09:11.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db5ffb640 10 monclient: dump: 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 10 monclient: _renew_subs 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4db77fe640 4 set_mon_vals no callback set 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.463+0000 7f4dc74d0640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4dc01a93c0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4dbc0052b0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5253+0+0 (unknown 1284764747 0 0) 0x7f4dbc047590 con 0x7f4dc009e1d0 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4dc74d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4dc4a6b640 10 monclient: get_auth_request con 0x7f4da4038f60 auth_method 0 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4dc74d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4dc74d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.464+0000 7f4dc74d0640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc00008d0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.469+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4dbc015ed0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.469+0000 7f4db5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.482 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.469+0000 7f4db5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:11.639 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.636+0000 7f4dc74d0640 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T04:09:11.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.636+0000 7f4dc74d0640 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T04:09:11.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.636+0000 7f4dc74d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.636+0000 7f4dc74d0640 1 -- 172.21.15.171:0/329669057 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree-from", "bucket": "osd.9"} v 0) -- 0x7f4dc01c8f60 con 0x7f4dc009e1d0 2024-09-16T04:09:11.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.637+0000 7f4db5ffb640 1 -- 172.21.15.171:0/329669057 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.9"}]=-22 "osd.9" is not a bucket v79) ==== 103+0+0 (unknown 313103928 0 0) 0x7f4dbc0157d0 con 0x7f4dc009e1d0 2024-09-16T04:09:11.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.637+0000 7f4db5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.9"}] 2024-09-16T04:09:11.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.637+0000 7f4db5ffb640 10 monclient: _finish_command 2 = mon:22 "osd.9" is not a bucket 2024-09-16T04:09:11.641 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: "osd.9" is not a bucket 2024-09-16T04:09:11.643 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:11.643 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush rm bar 2024-09-16T04:09:11.643 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:11.643 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm bar 2024-09-16T04:09:11.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.720+0000 7f09c0c3c640 10 monclient: get_monmap_and_config 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.720+0000 7f09c0c3c640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.720+0000 7f09c0c3c640 10 monclient: monmap: 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:11.720963+0000 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:11.720963+0000 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:11.722 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.720+0000 7f09c0c3c640 1 Processor -- start 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.720+0000 7f09c0c3c640 1 -- start start 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.720+0000 7f09c0c3c640 10 monclient: init 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient(hunting): picked mon.noname-b con 0x7f09bc136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient(hunting): picked mon.noname-c con 0x7f09bc139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient(hunting): picked mon.noname-a con 0x7f09bc13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc0a1340 con 0x7f09bc13d620 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc137650 con 0x7f09bc139b10 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc0a0aa0 con 0x7f09bc136b60 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09c0c3c640 10 monclient(hunting): authenticate will time out at 927.226548s 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09bad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f09bc13d620 legacy=0x7f09bc13fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b9d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f09bc139b10 0x7f09bc13bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36110/0 (socket says 172.21.15.171:36110) 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b9d9b640 1 -- 172.21.15.171:0/1607392063 learned_addr learned my addr 172.21.15.171:0/1607392063 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190341010 0 0) 0x7f09bc137650 con 0x7f09bc139b10 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b959a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b959a640 10 monclient(hunting): my global_id is 5194 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.721+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09a4003520 con 0x7f09bc139b10 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3802866243 0 0) 0x7f09a4003520 con 0x7f09bc139b10 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 >> v1:172.21.15.148:6790/0 conn(0x7f09bc136b60 legacy=0x7f09bc136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 >> v1:172.21.15.148:6789/0 conn(0x7f09bc13d620 legacy=0x7f09bc13fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 monclient: found mon.noname-c 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09bc0a0550 con 0x7f09bc139b10 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 5 monclient: authenticate success, global_id 5194 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f09b0001610 con 0x7f09bc139b10 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: dump: 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09b00018f0 con 0x7f09bc139b10 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 1 -- 172.21.15.171:0/1607392063 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f09b0001cc0 con 0x7f09bc139b10 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 4 set_mon_vals no callback set 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09b959a640 10 monclient: dump: 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 10 monclient: shutdown 2024-09-16T04:09:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 1 -- 172.21.15.171:0/1607392063 >> v1:172.21.15.171:6789/0 conn(0x7f09bc139b10 legacy=0x7f09bc13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 1 -- 172.21.15.171:0/1607392063 shutdown_connections 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 1 -- 172.21.15.171:0/1607392063 >> 172.21.15.171:0/1607392063 conn(0x7f09bc0a98b0 msgr2=0x7f09bc0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 1 -- 172.21.15.171:0/1607392063 shutdown_connections 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.722+0000 7f09c0c3c640 1 -- 172.21.15.171:0/1607392063 wait complete. 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.723+0000 7f09c0c3c640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.723+0000 7f09c0c3c640 10 monclient: monmap: 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.723+0000 7f09c0c3c640 1 Processor -- start 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.723+0000 7f09c0c3c640 1 -- start start 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.723+0000 7f09c0c3c640 10 monclient: init 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient(hunting): picked mon.noname-c con 0x7f09bc136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient(hunting): picked mon.noname-a con 0x7f09bc139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient(hunting): picked mon.noname-b con 0x7f09bc13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc1317e0 con 0x7f09bc139b10 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc1328e0 con 0x7f09bc13d620 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09bc0a89f0 con 0x7f09bc136b60 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 10 monclient(hunting): authenticate will time out at 927.229347s 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09b9d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f09bc139b10 legacy=0x7f09bc1310d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09bad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f09bc13d620 0x7f09bc1d3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36124/0 (socket says 172.21.15.171:36124) 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09bad9d640 1 -- 172.21.15.171:0/2460250694 learned_addr learned my addr 172.21.15.171:0/2460250694 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819136937 0 0) 0x7f09bc1328e0 con 0x7f09bc13d620 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): my global_id is 5197 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f099c003520 con 0x7f09bc13d620 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2957480187 0 0) 0x7f09bc0a89f0 con 0x7f09bc136b60 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): my global_id is 5228 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09bc1328e0 con 0x7f09bc136b60 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1340128111 0 0) 0x7f099c003520 con 0x7f09bc13d620 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09bc0a89f0 con 0x7f09bc13d620 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f09b4004440 con 0x7f09bc13d620 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): dump: 2024-09-16T04:09:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2176796936 0 0) 0x7f09bc0a89f0 con 0x7f09bc13d620 2024-09-16T04:09:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 >> v1:172.21.15.148:6790/0 conn(0x7f09bc136b60 legacy=0x7f09bc1309c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 >> v1:172.21.15.148:6789/0 conn(0x7f09bc139b10 legacy=0x7f09bc1310d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 monclient: found mon.b 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09bc0a0240 con 0x7f09bc13d620 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.724+0000 7f09c0c3c640 5 monclient: authenticate success, global_id 5197 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 monclient: _renew_subs 2024-09-16T04:09:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09bc09aa20 con 0x7f09bc13d620 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 monclient: _renew_subs 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f09bc09afe0 con 0x7f09bc13d620 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f09b40046f0 con 0x7f09bc13d620 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f09b4004ac0 con 0x7f09bc13d620 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09b8d99640 4 set_mon_vals no callback set 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 10 monclient: dump: 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f09b4004c60 con 0x7f09bc13d620 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.725+0000 7f09c0c3c640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a0005080 con 0x7f09bc13d620 2024-09-16T04:09:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.726+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5253+0+0 (unknown 1284764747 0 0) 0x7f09b40476e0 con 0x7f09bc13d620 2024-09-16T04:09:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.726+0000 7f09ba59c640 10 monclient: get_auth_request con 0x7f099c038c10 auth_method 0 2024-09-16T04:09:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.730+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f09b4015f00 con 0x7f09bc13d620 2024-09-16T04:09:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.730+0000 7f09aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.730+0000 7f09aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:11.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.900+0000 7f09c0c3c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.900+0000 7f09c0c3c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.900+0000 7f09c0c3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.900+0000 7f09c0c3c640 1 -- 172.21.15.171:0/2460250694 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7f09a0005370 con 0x7f09bc13d620 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.901+0000 7f09aaffd640 1 -- 172.21.15.171:0/2460250694 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v79) ==== 99+0+0 (unknown 2325897860 0 0) 0x7f09b4015800 con 0x7f09bc13d620 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.901+0000 7f09aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.901+0000 7f09aaffd640 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-16T04:09:11.903 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-16T04:09:11.906 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:11.907 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush unlink host2 2024-09-16T04:09:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.983+0000 7fdec2fa8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.983+0000 7fdec2fa8640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.983+0000 7fdec2fa8640 10 monclient: monmap: 2024-09-16T04:09:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.985 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:11.984448+0000 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:11.984448+0000 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:11.986 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 1 Processor -- start 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 1 -- start start 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 10 monclient: init 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:11.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.984+0000 7fdec2fa8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc1332b0 con 0x7fdebc09ec20 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc134310 con 0x7fdebc132e60 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc09d980 con 0x7fdebc132a80 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 10 monclient(hunting): authenticate will time out at 927.490040s 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb3fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdebc09ec20 legacy=0x7fdebc133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec1545640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdebc132e60 0x7fdebc13c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36138/0 (socket says 172.21.15.171:36138) 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec1545640 1 -- 172.21.15.171:0/3270486569 learned_addr learned my addr 172.21.15.171:0/3270486569 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026505320 0 0) 0x7fdebc134310 con 0x7fdebc132e60 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:11.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient(hunting): my global_id is 5200 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea4003520 con 0x7fdebc132e60 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 945069768 0 0) 0x7fdea4003520 con 0x7fdebc132e60 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 >> v1:172.21.15.148:6790/0 conn(0x7fdebc132a80 legacy=0x7fdebc1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 >> v1:172.21.15.148:6789/0 conn(0x7fdebc09ec20 legacy=0x7fdebc133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc09dc60 con 0x7fdebc132e60 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:11.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 5 monclient: authenticate success, global_id 5200 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdeb4001990 con 0x7fdebc132e60 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: dump: 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.996 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdeb4001c80 con 0x7fdebc132e60 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 1 -- 172.21.15.171:0/3270486569 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdeb4002090 con 0x7fdebc132e60 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdec2fa8640 4 set_mon_vals no callback set 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.985+0000 7fdeb37fe640 10 monclient: dump: 2024-09-16T04:09:11.997 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 10 monclient: shutdown 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 1 -- 172.21.15.171:0/3270486569 >> v1:172.21.15.171:6789/0 conn(0x7fdebc132e60 legacy=0x7fdebc13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 1 -- 172.21.15.171:0/3270486569 shutdown_connections 2024-09-16T04:09:11.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 1 -- 172.21.15.171:0/3270486569 >> 172.21.15.171:0/3270486569 conn(0x7fdebc0a52a0 msgr2=0x7fdebc0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 1 -- 172.21.15.171:0/3270486569 shutdown_connections 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 1 -- 172.21.15.171:0/3270486569 wait complete. 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 10 monclient: build_initial_monmap 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.986+0000 7fdec2fa8640 10 monclient: monmap: 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:11.999 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 1 Processor -- start 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 1 -- start start 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient: init 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc13dbe0 con 0x7fdebc132a80 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc13a140 con 0x7fdebc09ec20 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc13a420 con 0x7fdebc132e60 2024-09-16T04:09:12.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec2fa8640 10 monclient(hunting): authenticate will time out at 927.492872s 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec0d44640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdebc09ec20 0x7fdebc136770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36154/0 (socket says 172.21.15.171:36154) 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdeb3fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdebc132a80 legacy=0x7fdebc136e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.987+0000 7fdec0d44640 1 -- 172.21.15.171:0/842061044 learned_addr learned my addr 172.21.15.171:0/842061044 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2725811475 0 0) 0x7fdebc13a140 con 0x7fdebc09ec20 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): my global_id is 5203 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea8003520 con 0x7fdebc09ec20 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484837286 0 0) 0x7fdebc13a420 con 0x7fdebc132e60 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:12.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): my global_id is 5234 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdebc13a140 con 0x7fdebc132e60 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3845959738 0 0) 0x7fdea8003520 con 0x7fdebc09ec20 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdebc13a420 con 0x7fdebc09ec20 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdeb8001970 con 0x7fdebc09ec20 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): dump: 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:12.002 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1959846287 0 0) 0x7fdebc13a140 con 0x7fdebc132e60 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea8003520 con 0x7fdebc132e60 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdeb40017b0 con 0x7fdebc132e60 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:12.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient(hunting): dump: 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 495697768 0 0) 0x7fdebc13a420 con 0x7fdebc09ec20 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 >> v1:172.21.15.148:6790/0 conn(0x7fdebc132e60 legacy=0x7fdebc1d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:12.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 >> v1:172.21.15.148:6789/0 conn(0x7fdebc132a80 legacy=0x7fdebc136e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 monclient: found mon.b 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc137590 con 0x7fdebc09ec20 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 5 monclient: authenticate success, global_id 5203 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 10 monclient: _renew_subs 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc1391b0 con 0x7fdebc09ec20 2024-09-16T04:09:12.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 10 monclient: _renew_subs 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdebc1d2d10 con 0x7fdebc09ec20 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdeb8001c70 con 0x7fdebc09ec20 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdeb8003a60 con 0x7fdebc09ec20 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb2ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.988+0000 7fdeb17fa640 10 monclient: dump: 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:12.006 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdeb8003c00 con 0x7fdebc09ec20 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdec2fa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdec2fa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdec2fa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde9c005080 con 0x7fdebc09ec20 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(79..79 src has 1..79) ==== 5253+0+0 (unknown 1284764747 0 0) 0x7fdeb8047450 con 0x7fdebc09ec20 2024-09-16T04:09:12.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.989+0000 7fdeb3fff640 10 monclient: get_auth_request con 0x7fdea8038f60 auth_method 0 2024-09-16T04:09:12.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.994+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdeb8015cf0 con 0x7fdebc09ec20 2024-09-16T04:09:12.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.994+0000 7fdeb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:12.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:11.994+0000 7fdeb17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.166+0000 7fdec2fa8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T04:09:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.166+0000 7fdec2fa8640 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T04:09:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.166+0000 7fdec2fa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.166+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) -- 0x7fde9c005370 con 0x7fdebc09ec20 2024-09-16T04:09:12.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.277+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v80) ==== 128+0+0 (unknown 385943926 0 0) 0x7fdeb80155f0 con 0x7fdebc09ec20 2024-09-16T04:09:12.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.277+0000 7fdeb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T04:09:12.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.277+0000 7fdeb17fa640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T04:09:12.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.382+0000 7fdec2fa8640 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T04:09:12.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.382+0000 7fdec2fa8640 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T04:09:12.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.382+0000 7fdec2fa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:12.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:12.382+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) -- 0x7fde9c002210 con 0x7fdebc09ec20 2024-09-16T04:09:13.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.279+0000 7fdeb17fa640 1 -- 172.21.15.171:0/842061044 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v81) ==== 128+0+0 (unknown 2123614164 0 0) 0x7fdeb801e1a0 con 0x7fdebc09ec20 2024-09-16T04:09:13.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.279+0000 7fdeb17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-16T04:09:13.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.279+0000 7fdeb17fa640 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T04:09:13.282 INFO:tasks.workunit.client.0.smithi171.stderr:removed item id -16 name 'host2' from crush map 2024-09-16T04:09:13.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.282+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 >> v1:172.21.15.171:6812/7327273 conn(0x7fdea8038f60 legacy=0x7fdea803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:13.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.282+0000 7fdec2fa8640 10 monclient: shutdown 2024-09-16T04:09:13.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.282+0000 7fdec2fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:13.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.282+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 >> v1:172.21.15.171:6789/0 conn(0x7fdebc09ec20 legacy=0x7fdebc136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:13.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.282+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 shutdown_connections 2024-09-16T04:09:13.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.282+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 >> 172.21.15.171:0/842061044 conn(0x7fdebc0a52a0 msgr2=0x7fdebc0a3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:13.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.283+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 shutdown_connections 2024-09-16T04:09:13.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.283+0000 7fdec2fa8640 1 -- 172.21.15.171:0/842061044 wait complete. 2024-09-16T04:09:13.296 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.381+0000 7fe2fbdc4640 10 monclient: get_monmap_and_config 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.381+0000 7fe2fbdc4640 10 monclient: build_initial_monmap 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.381+0000 7fe2fbdc4640 10 monclient: monmap: 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:13.382218+0000 2024-09-16T04:09:13.383 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:13.382218+0000 2024-09-16T04:09:13.384 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:13.384 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:13.384 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:13.384 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:13.384 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:13.384 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:13.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 1 Processor -- start 2024-09-16T04:09:13.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 1 -- start start 2024-09-16T04:09:13.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient: init 2024-09-16T04:09:13.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f4133700 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f41341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f413a350 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f409da90 con 0x7fe2f413a350 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f409dd70 con 0x7fe2f41341f0 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f409e050 con 0x7fe2f4133700 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fbdc4640 10 monclient(hunting): authenticate will time out at 928.887849s 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2fa361640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe2f413a350 legacy=0x7fe2f413c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2f935f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe2f41341f0 0x7fe2f4138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36164/0 (socket says 172.21.15.171:36164) 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.382+0000 7fe2f935f640 1 -- 172.21.15.171:0/2734322857 learned_addr learned my addr 172.21.15.171:0/2734322857 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:13.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364428780 0 0) 0x7fe2f409dd70 con 0x7fe2f41341f0 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): _negotiate 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): my global_id is 5206 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2e0003520 con 0x7fe2f41341f0 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 861430971 0 0) 0x7fe2f409e050 con 0x7fe2f4133700 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): _negotiate 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient(hunting): my global_id is 5237 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2f409dd70 con 0x7fe2f4133700 2024-09-16T04:09:13.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1155538413 0 0) 0x7fe2e0003520 con 0x7fe2f41341f0 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 >> v1:172.21.15.148:6790/0 conn(0x7fe2f4133700 legacy=0x7fe2f4133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 >> v1:172.21.15.148:6789/0 conn(0x7fe2f413a350 legacy=0x7fe2f413c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: _finish_hunting 0 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 monclient: found mon.noname-c 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f409d4f0 con 0x7fe2f41341f0 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: _finish_auth 0 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: _check_auth_tickets 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe2f0001610 con 0x7fe2f41341f0 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2fbdc4640 5 monclient: authenticate success, global_id 5206 2024-09-16T04:09:13.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2fbdc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: dump: 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:13.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2fbdc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2f0001930 con 0x7fe2f41341f0 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 1 -- 172.21.15.171:0/2734322857 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe2f0001d00 con 0x7fe2f41341f0 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2fbdc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2fbdc4640 4 set_mon_vals no callback set 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.383+0000 7fe2f8b5e640 10 monclient: dump: 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.396 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 10 monclient: shutdown 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/2734322857 >> v1:172.21.15.171:6789/0 conn(0x7fe2f41341f0 legacy=0x7fe2f4138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/2734322857 shutdown_connections 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/2734322857 >> 172.21.15.171:0/2734322857 conn(0x7fe2f40a5450 msgr2=0x7fe2f40a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/2734322857 shutdown_connections 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/2734322857 wait complete. 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 10 monclient: build_initial_monmap 2024-09-16T04:09:13.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.384+0000 7fe2fbdc4640 10 monclient: monmap: 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 1 Processor -- start 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 1 -- start start 2024-09-16T04:09:13.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient: init 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f4133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f41341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f413a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f411eb90 con 0x7fe2f41341f0 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f411ee70 con 0x7fe2f4133700 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f409b6d0 con 0x7fe2f413a350 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2fbdc4640 10 monclient(hunting): authenticate will time out at 928.890527s 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2f9b60640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe2f4133700 0x7fe2f41320a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36180/0 (socket says 172.21.15.171:36180) 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2f9b60640 1 -- 172.21.15.171:0/3801592074 learned_addr learned my addr 172.21.15.171:0/3801592074 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2f935f640 1 -- 172.21.15.171:0/3801592074 >> v1:172.21.15.148:6789/0 conn(0x7fe2f41341f0 legacy=0x7fe2f41ce930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:13.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684283567 0 0) 0x7fe2f411ee70 con 0x7fe2f4133700 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 10 monclient(hunting): my global_id is 5209 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d8003520 con 0x7fe2f4133700 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.385+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1961385804 0 0) 0x7fe2d8003520 con 0x7fe2f4133700 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2f411ee70 con 0x7fe2f4133700 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe2e4001910 con 0x7fe2f4133700 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:13.400 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863287052 0 0) 0x7fe2f409b6d0 con 0x7fe2f413a350 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:13.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient(hunting): my global_id is 5240 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d8003520 con 0x7fe2f413a350 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1891219390 0 0) 0x7fe2f411ee70 con 0x7fe2f4133700 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 >> v1:172.21.15.148:6790/0 conn(0x7fe2f413a350 legacy=0x7fe2f41d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 >> v1:172.21.15.148:6789/0 conn(0x7fe2f41341f0 legacy=0x7fe2f41ce930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 monclient: found mon.b 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f409c0b0 con 0x7fe2f4133700 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 5 monclient: authenticate success, global_id 5209 2024-09-16T04:09:13.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 10 monclient: _renew_subs 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f40a2ee0 con 0x7fe2f4133700 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 10 monclient: _renew_subs 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2f411d460 con 0x7fe2f4133700 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2e4003940 con 0x7fe2f4133700 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe2e4003d10 con 0x7fe2f4133700 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ebfff640 4 set_mon_vals no callback set 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.386+0000 7fe2ea7fc640 10 monclient: dump: 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:13.403 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe2e4003eb0 con 0x7fe2f4133700 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2fbdc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2fbdc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2fbdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2dc005080 con 0x7fe2f4133700 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(81..81 src has 1..81) ==== 5205+0+0 (unknown 1996043578 0 0) 0x7fe2e40474d0 con 0x7fe2f4133700 2024-09-16T04:09:13.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.387+0000 7fe2f935f640 10 monclient: get_auth_request con 0x7fe2d8038f90 auth_method 0 2024-09-16T04:09:13.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.391+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe2e4015de0 con 0x7fe2f4133700 2024-09-16T04:09:13.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.391+0000 7fe2ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:13.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.391+0000 7fe2ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:13.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.559+0000 7fe2fbdc4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T04:09:13.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.559+0000 7fe2fbdc4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T04:09:13.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.559+0000 7fe2fbdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:13.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:13.559+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) -- 0x7fe2dc005370 con 0x7fe2f4133700 2024-09-16T04:09:14.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.288+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v82) ==== 251+0+0 (unknown 1717332439 0 0) 0x7fe2e40156e0 con 0x7fe2f4133700 2024-09-16T04:09:14.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.288+0000 7fe2ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T04:09:14.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.288+0000 7fe2ea7fc640 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-16T04:09:14.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2fbdc4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T04:09:14.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2fbdc4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T04:09:14.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2fbdc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) -- 0x7fe2dc005d70 con 0x7fe2f4133700 2024-09-16T04:09:14.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2ea7fc640 1 -- 172.21.15.171:0/3801592074 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v82) ==== 200+0+0 (unknown 385475284 0 0) 0x7fe2e401e290 con 0x7fe2f4133700 2024-09-16T04:09:14.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2ea7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-16T04:09:14.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.394+0000 7fe2ea7fc640 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-16T04:09:14.398 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'host-for-test' already exists 2024-09-16T04:09:14.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 >> v1:172.21.15.171:6812/7327273 conn(0x7fe2d8038f90 legacy=0x7fe2d803b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 10 monclient: shutdown 2024-09-16T04:09:14.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:14.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 >> v1:172.21.15.171:6789/0 conn(0x7fe2f4133700 legacy=0x7fe2f41320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 shutdown_connections 2024-09-16T04:09:14.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 >> 172.21.15.171:0/3801592074 conn(0x7fe2f40a5450 msgr2=0x7fe2f409f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:14.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 shutdown_connections 2024-09-16T04:09:14.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.396+0000 7fe2fbdc4640 1 -- 172.21.15.171:0/3801592074 wait complete. 2024-09-16T04:09:14.408 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:14.408 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep host-for-test 2024-09-16T04:09:14.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.483+0000 7fa0747e8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:14.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.483+0000 7fa0747e8640 10 monclient: build_initial_monmap 2024-09-16T04:09:14.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.483+0000 7fa0747e8640 10 monclient: monmap: 2024-09-16T04:09:14.501 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:14.501 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.501 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:14.483851+0000 2024-09-16T04:09:14.502 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:14.483851+0000 2024-09-16T04:09:14.502 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:14.502 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.502 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:14.502 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:14.502 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:14.503 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.483+0000 7fa0747e8640 1 Processor -- start 2024-09-16T04:09:14.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.483+0000 7fa0747e8640 1 -- start start 2024-09-16T04:09:14.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.483+0000 7fa0747e8640 10 monclient: init 2024-09-16T04:09:14.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:14.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:14.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa06c136cf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:14.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa06c139ad0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:14.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa06c13d5e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:14.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0a4f90 con 0x7fa06c13d5e0 2024-09-16T04:09:14.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0a0a30 con 0x7fa06c139ad0 2024-09-16T04:09:14.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0a0d10 con 0x7fa06c136cf0 2024-09-16T04:09:14.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:14.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa0747e8640 10 monclient(hunting): authenticate will time out at 929.989479s 2024-09-16T04:09:14.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa072d85640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa06c13d5e0 legacy=0x7fa06c13f9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:14.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071d83640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa06c139ad0 0x7fa06c13bec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36182/0 (socket says 172.21.15.171:36182) 2024-09-16T04:09:14.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071d83640 1 -- 172.21.15.171:0/4200209437 learned_addr learned my addr 172.21.15.171:0/4200209437 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:14.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135335438 0 0) 0x7fa06c0a0a30 con 0x7fa06c139ad0 2024-09-16T04:09:14.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071582640 10 monclient(hunting): _negotiate 2024-09-16T04:09:14.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071582640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071582640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:14.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071582640 10 monclient(hunting): my global_id is 5212 2024-09-16T04:09:14.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.484+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa054003520 con 0x7fa06c139ad0 2024-09-16T04:09:14.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 556980264 0 0) 0x7fa054003520 con 0x7fa06c139ad0 2024-09-16T04:09:14.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 >> v1:172.21.15.148:6790/0 conn(0x7fa06c136cf0 legacy=0x7fa06c1370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 >> v1:172.21.15.148:6789/0 conn(0x7fa06c13d5e0 legacy=0x7fa06c13f9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:14.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: _finish_hunting 0 2024-09-16T04:09:14.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 monclient: found mon.noname-c 2024-09-16T04:09:14.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:14.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c0a0ff0 con 0x7fa06c139ad0 2024-09-16T04:09:14.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: _finish_auth 0 2024-09-16T04:09:14.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: _check_auth_tickets 2024-09-16T04:09:14.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:14.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa060001610 con 0x7fa06c139ad0 2024-09-16T04:09:14.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 5 monclient: authenticate success, global_id 5212 2024-09-16T04:09:14.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:14.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:14.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: dump: 2024-09-16T04:09:14.510 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.510 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.510 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.510 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.510 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.511 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.511 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.511 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.511 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.511 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:14.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa060001900 con 0x7fa06c139ad0 2024-09-16T04:09:14.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:14.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 1 -- 172.21.15.171:0/4200209437 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa060001d10 con 0x7fa06c139ad0 2024-09-16T04:09:14.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:14.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:14.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa071582640 10 monclient: dump: 2024-09-16T04:09:14.513 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.513 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.513 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.513 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.513 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.514 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.514 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.514 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.514 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.514 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:14.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 4 set_mon_vals no callback set 2024-09-16T04:09:14.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 10 monclient: shutdown 2024-09-16T04:09:14.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:14.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 1 -- 172.21.15.171:0/4200209437 >> v1:172.21.15.171:6789/0 conn(0x7fa06c139ad0 legacy=0x7fa06c13bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 1 -- 172.21.15.171:0/4200209437 shutdown_connections 2024-09-16T04:09:14.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 1 -- 172.21.15.171:0/4200209437 >> 172.21.15.171:0/4200209437 conn(0x7fa06c0a9830 msgr2=0x7fa06c0a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 1 -- 172.21.15.171:0/4200209437 shutdown_connections 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.485+0000 7fa0747e8640 1 -- 172.21.15.171:0/4200209437 wait complete. 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.486+0000 7fa0747e8640 10 monclient: build_initial_monmap 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.486+0000 7fa0747e8640 10 monclient: monmap: 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:14.516 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.486+0000 7fa0747e8640 1 Processor -- start 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.486+0000 7fa0747e8640 1 -- start start 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.486+0000 7fa0747e8640 10 monclient: init 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa06c136cf0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:14.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa06c139ad0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa06c13d5e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c120a20 con 0x7fa06c13d5e0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c120d00 con 0x7fa06c136cf0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06c0a6eb0 con 0x7fa06c139ad0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient(hunting): authenticate will time out at 929.992166s 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa072584640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa06c136cf0 0x7fa06c134df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36198/0 (socket says 172.21.15.171:36198) 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa072d85640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa06c13d5e0 legacy=0x7fa06c1d40d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa072584640 1 -- 172.21.15.171:0/1672324594 learned_addr learned my addr 172.21.15.171:0/1672324594 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110845372 0 0) 0x7fa06c120d00 con 0x7fa06c136cf0 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): my global_id is 5215 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa058003520 con 0x7fa06c136cf0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668358094 0 0) 0x7fa06c0a6eb0 con 0x7fa06c139ad0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): my global_id is 5246 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa06c120d00 con 0x7fa06c139ad0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 28378788 0 0) 0x7fa058003520 con 0x7fa06c136cf0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa06c0a6eb0 con 0x7fa06c136cf0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa05c001cf0 con 0x7fa06c136cf0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:14.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): dump: 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:14.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1938514297 0 0) 0x7fa06c120d00 con 0x7fa06c139ad0 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa058003520 con 0x7fa06c139ad0 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa060001430 con 0x7fa06c139ad0 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient(hunting): dump: 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.521 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 743238572 0 0) 0x7fa06c0a6eb0 con 0x7fa06c136cf0 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 >> v1:172.21.15.148:6790/0 conn(0x7fa06c139ad0 legacy=0x7fa06c1d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 >> v1:172.21.15.148:6789/0 conn(0x7fa06c13d5e0 legacy=0x7fa06c1d40d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 monclient: found mon.b 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c0a8ae0 con 0x7fa06c136cf0 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient: _finish_auth 0 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:14.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa06affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 5 monclient: authenticate success, global_id 5215 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient: _renew_subs 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa06c120740 con 0x7fa06c136cf0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient: _renew_subs 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.487+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa06c09e4e0 con 0x7fa06c136cf0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa05c003c40 con 0x7fa06c136cf0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa06affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa05c003df0 con 0x7fa06c136cf0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa06affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:14.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa06affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa06affd640 10 monclient: dump: 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa070d81640 4 set_mon_vals no callback set 2024-09-16T04:09:14.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa0747e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa0747e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa0747e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.488+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04c005080 con 0x7fa06c136cf0 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.489+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa05c004010 con 0x7fa06c136cf0 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.489+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5730+0+0 (unknown 3353785277 0 0) 0x7fa05c047920 con 0x7fa06c136cf0 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.489+0000 7fa071d83640 10 monclient: get_auth_request con 0x7fa058038ea0 auth_method 0 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.493+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa05c0160e0 con 0x7fa06c136cf0 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.493+0000 7fa06affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:14.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.493+0000 7fa06affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa0747e8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa0747e8640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa0747e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fa04c005b80 con 0x7fa06c136cf0 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1500 (unknown 2059188187 0 1015378484) 0x7fa05c0159e0 con 0x7fa06c136cf0 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa06affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:14.662 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.660+0000 7fa06affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:14.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa0747e8640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:14.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa0747e8640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:14.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa0747e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fa04c005d50 con 0x7fa06c136cf0 2024-09-16T04:09:14.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa06affd640 1 -- 172.21.15.171:0/1672324594 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1500 (unknown 2059188187 0 1015378484) 0x7fa05c01e5d0 con 0x7fa06c136cf0 2024-09-16T04:09:14.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa06affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:14.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.759+0000 7fa06affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:14.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 >> v1:172.21.15.171:6812/7327273 conn(0x7fa058038ea0 legacy=0x7fa05803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 10 monclient: shutdown 2024-09-16T04:09:14.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:14.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 >> v1:172.21.15.171:6789/0 conn(0x7fa06c136cf0 legacy=0x7fa06c134df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 shutdown_connections 2024-09-16T04:09:14.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 >> 172.21.15.171:0/1672324594 conn(0x7fa06c0a9830 msgr2=0x7fa06c1391d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:14.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 shutdown_connections 2024-09-16T04:09:14.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.761+0000 7fa0747e8640 1 -- 172.21.15.171:0/1672324594 wait complete. 2024-09-16T04:09:14.772 INFO:tasks.workunit.client.0.smithi171.stdout:-11 0 host host-for-test 2024-09-16T04:09:14.773 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:14.773 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep rack-for-test 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.848+0000 7f5ba17c4640 10 monclient: get_monmap_and_config 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.848+0000 7f5ba17c4640 10 monclient: build_initial_monmap 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.848+0000 7f5ba17c4640 10 monclient: monmap: 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:14.849297+0000 2024-09-16T04:09:14.850 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:14.849297+0000 2024-09-16T04:09:14.851 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:14.851 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.851 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:14.851 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:14.851 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:14.851 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 1 Processor -- start 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 1 -- start start 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient: init 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient(hunting): picked mon.noname-b con 0x7f5b9c136cf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:14.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9c139ad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient(hunting): picked mon.noname-c con 0x7f5b9c13d5e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c0a4f90 con 0x7f5b9c139ad0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c0a0a30 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c0a0d10 con 0x7f5b9c136cf0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5ba17c4640 10 monclient(hunting): authenticate will time out at 930.354804s 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5b93fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f5b9c139ad0 legacy=0x7f5b9c13bec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5b9b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5b9c13d5e0 0x7f5b9c13f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36210/0 (socket says 172.21.15.171:36210) 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.849+0000 7f5b9b7fe640 1 -- 172.21.15.171:0/1645523570 learned_addr learned my addr 172.21.15.171:0/1645523570 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465914795 0 0) 0x7f5b9c0a0a30 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:14.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient(hunting): my global_id is 5218 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b88003520 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3099597548 0 0) 0x7f5b88003520 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 >> v1:172.21.15.148:6790/0 conn(0x7f5b9c136cf0 legacy=0x7f5b9c1370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 >> v1:172.21.15.148:6789/0 conn(0x7f5b9c139ad0 legacy=0x7f5b9c13bec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c0a0ff0 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:14.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b94001990 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: dump: 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.860 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 5 monclient: authenticate success, global_id 5218 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b94001c80 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 1 -- 172.21.15.171:0/1645523570 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b94002050 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 4 set_mon_vals no callback set 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5b9a7fc640 10 monclient: dump: 2024-09-16T04:09:14.861 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 10 monclient: shutdown 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.850+0000 7f5ba17c4640 1 -- 172.21.15.171:0/1645523570 >> v1:172.21.15.171:6789/0 conn(0x7f5b9c13d5e0 legacy=0x7f5b9c13f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 1 -- 172.21.15.171:0/1645523570 shutdown_connections 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 1 -- 172.21.15.171:0/1645523570 >> 172.21.15.171:0/1645523570 conn(0x7f5b9c0a9830 msgr2=0x7f5b9c0a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 1 -- 172.21.15.171:0/1645523570 shutdown_connections 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 1 -- 172.21.15.171:0/1645523570 wait complete. 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 10 monclient: build_initial_monmap 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 10 monclient: monmap: 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:14.863 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 1 Processor -- start 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.851+0000 7f5ba17c4640 1 -- start start 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient: init 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient(hunting): picked mon.noname-c con 0x7f5b9c136cf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9c139ad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient(hunting): picked mon.noname-b con 0x7f5b9c13d5e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c09f630 con 0x7f5b9c139ad0 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c09d800 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9c072f70 con 0x7f5b9c136cf0 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5ba17c4640 10 monclient(hunting): authenticate will time out at 930.357503s 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b9b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5b9c13d5e0 0x7f5b9c1d40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36216/0 (socket says 172.21.15.171:36216) 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b9b7fe640 1 -- 172.21.15.171:0/3770421613 learned_addr learned my addr 172.21.15.171:0/3770421613 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b93fff640 1 -- 172.21.15.171:0/3770421613 >> v1:172.21.15.148:6789/0 conn(0x7f5b9c139ad0 legacy=0x7f5b9c1d09a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3673750711 0 0) 0x7f5b9c09d800 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): my global_id is 5221 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b7c003520 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.865 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453711153 0 0) 0x7f5b9c072f70 con 0x7f5b9c136cf0 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): my global_id is 5252 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b9c09d800 con 0x7f5b9c136cf0 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1311361549 0 0) 0x7f5b7c003520 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b9c072f70 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b940017b0 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): dump: 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.867 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.852+0000 7f5b937fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3057414655 0 0) 0x7f5b9c072f70 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 >> v1:172.21.15.148:6790/0 conn(0x7f5b9c136cf0 legacy=0x7f5b9c1cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 >> v1:172.21.15.148:6789/0 conn(0x7f5b9c139ad0 legacy=0x7f5b9c1d09a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 monclient: found mon.b 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c0a01d0 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:14.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 5 monclient: authenticate success, global_id 5221 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 10 monclient: _renew_subs 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9c09cf20 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 10 monclient: _renew_subs 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b9c09dca0 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b94005040 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b94005390 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:14.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 10 monclient: dump: 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b99ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.853+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5b940107c0 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.854+0000 7f5ba17c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.854+0000 7f5ba17c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.854+0000 7f5ba17c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.854+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b9c0008d0 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.854+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5730+0+0 (unknown 3353785277 0 0) 0x7f5b94047a90 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.854+0000 7f5b9affd640 10 monclient: get_auth_request con 0x7f5b7c038b90 auth_method 0 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.858+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5b94016190 con 0x7f5b9c13d5e0 2024-09-16T04:09:14.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.858+0000 7f5b937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:14.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:14.858+0000 7f5b937fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:15.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.023+0000 7f5ba17c4640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:15.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.023+0000 7f5ba17c4640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.023+0000 7f5ba17c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.023+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f5b9c073810 con 0x7f5b9c13d5e0 2024-09-16T04:09:15.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.024+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1500 (unknown 2059188187 0 1015378484) 0x7f5b94015a90 con 0x7f5b9c13d5e0 2024-09-16T04:09:15.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.024+0000 7f5b937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.024+0000 7f5b937fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:15.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5ba17c4640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:15.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5ba17c4640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5ba17c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f5b9c09ee90 con 0x7f5b9c13d5e0 2024-09-16T04:09:15.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5b937fe640 1 -- 172.21.15.171:0/3770421613 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1500 (unknown 2059188187 0 1015378484) 0x7f5b9401e5e0 con 0x7f5b9c13d5e0 2024-09-16T04:09:15.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5b937fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.122+0000 7f5b937fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 >> v1:172.21.15.171:6812/7327273 conn(0x7f5b7c038b90 legacy=0x7f5b7c03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 10 monclient: shutdown 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 >> v1:172.21.15.171:6789/0 conn(0x7f5b9c13d5e0 legacy=0x7f5b9c1d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 shutdown_connections 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 >> 172.21.15.171:0/3770421613 conn(0x7f5b9c0a9830 msgr2=0x7f5b9c0a8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 shutdown_connections 2024-09-16T04:09:15.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.125+0000 7f5ba17c4640 1 -- 172.21.15.171:0/3770421613 wait complete. 2024-09-16T04:09:15.136 INFO:tasks.workunit.client.0.smithi171.stdout:-12 0 rack rack-for-test 2024-09-16T04:09:15.136 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:15.136 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep root-for-test 2024-09-16T04:09:15.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.210+0000 7f9cd7a77640 10 monclient: get_monmap_and_config 2024-09-16T04:09:15.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.210+0000 7f9cd7a77640 10 monclient: build_initial_monmap 2024-09-16T04:09:15.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.210+0000 7f9cd7a77640 10 monclient: monmap: 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:15.211351+0000 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:15.211351+0000 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:15.213 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 1 Processor -- start 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 1 -- start start 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient: init 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd0134ad0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd0137a20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient(hunting): picked mon.noname-a con 0x7f9cd013b530 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd00a0a50 con 0x7f9cd013b530 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd00a0cf0 con 0x7f9cd0134ad0 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd01355c0 con 0x7f9cd0137a20 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:15.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.211+0000 7f9cd7a77640 10 monclient(hunting): authenticate will time out at 930.716989s 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd5813640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9cd0134ad0 0x7f9cd0134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36232/0 (socket says 172.21.15.171:36232) 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd5813640 1 -- 172.21.15.171:0/3467704148 learned_addr learned my addr 172.21.15.171:0/3467704148 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd6014640 1 -- 172.21.15.171:0/3467704148 >> v1:172.21.15.148:6789/0 conn(0x7f9cd013b530 legacy=0x7f9cd013d920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089306759 0 0) 0x7f9cd00a0cf0 con 0x7f9cd0134ad0 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient(hunting): _negotiate 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient(hunting): my global_id is 5224 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cbc003520 con 0x7f9cd0134ad0 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1504815479 0 0) 0x7f9cbc003520 con 0x7f9cd0134ad0 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 >> v1:172.21.15.148:6790/0 conn(0x7f9cd0137a20 legacy=0x7f9cd0139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 >> v1:172.21.15.148:6789/0 conn(0x7f9cd013b530 legacy=0x7f9cd013d920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: _finish_hunting 0 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 monclient: found mon.noname-c 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd009be50 con 0x7f9cd0134ad0 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: _finish_auth 0 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: _check_auth_tickets 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ccc001630 con 0x7f9cd0134ad0 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd7a77640 5 monclient: authenticate success, global_id 5224 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:15.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: dump: 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd7a77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ccc001990 con 0x7f9cd0134ad0 2024-09-16T04:09:15.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 1 -- 172.21.15.171:0/3467704148 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ccc0048a0 con 0x7f9cd0134ad0 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd7a77640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd7a77640 4 set_mon_vals no callback set 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.212+0000 7f9cd4811640 10 monclient: dump: 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.226 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 10 monclient: shutdown 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 1 -- 172.21.15.171:0/3467704148 >> v1:172.21.15.171:6789/0 conn(0x7f9cd0134ad0 legacy=0x7f9cd0134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 1 -- 172.21.15.171:0/3467704148 shutdown_connections 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 1 -- 172.21.15.171:0/3467704148 >> 172.21.15.171:0/3467704148 conn(0x7f9cd00a1ec0 msgr2=0x7f9cd00a22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 1 -- 172.21.15.171:0/3467704148 shutdown_connections 2024-09-16T04:09:15.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 1 -- 172.21.15.171:0/3467704148 wait complete. 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 10 monclient: build_initial_monmap 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.213+0000 7f9cd7a77640 10 monclient: monmap: 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 1 Processor -- start 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 1 -- start start 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient: init 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd0134ad0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd0137a20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient(hunting): picked mon.noname-a con 0x7f9cd013b530 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd013a180 con 0x7f9cd013b530 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd01374d0 con 0x7f9cd0137a20 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd0070150 con 0x7f9cd0134ad0 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd7a77640 10 monclient(hunting): authenticate will time out at 930.719624s 2024-09-16T04:09:15.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd5012640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9cd0137a20 0x7f9cd0138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36244/0 (socket says 172.21.15.171:36244) 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd5012640 1 -- 172.21.15.171:0/1535310461 learned_addr learned my addr 172.21.15.171:0/1535310461 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cd6014640 1 -- 172.21.15.171:0/1535310461 >> v1:172.21.15.148:6789/0 conn(0x7f9cd013b530 legacy=0x7f9cd01d1700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4192517247 0 0) 0x7f9cd01355c0 con 0x7f9cd0137a20 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): my global_id is 5227 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cb4003520 con 0x7f9cd0137a20 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3722788931 0 0) 0x7f9cd0070150 con 0x7f9cd0134ad0 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 10 monclient(hunting): my global_id is 5258 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.214+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cd01355c0 con 0x7f9cd0134ad0 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4141431124 0 0) 0x7f9cb4003520 con 0x7f9cd0137a20 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9cd0070150 con 0x7f9cd0137a20 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cc0001970 con 0x7f9cd0137a20 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): dump: 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.231 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2593492317 0 0) 0x7f9cd01355c0 con 0x7f9cd0134ad0 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9cb4003520 con 0x7f9cd0134ad0 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ccc001450 con 0x7f9cd0134ad0 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient(hunting): dump: 2024-09-16T04:09:15.232 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1609897513 0 0) 0x7f9cd0070150 con 0x7f9cd0137a20 2024-09-16T04:09:15.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 >> v1:172.21.15.148:6790/0 conn(0x7f9cd0134ad0 legacy=0x7f9cd01387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 >> v1:172.21.15.148:6789/0 conn(0x7f9cd013b530 legacy=0x7f9cd01d1700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 monclient: found mon.b 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd009c190 con 0x7f9cd0137a20 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 5 monclient: authenticate success, global_id 5227 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 10 monclient: _renew_subs 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd006fe70 con 0x7f9cd0137a20 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 10 monclient: _renew_subs 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9cd013b2e0 con 0x7f9cd0137a20 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9cc00038c0 con 0x7f9cd0137a20 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cc0003af0 con 0x7f9cd0137a20 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc7fff640 4 set_mon_vals no callback set 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.215+0000 7f9cc67fc640 10 monclient: dump: 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.235 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9cc0003c90 con 0x7f9cd0137a20 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5730+0+0 (unknown 3353785277 0 0) 0x7f9cc00475e0 con 0x7f9cd0137a20 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cd7a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cd7a77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:15.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cd7a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cb8005080 con 0x7f9cd0137a20 2024-09-16T04:09:15.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.216+0000 7f9cd5813640 10 monclient: get_auth_request con 0x7f9cb4038f10 auth_method 0 2024-09-16T04:09:15.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.220+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9cd01374d0 con 0x7f9cd0137a20 2024-09-16T04:09:15.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.220+0000 7f9cc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:15.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.220+0000 7f9cc67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:15.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cd7a77640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:15.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cd7a77640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cd7a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f9cb8005b80 con 0x7f9cd0137a20 2024-09-16T04:09:15.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1500 (unknown 2059188187 0 1015378484) 0x7f9cc0015c80 con 0x7f9cd0137a20 2024-09-16T04:09:15.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.390+0000 7f9cc67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:15.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cd7a77640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:15.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cd7a77640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cd7a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f9cb8006730 con 0x7f9cd0137a20 2024-09-16T04:09:15.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cc67fc640 1 -- 172.21.15.171:0/1535310461 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v82) ==== 56+0+1500 (unknown 2059188187 0 1015378484) 0x7f9cc0015580 con 0x7f9cd0137a20 2024-09-16T04:09:15.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cc67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:15.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.489+0000 7f9cc67fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:15.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 >> v1:172.21.15.171:6812/7327273 conn(0x7f9cb4038f10 legacy=0x7f9cb403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 10 monclient: shutdown 2024-09-16T04:09:15.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:15.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 >> v1:172.21.15.171:6789/0 conn(0x7f9cd0137a20 legacy=0x7f9cd0138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 shutdown_connections 2024-09-16T04:09:15.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 >> 172.21.15.171:0/1535310461 conn(0x7f9cd00a1ec0 msgr2=0x7f9cd0133dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:15.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 shutdown_connections 2024-09-16T04:09:15.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.491+0000 7f9cd7a77640 1 -- 172.21.15.171:0/1535310461 wait complete. 2024-09-16T04:09:15.502 INFO:tasks.workunit.client.0.smithi171.stdout:-14 0 root root-for-test 2024-09-16T04:09:15.503 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm host-for-test 2024-09-16T04:09:15.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.580+0000 7fd674dc4640 10 monclient: get_monmap_and_config 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.580+0000 7fd674dc4640 10 monclient: build_initial_monmap 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.580+0000 7fd674dc4640 10 monclient: monmap: 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:15.581418+0000 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:15.581418+0000 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:15.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 1 Processor -- start 2024-09-16T04:09:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 1 -- start start 2024-09-16T04:09:15.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient: init 2024-09-16T04:09:15.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:15.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:15.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient(hunting): picked mon.noname-c con 0x7fd670132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:15.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient(hunting): picked mon.noname-a con 0x7fd67009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:15.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient(hunting): picked mon.noname-b con 0x7fd670132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:15.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6701332b0 con 0x7fd67009ec20 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd670134310 con 0x7fd670132a80 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd67009d980 con 0x7fd670132e60 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd674dc4640 10 monclient(hunting): authenticate will time out at 931.086783s 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd66e59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd670132a80 0x7fd6701334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36248/0 (socket says 172.21.15.171:36248) 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd66e59c640 1 -- 172.21.15.171:0/2922908845 learned_addr learned my addr 172.21.15.171:0/2922908845 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.581+0000 7fd665d9b640 1 -- 172.21.15.171:0/2922908845 >> v1:172.21.15.148:6789/0 conn(0x7fd67009ec20 legacy=0x7fd670133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037148638 0 0) 0x7fd670134310 con 0x7fd670132a80 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient(hunting): my global_id is 5230 2024-09-16T04:09:15.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd65c003520 con 0x7fd670132a80 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 160457144 0 0) 0x7fd65c003520 con 0x7fd670132a80 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 >> v1:172.21.15.148:6790/0 conn(0x7fd670132e60 legacy=0x7fd67013c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 >> v1:172.21.15.148:6789/0 conn(0x7fd67009ec20 legacy=0x7fd670133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 monclient: found mon.noname-c 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67009dc60 con 0x7fd670132a80 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 5 monclient: authenticate success, global_id 5230 2024-09-16T04:09:15.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd660001610 con 0x7fd670132a80 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: dump: 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.594 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd660001900 con 0x7fd670132a80 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 1 -- 172.21.15.171:0/2922908845 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd660001cd0 con 0x7fd670132a80 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 4 set_mon_vals no callback set 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd66dd9b640 10 monclient: dump: 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.595 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 10 monclient: shutdown 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 1 -- 172.21.15.171:0/2922908845 >> v1:172.21.15.171:6789/0 conn(0x7fd670132a80 legacy=0x7fd6701334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 1 -- 172.21.15.171:0/2922908845 shutdown_connections 2024-09-16T04:09:15.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.582+0000 7fd674dc4640 1 -- 172.21.15.171:0/2922908845 >> 172.21.15.171:0/2922908845 conn(0x7fd6700a52a0 msgr2=0x7fd6700a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:15.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.583+0000 7fd674dc4640 1 -- 172.21.15.171:0/2922908845 shutdown_connections 2024-09-16T04:09:15.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.583+0000 7fd674dc4640 1 -- 172.21.15.171:0/2922908845 wait complete. 2024-09-16T04:09:15.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.583+0000 7fd674dc4640 10 monclient: build_initial_monmap 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.583+0000 7fd674dc4640 10 monclient: monmap: 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.600 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 1 Processor -- start 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 1 -- start start 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient: init 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient(hunting): picked mon.noname-b con 0x7fd67009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient(hunting): picked mon.noname-c con 0x7fd670132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient(hunting): picked mon.noname-a con 0x7fd670132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd670123960 con 0x7fd670132e60 2024-09-16T04:09:15.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6701217f0 con 0x7fd67009ec20 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6700a3d90 con 0x7fd670132a80 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd674dc4640 10 monclient(hunting): authenticate will time out at 931.089711s 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd66ed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd670132e60 legacy=0x7fd6701d3e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd66e59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd67009ec20 0x7fd6701cb960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36258/0 (socket says 172.21.15.171:36258) 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd66e59c640 1 -- 172.21.15.171:0/2220430505 learned_addr learned my addr 172.21.15.171:0/2220430505 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.584+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562512621 0 0) 0x7fd6701217f0 con 0x7fd67009ec20 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): my global_id is 5233 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd650003520 con 0x7fd67009ec20 2024-09-16T04:09:15.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637121558 0 0) 0x7fd6700a3d90 con 0x7fd670132a80 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): my global_id is 5264 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6701217f0 con 0x7fd670132a80 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1072029308 0 0) 0x7fd650003520 con 0x7fd67009ec20 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd6700a3d90 con 0x7fd67009ec20 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd6600015d0 con 0x7fd67009ec20 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): dump: 2024-09-16T04:09:15.603 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1878557857 0 0) 0x7fd6701217f0 con 0x7fd670132a80 2024-09-16T04:09:15.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd650003520 con 0x7fd670132a80 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd6580040f0 con 0x7fd670132a80 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient(hunting): dump: 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.605 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3169335626 0 0) 0x7fd6700a3d90 con 0x7fd67009ec20 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 >> v1:172.21.15.148:6790/0 conn(0x7fd670132a80 legacy=0x7fd6701d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 >> v1:172.21.15.148:6789/0 conn(0x7fd670132e60 legacy=0x7fd6701d3e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 monclient: found mon.b 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67009c910 con 0x7fd67009ec20 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 5 monclient: authenticate success, global_id 5233 2024-09-16T04:09:15.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 10 monclient: _renew_subs 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6701229c0 con 0x7fd67009ec20 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 10 monclient: _renew_subs 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd670122f80 con 0x7fd67009ec20 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd660001900 con 0x7fd67009ec20 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd660004cc0 con 0x7fd67009ec20 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd66d59a640 4 set_mon_vals no callback set 2024-09-16T04:09:15.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.585+0000 7fd6677fe640 10 monclient: dump: 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd6600100f0 con 0x7fd67009ec20 2024-09-16T04:09:15.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd674dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:15.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd674dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:15.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd674dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd654005080 con 0x7fd67009ec20 2024-09-16T04:09:15.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(82..82 src has 1..82) ==== 5730+0+0 (unknown 3353785277 0 0) 0x7fd660047710 con 0x7fd67009ec20 2024-09-16T04:09:15.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.586+0000 7fd665d9b640 10 monclient: get_auth_request con 0x7fd650038f10 auth_method 0 2024-09-16T04:09:15.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.591+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd660015db0 con 0x7fd67009ec20 2024-09-16T04:09:15.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.591+0000 7fd6677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:15.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.591+0000 7fd6677fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:15.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.765+0000 7fd674dc4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T04:09:15.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.765+0000 7fd674dc4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T04:09:15.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.765+0000 7fd674dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:15.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.765+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) -- 0x7fd654005370 con 0x7fd67009ec20 2024-09-16T04:09:15.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.987+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v83) ==== 140+0+0 (unknown 2911421706 0 0) 0x7fd66001e260 con 0x7fd67009ec20 2024-09-16T04:09:15.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.987+0000 7fd6677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T04:09:15.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:15.987+0000 7fd6677fe640 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd674dc4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd674dc4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd674dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) -- 0x7fd654002210 con 0x7fd67009ec20 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd6677fe640 1 -- 172.21.15.171:0/2220430505 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v83) ==== 140+0+0 (unknown 1122948940 0 0) 0x7fd660013a80 con 0x7fd67009ec20 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd6677fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.094+0000 7fd6677fe640 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-16T04:09:16.096 INFO:tasks.workunit.client.0.smithi171.stderr:device 'host-for-test' does not appear in the crush map 2024-09-16T04:09:16.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 >> v1:172.21.15.171:6812/7327273 conn(0x7fd650038f10 legacy=0x7fd65003b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:16.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 10 monclient: shutdown 2024-09-16T04:09:16.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:16.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 >> v1:172.21.15.171:6789/0 conn(0x7fd67009ec20 legacy=0x7fd6701cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:16.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 shutdown_connections 2024-09-16T04:09:16.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 >> 172.21.15.171:0/2220430505 conn(0x7fd6700a52a0 msgr2=0x7fd6700a4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:16.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 shutdown_connections 2024-09-16T04:09:16.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.096+0000 7fd674dc4640 1 -- 172.21.15.171:0/2220430505 wait complete. 2024-09-16T04:09:16.107 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm rack-for-test 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.185+0000 7f8ceef86640 10 monclient: get_monmap_and_config 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.185+0000 7f8ceef86640 10 monclient: build_initial_monmap 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.185+0000 7f8ceef86640 10 monclient: monmap: 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:16.186412+0000 2024-09-16T04:09:16.187 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:16.186412+0000 2024-09-16T04:09:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:16.188 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:16.188 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 1 Processor -- start 2024-09-16T04:09:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 1 -- start start 2024-09-16T04:09:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient: init 2024-09-16T04:09:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:16.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient(hunting): picked mon.noname-a con 0x7f8ce8132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient(hunting): picked mon.noname-b con 0x7f8ce809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient(hunting): picked mon.noname-c con 0x7f8ce8132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce81332b0 con 0x7f8ce8132a80 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce8134310 con 0x7f8ce8132e60 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce809d980 con 0x7f8ce809ec20 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ceef86640 10 monclient(hunting): authenticate will time out at 931.691860s 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8cecd22640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8ce8132a80 legacy=0x7f8ce81334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.186+0000 7f8ced523640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8ce8132e60 0x7f8ce813c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36264/0 (socket says 172.21.15.171:36264) 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ced523640 1 -- 172.21.15.171:0/3155857587 learned_addr learned my addr 172.21.15.171:0/3155857587 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165412638 0 0) 0x7f8ce8134310 con 0x7f8ce8132e60 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:16.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient(hunting): my global_id is 5236 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cd8003520 con 0x7f8ce8132e60 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2146869040 0 0) 0x7f8cd8003520 con 0x7f8ce8132e60 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 >> v1:172.21.15.148:6790/0 conn(0x7f8ce809ec20 legacy=0x7f8ce8133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 >> v1:172.21.15.148:6789/0 conn(0x7f8ce8132a80 legacy=0x7f8ce81334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce809dc60 con 0x7f8ce8132e60 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:16.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ceef86640 5 monclient: authenticate success, global_id 5236 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ce0001990 con 0x7f8ce8132e60 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: dump: 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:16.198 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ceef86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ce0001c80 con 0x7f8ce8132e60 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ceef86640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 1 -- 172.21.15.171:0/3155857587 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ce0002090 con 0x7f8ce8132e60 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ceef86640 4 set_mon_vals no callback set 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.187+0000 7f8ce77fe640 10 monclient: dump: 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.199 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 10 monclient: shutdown 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 1 -- 172.21.15.171:0/3155857587 >> v1:172.21.15.171:6789/0 conn(0x7f8ce8132e60 legacy=0x7f8ce813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 1 -- 172.21.15.171:0/3155857587 shutdown_connections 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 1 -- 172.21.15.171:0/3155857587 >> 172.21.15.171:0/3155857587 conn(0x7f8ce80a52a0 msgr2=0x7f8ce80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 1 -- 172.21.15.171:0/3155857587 shutdown_connections 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 1 -- 172.21.15.171:0/3155857587 wait complete. 2024-09-16T04:09:16.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 10 monclient: build_initial_monmap 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 10 monclient: monmap: 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.188+0000 7f8ceef86640 1 Processor -- start 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 1 -- start start 2024-09-16T04:09:16.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient: init 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient(hunting): picked mon.noname-a con 0x7f8ce809ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient(hunting): picked mon.noname-c con 0x7f8ce8132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient(hunting): picked mon.noname-b con 0x7f8ce8132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce813a320 con 0x7f8ce809ec20 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce8131300 con 0x7f8ce8132e60 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce806ceb0 con 0x7f8ce8132a80 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ceef86640 10 monclient(hunting): authenticate will time out at 931.694630s 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8cecd22640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8ce809ec20 legacy=0x7f8ce81304e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ced523640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8ce8132e60 0x7f8ce81d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36278/0 (socket says 172.21.15.171:36278) 2024-09-16T04:09:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ced523640 1 -- 172.21.15.171:0/3778521624 learned_addr learned my addr 172.21.15.171:0/3778521624 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825754263 0 0) 0x7f8ce8131300 con 0x7f8ce8132e60 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ce57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ce57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ce57fa640 10 monclient(hunting): my global_id is 5239 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.189+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ccc003520 con 0x7f8ce8132e60 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4078229922 0 0) 0x7f8ce806ceb0 con 0x7f8ce8132a80 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): my global_id is 5270 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ce8131300 con 0x7f8ce8132a80 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 57379303 0 0) 0x7f8ccc003520 con 0x7f8ce8132e60 2024-09-16T04:09:16.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ce806ceb0 con 0x7f8ce8132e60 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ce00017b0 con 0x7f8ce8132e60 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): dump: 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:16.204 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2811045435 0 0) 0x7f8ce8131300 con 0x7f8ce8132a80 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ccc003520 con 0x7f8ce8132a80 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8cd4001830 con 0x7f8ce8132a80 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient(hunting): dump: 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:16.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2434196366 0 0) 0x7f8ce806ceb0 con 0x7f8ce8132e60 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 >> v1:172.21.15.148:6790/0 conn(0x7f8ce8132a80 legacy=0x7f8ce8130bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 >> v1:172.21.15.148:6789/0 conn(0x7f8ce809ec20 legacy=0x7f8ce81304e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 monclient: found mon.b 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce809c610 con 0x7f8ce8132e60 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:16.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 5 monclient: authenticate success, global_id 5239 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 10 monclient: _renew_subs 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce8140b70 con 0x7f8ce8132e60 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 10 monclient: _renew_subs 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ce813a8e0 con 0x7f8ce8132e60 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ce0001c80 con 0x7f8ce8132e60 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ce00051e0 con 0x7f8ce8132e60 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:16.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce6ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.190+0000 7f8ce57fa640 10 monclient: dump: 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8ce0010610 con 0x7f8ce8132e60 2024-09-16T04:09:16.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8ceef86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:16.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8ceef86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:16.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8ceef86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:16.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc8005080 con 0x7f8ce8132e60 2024-09-16T04:09:16.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(83..83 src has 1..83) ==== 5547+0+0 (unknown 89963815 0 0) 0x7f8ce0047790 con 0x7f8ce8132e60 2024-09-16T04:09:16.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.191+0000 7f8cecd22640 10 monclient: get_auth_request con 0x7f8ccc038f90 auth_method 0 2024-09-16T04:09:16.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.195+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8ce0015fb0 con 0x7f8ce8132e60 2024-09-16T04:09:16.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.195+0000 7f8ce57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:16.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.195+0000 7f8ce57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:16.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.370+0000 7f8ceef86640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T04:09:17.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.370+0000 7f8ceef86640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T04:09:17.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.370+0000 7f8ceef86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:16.370+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) -- 0x7f8cc8005370 con 0x7f8ce8132e60 2024-09-16T04:09:17.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.306+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v84) ==== 140+0+0 (unknown 1994927815 0 0) 0x7f8ce00158b0 con 0x7f8ce8132e60 2024-09-16T04:09:17.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.306+0000 7f8ce57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T04:09:17.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.306+0000 7f8ce57fa640 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.415+0000 7f8ceef86640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.415+0000 7f8ceef86640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.415+0000 7f8ceef86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.415+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) -- 0x7f8cc8002210 con 0x7f8ce8132e60 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.416+0000 7f8ce57fa640 1 -- 172.21.15.171:0/3778521624 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v84) ==== 140+0+0 (unknown 1719904899 0 0) 0x7f8ce0013c10 con 0x7f8ce8132e60 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.416+0000 7f8ce57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.416+0000 7f8ce57fa640 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-16T04:09:17.418 INFO:tasks.workunit.client.0.smithi171.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 >> v1:172.21.15.171:6812/7327273 conn(0x7f8ccc038f90 legacy=0x7f8ccc03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 10 monclient: shutdown 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 >> v1:172.21.15.171:6789/0 conn(0x7f8ce8132e60 legacy=0x7f8ce81d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 shutdown_connections 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 >> 172.21.15.171:0/3778521624 conn(0x7f8ce80a52a0 msgr2=0x7f8ce80a3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 shutdown_connections 2024-09-16T04:09:17.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.418+0000 7f8ceef86640 1 -- 172.21.15.171:0/3778521624 wait complete. 2024-09-16T04:09:17.430 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm root-for-test 2024-09-16T04:09:17.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.506+0000 7f56ff0ac640 10 monclient: get_monmap_and_config 2024-09-16T04:09:17.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.506+0000 7f56ff0ac640 10 monclient: build_initial_monmap 2024-09-16T04:09:17.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.506+0000 7f56ff0ac640 10 monclient: monmap: 2024-09-16T04:09:17.509 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:17.509 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:17.509 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:17.507490+0000 2024-09-16T04:09:17.509 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:17.507490+0000 2024-09-16T04:09:17.509 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:17.510 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.510 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:17.510 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:17.510 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:17.510 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 1 Processor -- start 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 1 -- start start 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient: init 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient(hunting): picked mon.noname-c con 0x7f56f8132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient(hunting): picked mon.noname-a con 0x7f56f809ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient(hunting): picked mon.noname-b con 0x7f56f8132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f81332b0 con 0x7f56f809ec20 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f8134310 con 0x7f56f8132a80 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f809d980 con 0x7f56f8132e60 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.507+0000 7f56ff0ac640 10 monclient(hunting): authenticate will time out at 933.012929s 2024-09-16T04:09:17.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56fce48640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f56f8132a80 0x7f56f81334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46120/0 (socket says 172.21.15.171:46120) 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56fce48640 1 -- 172.21.15.171:0/2751737771 learned_addr learned my addr 172.21.15.171:0/2751737771 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f7fff640 1 -- 172.21.15.171:0/2751737771 >> v1:172.21.15.148:6789/0 conn(0x7f56f809ec20 legacy=0x7f56f8133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4208526467 0 0) 0x7f56f8134310 con 0x7f56f8132a80 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): my global_id is 5242 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e8003520 con 0x7f56f8132a80 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3761656081 0 0) 0x7f56f809d980 con 0x7f56f8132e60 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:17.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient(hunting): my global_id is 5273 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f8134310 con 0x7f56f8132e60 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1082579183 0 0) 0x7f56e8003520 con 0x7f56f8132a80 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 >> v1:172.21.15.148:6790/0 conn(0x7f56f8132e60 legacy=0x7f56f813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 >> v1:172.21.15.148:6789/0 conn(0x7f56f809ec20 legacy=0x7f56f8133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f809dc60 con 0x7f56f8132a80 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56ff0ac640 5 monclient: authenticate success, global_id 5242 2024-09-16T04:09:17.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f56ec001610 con 0x7f56f8132a80 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56ff0ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: dump: 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:17.524 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56ff0ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56ec001870 con 0x7f56f8132a80 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 1 -- 172.21.15.171:0/2751737771 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f56ec001bc0 con 0x7f56f8132a80 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56ff0ac640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56ff0ac640 4 set_mon_vals no callback set 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.508+0000 7f56f77fe640 10 monclient: dump: 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:17.525 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.526 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.526 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:17.526 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.526 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:17.526 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:17.527 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:17.527 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 10 monclient: shutdown 2024-09-16T04:09:17.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:17.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2751737771 >> v1:172.21.15.171:6789/0 conn(0x7f56f8132a80 legacy=0x7f56f81334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:17.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2751737771 shutdown_connections 2024-09-16T04:09:17.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2751737771 >> 172.21.15.171:0/2751737771 conn(0x7f56f80a52a0 msgr2=0x7f56f80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:17.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2751737771 shutdown_connections 2024-09-16T04:09:17.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2751737771 wait complete. 2024-09-16T04:09:17.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 10 monclient: build_initial_monmap 2024-09-16T04:09:17.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.509+0000 7f56ff0ac640 10 monclient: monmap: 2024-09-16T04:09:17.528 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 1 Processor -- start 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 1 -- start start 2024-09-16T04:09:17.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient: init 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient(hunting): picked mon.noname-c con 0x7f56f809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient(hunting): picked mon.noname-b con 0x7f56f8132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient(hunting): picked mon.noname-a con 0x7f56f8132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f81221c0 con 0x7f56f8132e60 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f8120460 con 0x7f56f8132a80 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f80a3450 con 0x7f56f809ec20 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:17.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56ff0ac640 10 monclient(hunting): authenticate will time out at 933.015839s 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56fd649640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f56f8132e60 legacy=0x7f56f81d1860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56f7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f56f8132a80 0x7f56f811fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46134/0 (socket says 172.21.15.171:46134) 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.510+0000 7f56f7fff640 1 -- 172.21.15.171:0/2116202752 learned_addr learned my addr 172.21.15.171:0/2116202752 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2060740046 0 0) 0x7f56f8120460 con 0x7f56f8132a80 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): my global_id is 5245 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56dc003520 con 0x7f56f8132a80 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2290449634 0 0) 0x7f56f80a3450 con 0x7f56f809ec20 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:17.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): my global_id is 5276 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f8120460 con 0x7f56f809ec20 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3651362799 0 0) 0x7f56dc003520 con 0x7f56f8132a80 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56f80a3450 con 0x7f56f8132a80 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f56e4004180 con 0x7f56f8132a80 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): dump: 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:17.532 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1043196814 0 0) 0x7f56f8120460 con 0x7f56f809ec20 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56dc003520 con 0x7f56f809ec20 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f56ec001510 con 0x7f56f809ec20 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:17.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient(hunting): dump: 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3987957659 0 0) 0x7f56f80a3450 con 0x7f56f8132a80 2024-09-16T04:09:17.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 >> v1:172.21.15.148:6790/0 conn(0x7f56f809ec20 legacy=0x7f56f811f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 >> v1:172.21.15.148:6789/0 conn(0x7f56f8132e60 legacy=0x7f56f81d1860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 monclient: found mon.b 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f8131a10 con 0x7f56f8132a80 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 5 monclient: authenticate success, global_id 5245 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 10 monclient: _renew_subs 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56f8121ee0 con 0x7f56f8132a80 2024-09-16T04:09:17.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 10 monclient: _renew_subs 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f56f8130400 con 0x7f56f8132a80 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56e4004430 con 0x7f56f8132a80 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f56e4004800 con 0x7f56f8132a80 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f57fa640 10 monclient: dump: 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:17.536 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.511+0000 7f56f6ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f56e4004a20 con 0x7f56f8132a80 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56ff0ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56ff0ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56ff0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f8071760 con 0x7f56f8132a80 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(84..84 src has 1..84) ==== 5364+0+0 (unknown 3461029995 0 0) 0x7f56e4047470 con 0x7f56f8132a80 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.512+0000 7f56fce48640 10 monclient: get_auth_request con 0x7f56dc038ef0 auth_method 0 2024-09-16T04:09:17.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.517+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f56e4015c20 con 0x7f56f8132a80 2024-09-16T04:09:17.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.517+0000 7f56f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:17.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.517+0000 7f56f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:17.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.687+0000 7f56ff0ac640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T04:09:17.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.687+0000 7f56ff0ac640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T04:09:17.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.687+0000 7f56ff0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:17.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:17.687+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) -- 0x7f56f806bcf0 con 0x7f56f8132a80 2024-09-16T04:09:18.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.315+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v85) ==== 140+0+0 (unknown 1057864952 0 0) 0x7f56e40198d0 con 0x7f56f8132a80 2024-09-16T04:09:18.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.315+0000 7f56f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T04:09:18.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.315+0000 7f56f57fa640 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-16T04:09:18.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.419+0000 7f56ff0ac640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T04:09:18.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.419+0000 7f56ff0ac640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T04:09:18.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.419+0000 7f56ff0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.419+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) -- 0x7f56f809f8e0 con 0x7f56f8132a80 2024-09-16T04:09:18.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.420+0000 7f56f57fa640 1 -- 172.21.15.171:0/2116202752 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v85) ==== 140+0+0 (unknown 429848410 0 0) 0x7f56e401e190 con 0x7f56f8132a80 2024-09-16T04:09:18.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.420+0000 7f56f57fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-16T04:09:18.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.420+0000 7f56f57fa640 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-16T04:09:18.423 INFO:tasks.workunit.client.0.smithi171.stderr:device 'root-for-test' does not appear in the crush map 2024-09-16T04:09:18.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 >> v1:172.21.15.171:6812/7327273 conn(0x7f56dc038ef0 legacy=0x7f56dc03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 10 monclient: shutdown 2024-09-16T04:09:18.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:18.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 >> v1:172.21.15.171:6789/0 conn(0x7f56f8132a80 legacy=0x7f56f811fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 shutdown_connections 2024-09-16T04:09:18.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 >> 172.21.15.171:0/2116202752 conn(0x7f56f80a52a0 msgr2=0x7f56f8131300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:18.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 shutdown_connections 2024-09-16T04:09:18.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.422+0000 7f56ff0ac640 1 -- 172.21.15.171:0/2116202752 wait complete. 2024-09-16T04:09:18.433 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-16T04:09:18.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.513+0000 7fd1a7de4640 10 monclient: get_monmap_and_config 2024-09-16T04:09:18.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.513+0000 7fd1a7de4640 10 monclient: build_initial_monmap 2024-09-16T04:09:18.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.513+0000 7fd1a7de4640 10 monclient: monmap: 2024-09-16T04:09:18.523 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:18.523 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:18.514287+0000 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:18.514287+0000 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 1 Processor -- start 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 1 -- start start 2024-09-16T04:09:18.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient: init 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a01332b0 con 0x7fd1a0132a80 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a0134310 con 0x7fd1a009ec20 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a009d980 con 0x7fd1a0132e60 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a7de4640 10 monclient(hunting): authenticate will time out at 934.019749s 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a5b80640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd1a0132a80 legacy=0x7fd1a01334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a537f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd1a009ec20 0x7fd1a0133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46142/0 (socket says 172.21.15.171:46142) 2024-09-16T04:09:18.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.514+0000 7fd1a537f640 1 -- 172.21.15.171:0/2136729897 learned_addr learned my addr 172.21.15.171:0/2136729897 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2620465303 0 0) 0x7fd1a0134310 con 0x7fd1a009ec20 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): _negotiate 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): my global_id is 5248 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18c003520 con 0x7fd1a009ec20 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187147544 0 0) 0x7fd1a009d980 con 0x7fd1a0132e60 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): _negotiate 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient(hunting): my global_id is 5279 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a0134310 con 0x7fd1a0132e60 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2569473619 0 0) 0x7fd18c003520 con 0x7fd1a009ec20 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 >> v1:172.21.15.148:6790/0 conn(0x7fd1a0132e60 legacy=0x7fd1a013c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 >> v1:172.21.15.148:6789/0 conn(0x7fd1a0132a80 legacy=0x7fd1a01334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: _finish_hunting 0 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 monclient: found mon.noname-c 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a009dc60 con 0x7fd1a009ec20 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: _finish_auth 0 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: _check_auth_tickets 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:18.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd190001610 con 0x7fd1a009ec20 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 5 monclient: authenticate success, global_id 5248 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: dump: 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:18.528 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1900018c0 con 0x7fd1a009ec20 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 1 -- 172.21.15.171:0/2136729897 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd190001c90 con 0x7fd1a009ec20 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:18.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 4 set_mon_vals no callback set 2024-09-16T04:09:18.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:18.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a4b7e640 10 monclient: dump: 2024-09-16T04:09:18.531 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:18.531 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:18.532 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.532 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.532 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.533 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.533 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:18.533 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:18.533 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:18.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 10 monclient: shutdown 2024-09-16T04:09:18.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:18.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 1 -- 172.21.15.171:0/2136729897 >> v1:172.21.15.171:6789/0 conn(0x7fd1a009ec20 legacy=0x7fd1a0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 1 -- 172.21.15.171:0/2136729897 shutdown_connections 2024-09-16T04:09:18.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.515+0000 7fd1a7de4640 1 -- 172.21.15.171:0/2136729897 >> 172.21.15.171:0/2136729897 conn(0x7fd1a00a52a0 msgr2=0x7fd1a00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:18.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.516+0000 7fd1a7de4640 1 -- 172.21.15.171:0/2136729897 shutdown_connections 2024-09-16T04:09:18.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.516+0000 7fd1a7de4640 1 -- 172.21.15.171:0/2136729897 wait complete. 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.516+0000 7fd1a7de4640 10 monclient: build_initial_monmap 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.516+0000 7fd1a7de4640 10 monclient: monmap: 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:18.535 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:18.536 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.536 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:18.536 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:18.536 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:18.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 1 Processor -- start 2024-09-16T04:09:18.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 1 -- start start 2024-09-16T04:09:18.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient: init 2024-09-16T04:09:18.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a009ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a0132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a01c8e50 con 0x7fd1a0132a80 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a01c83d0 con 0x7fd1a0132e60 2024-09-16T04:09:18.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a0131930 con 0x7fd1a009ec20 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a7de4640 10 monclient(hunting): authenticate will time out at 934.022556s 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a537f640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd1a0132a80 legacy=0x7fd1a01cd270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a6381640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd1a0132e60 0x7fd1a01d09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46150/0 (socket says 172.21.15.171:46150) 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1a6381640 1 -- 172.21.15.171:0/3481865845 learned_addr learned my addr 172.21.15.171:0/3481865845 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035215133 0 0) 0x7fd1a0134310 con 0x7fd1a0132e60 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): my global_id is 5251 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd184003520 con 0x7fd1a0132e60 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577184125 0 0) 0x7fd1a0131930 con 0x7fd1a009ec20 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:18.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): my global_id is 5282 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a0134310 con 0x7fd1a009ec20 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1602023500 0 0) 0x7fd184003520 con 0x7fd1a0132e60 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1a0131930 con 0x7fd1a0132e60 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd198001c50 con 0x7fd1a0132e60 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.517+0000 7fd1967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient(hunting): dump: 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.540 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2006155929 0 0) 0x7fd1a0134310 con 0x7fd1a009ec20 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd184003520 con 0x7fd1a009ec20 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd19c0040f0 con 0x7fd1a009ec20 2024-09-16T04:09:18.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient(hunting): dump: 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:18.542 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3109371908 0 0) 0x7fd1a0131930 con 0x7fd1a0132e60 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 >> v1:172.21.15.148:6790/0 conn(0x7fd1a009ec20 legacy=0x7fd1a0130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 >> v1:172.21.15.148:6789/0 conn(0x7fd1a0132a80 legacy=0x7fd1a01cd270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 monclient: found mon.b 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a00a3310 con 0x7fd1a0132e60 2024-09-16T04:09:18.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 5 monclient: authenticate success, global_id 5251 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 10 monclient: _renew_subs 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a006ceb0 con 0x7fd1a0132e60 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 10 monclient: _renew_subs 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1a011e820 con 0x7fd1a0132e60 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd198002020 con 0x7fd1a0132e60 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:18.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd198003de0 con 0x7fd1a0132e60 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd197fff640 4 set_mon_vals no callback set 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.518+0000 7fd1967fc640 10 monclient: dump: 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:18.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd198004080 con 0x7fd1a0132e60 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1a7de4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1a7de4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1a7de4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd188005080 con 0x7fd1a0132e60 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(85..85 src has 1..85) ==== 5205+0+0 (unknown 1316711799 0 0) 0x7fd198047790 con 0x7fd1a0132e60 2024-09-16T04:09:18.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.519+0000 7fd1a5b80640 10 monclient: get_auth_request con 0x7fd184038ea0 auth_method 0 2024-09-16T04:09:18.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.523+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd1a01c83d0 con 0x7fd1a0132e60 2024-09-16T04:09:18.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.523+0000 7fd1967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:18.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.523+0000 7fd1967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:18.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.690+0000 7fd1a7de4640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T04:09:18.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.690+0000 7fd1a7de4640 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T04:09:18.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.690+0000 7fd1a7de4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:18.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:18.690+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) -- 0x7fd188005370 con 0x7fd1a0132e60 2024-09-16T04:09:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.319+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v86) ==== 145+0+0 (unknown 2196621694 0 0) 0x7fd1980160c0 con 0x7fd1a0132e60 2024-09-16T04:09:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.319+0000 7fd1967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T04:09:19.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.319+0000 7fd1967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:19.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.424+0000 7fd1a7de4640 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.424+0000 7fd1a7de4640 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.424+0000 7fd1a7de4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.424+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) -- 0x7fd188001150 con 0x7fd1a0132e60 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.425+0000 7fd1967fc640 1 -- 172.21.15.171:0/3481865845 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v86) ==== 173+0+0 (unknown 4041671292 0 0) 0x7fd1980159c0 con 0x7fd1a0132e60 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.425+0000 7fd1967fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.425+0000 7fd1967fc640 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-16T04:09:19.427 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo-rule already exists 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 >> v1:172.21.15.171:6812/7327273 conn(0x7fd184038ea0 legacy=0x7fd18403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 10 monclient: shutdown 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 >> v1:172.21.15.171:6789/0 conn(0x7fd1a0132e60 legacy=0x7fd1a01d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 shutdown_connections 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 >> 172.21.15.171:0/3481865845 conn(0x7fd1a00a52a0 msgr2=0x7fd1a0133b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 shutdown_connections 2024-09-16T04:09:19.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.427+0000 7fd1a7de4640 1 -- 172.21.15.171:0/3481865845 wait complete. 2024-09-16T04:09:19.438 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush rm foo 2024-09-16T04:09:19.438 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:19.438 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm foo 2024-09-16T04:09:19.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.513+0000 7f171fd66640 10 monclient: get_monmap_and_config 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.513+0000 7f171fd66640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.513+0000 7f171fd66640 10 monclient: monmap: 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:19.514732+0000 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:19.514732+0000 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:19.516 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.514+0000 7f171fd66640 1 Processor -- start 2024-09-16T04:09:19.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.514+0000 7f171fd66640 1 -- start start 2024-09-16T04:09:19.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.514+0000 7f171fd66640 10 monclient: init 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient(hunting): picked mon.noname-a con 0x7f171809f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient(hunting): picked mon.noname-b con 0x7f17180a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient(hunting): picked mon.noname-c con 0x7f171809e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17180a1260 con 0x7f171809f640 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171809d0c0 con 0x7f171809e1d0 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171809d3a0 con 0x7f17180a0130 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient(hunting): authenticate will time out at 935.020283s 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171e303640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f171809e1d0 0x7f171813bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46162/0 (socket says 172.21.15.171:46162) 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171e303640 1 -- 172.21.15.171:0/2614389188 learned_addr learned my addr 172.21.15.171:0/2614389188 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171db02640 1 -- 172.21.15.171:0/2614389188 >> v1:172.21.15.148:6789/0 conn(0x7f171809f640 legacy=0x7f171809fa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:19.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360566378 0 0) 0x7f171809d0c0 con 0x7f171809e1d0 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient(hunting): _negotiate 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient(hunting): my global_id is 5254 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1704004630 con 0x7f171809e1d0 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3681475977 0 0) 0x7f1704004630 con 0x7f171809e1d0 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 >> v1:172.21.15.148:6790/0 conn(0x7f17180a0130 legacy=0x7f1718138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 >> v1:172.21.15.148:6789/0 conn(0x7f171809f640 legacy=0x7f171809fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 monclient: found mon.noname-c 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171809d680 con 0x7f171809e1d0 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 5 monclient: authenticate success, global_id 5254 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1710001610 con 0x7f171809e1d0 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: dump: 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.526 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1710001900 con 0x7f171809e1d0 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 1 -- 172.21.15.171:0/2614389188 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1710001cd0 con 0x7f171809e1d0 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171fd66640 4 set_mon_vals no callback set 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.515+0000 7f171cb00640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171cb00640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:19.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171cb00640 10 monclient: dump: 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 10 monclient: shutdown 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 1 -- 172.21.15.171:0/2614389188 >> v1:172.21.15.171:6789/0 conn(0x7f171809e1d0 legacy=0x7f171813bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 1 -- 172.21.15.171:0/2614389188 shutdown_connections 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 1 -- 172.21.15.171:0/2614389188 >> 172.21.15.171:0/2614389188 conn(0x7f17180a52a0 msgr2=0x7f17180a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 1 -- 172.21.15.171:0/2614389188 shutdown_connections 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.516+0000 7f171fd66640 1 -- 172.21.15.171:0/2614389188 wait complete. 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient: monmap: 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:19.529 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 1 Processor -- start 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 1 -- start start 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient: init 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient(hunting): picked mon.noname-c con 0x7f171809e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient(hunting): picked mon.noname-a con 0x7f171809f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient(hunting): picked mon.noname-b con 0x7f17180a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17181a9970 con 0x7f171809f640 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17181a7b40 con 0x7f17180a0130 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17180a3e40 con 0x7f171809e1d0 2024-09-16T04:09:19.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171fd66640 10 monclient(hunting): authenticate will time out at 935.022957s 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.517+0000 7f171d301640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f171809f640 legacy=0x7f17181d0930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171e303640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f17180a0130 0x7f17181d4060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46166/0 (socket says 172.21.15.171:46166) 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171e303640 1 -- 172.21.15.171:0/1020345357 learned_addr learned my addr 172.21.15.171:0/1020345357 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4205798048 0 0) 0x7f17181a7b40 con 0x7f17180a0130 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): my global_id is 5257 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16fc003520 con 0x7f17180a0130 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224882435 0 0) 0x7f17180a3e40 con 0x7f171809e1d0 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:19.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): my global_id is 5288 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17181a7b40 con 0x7f171809e1d0 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435747623 0 0) 0x7f16fc003520 con 0x7f17180a0130 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17180a3e40 con 0x7f17180a0130 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1710001430 con 0x7f17180a0130 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.532 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 477716918 0 0) 0x7f17180a3e40 con 0x7f17180a0130 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 >> v1:172.21.15.148:6790/0 conn(0x7f171809e1d0 legacy=0x7f17181cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 >> v1:172.21.15.148:6789/0 conn(0x7f171809f640 legacy=0x7f17181d0930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 monclient: found mon.b 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171806ceb0 con 0x7f17180a0130 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 5 monclient: authenticate success, global_id 5257 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 10 monclient: _renew_subs 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17181a7260 con 0x7f17180a0130 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 10 monclient: _renew_subs 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f171fd66640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f17181a9550 con 0x7f17180a0130 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1710004cc0 con 0x7f17180a0130 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1710005010 con 0x7f17180a0130 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170ffff640 4 set_mon_vals no callback set 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.518+0000 7f170e7fc640 10 monclient: dump: 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.535 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1710010440 con 0x7f17180a0130 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f171fd66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f171fd66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f171fd66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f171fd66640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1700005080 con 0x7f17180a0130 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(86..86 src has 1..86) ==== 5265+0+0 (unknown 1424395690 0 0) 0x7f17100474e0 con 0x7f17180a0130 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.519+0000 7f171db02640 10 monclient: get_auth_request con 0x7f16fc038bc0 auth_method 0 2024-09-16T04:09:19.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.524+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1710015e10 con 0x7f17180a0130 2024-09-16T04:09:19.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.524+0000 7f170e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.524+0000 7f170e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.691+0000 7f171fd66640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.691+0000 7f171fd66640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.691+0000 7f171fd66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.691+0000 7f171fd66640 1 -- 172.21.15.171:0/1020345357 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f1700005370 con 0x7f17180a0130 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.692+0000 7f170e7fc640 1 -- 172.21.15.171:0/1020345357 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v86) ==== 103+0+0 (unknown 2579682189 0 0) 0x7f1710015710 con 0x7f17180a0130 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.692+0000 7f170e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.692+0000 7f170e7fc640 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-16T04:09:19.694 INFO:tasks.workunit.client.0.smithi171.stderr:Error EBUSY: (16) Device or resource busy 2024-09-16T04:09:19.697 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:19.697 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rule rm foo-rule 2024-09-16T04:09:19.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.776+0000 7f8492f6d640 10 monclient: get_monmap_and_config 2024-09-16T04:09:19.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.776+0000 7f8492f6d640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.776+0000 7f8492f6d640 10 monclient: monmap: 2024-09-16T04:09:19.778 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:19.776995+0000 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:19.776995+0000 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:19.779 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:19.780 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 1 Processor -- start 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 1 -- start start 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient: init 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f848c136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f848c139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:19.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f848c13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a1340 con 0x7f848c136b60 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c137650 con 0x7f848c13d620 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a0aa0 con 0x7f848c139b10 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8492f6d640 10 monclient(hunting): authenticate will time out at 935.282683s 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f8490d09640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f848c136b60 legacy=0x7f848c136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f849150a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f848c13d620 0x7f848c13fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46178/0 (socket says 172.21.15.171:46178) 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f849150a640 1 -- 172.21.15.171:0/1859641314 learned_addr learned my addr 172.21.15.171:0/1859641314 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.777+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942164606 0 0) 0x7f848c137650 con 0x7f848c13d620 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient(hunting): my global_id is 5260 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8474003520 con 0x7f848c13d620 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1053711244 0 0) 0x7f8474003520 con 0x7f848c13d620 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 >> v1:172.21.15.148:6790/0 conn(0x7f848c139b10 legacy=0x7f848c13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 >> v1:172.21.15.148:6789/0 conn(0x7f848c136b60 legacy=0x7f848c136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c0a0550 con 0x7f848c13d620 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8484001990 con 0x7f848c13d620 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 5 monclient: authenticate success, global_id 5260 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: dump: 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.793 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8484001cf0 con 0x7f848c13d620 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 1 -- 172.21.15.171:0/1859641314 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8484004c20 con 0x7f848c13d620 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 4 set_mon_vals no callback set 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f848b7fe640 10 monclient: dump: 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 10 monclient: shutdown 2024-09-16T04:09:19.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 1 -- 172.21.15.171:0/1859641314 >> v1:172.21.15.171:6789/0 conn(0x7f848c13d620 legacy=0x7f848c13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 1 -- 172.21.15.171:0/1859641314 shutdown_connections 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 1 -- 172.21.15.171:0/1859641314 >> 172.21.15.171:0/1859641314 conn(0x7f848c0a98b0 msgr2=0x7f848c0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 1 -- 172.21.15.171:0/1859641314 shutdown_connections 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.778+0000 7f8492f6d640 1 -- 172.21.15.171:0/1859641314 wait complete. 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.779+0000 7f8492f6d640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.779+0000 7f8492f6d640 10 monclient: monmap: 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:19.796 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.779+0000 7f8492f6d640 1 Processor -- start 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.779+0000 7f8492f6d640 1 -- start start 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.779+0000 7f8492f6d640 10 monclient: init 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f848c136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f848c139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:19.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f848c13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c120d10 con 0x7f848c13d620 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c1cc5c0 con 0x7f848c139b10 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f848c0a0240 con 0x7f848c136b60 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f8492f6d640 10 monclient(hunting): authenticate will time out at 935.285469s 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f848bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f848c139b10 0x7f848c1cbeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46184/0 (socket says 172.21.15.171:46184) 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f848bfff640 1 -- 172.21.15.171:0/2382942254 learned_addr learned my addr 172.21.15.171:0/2382942254 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f849150a640 1 -- 172.21.15.171:0/2382942254 >> v1:172.21.15.148:6789/0 conn(0x7f848c13d620 legacy=0x7f848c1d3a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2665433304 0 0) 0x7f848c1cc5c0 con 0x7f848c139b10 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): my global_id is 5263 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8478003550 con 0x7f848c139b10 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2442796782 0 0) 0x7f8478003550 con 0x7f848c139b10 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f848c1cc5c0 con 0x7f848c139b10 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8480001970 con 0x7f848c139b10 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): dump: 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.799 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.780+0000 7f84897fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155730907 0 0) 0x7f848c0a0240 con 0x7f848c136b60 2024-09-16T04:09:19.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient(hunting): my global_id is 5294 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8478003550 con 0x7f848c136b60 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3477515503 0 0) 0x7f848c1cc5c0 con 0x7f848c139b10 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 >> v1:172.21.15.148:6790/0 conn(0x7f848c136b60 legacy=0x7f848c1cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 >> v1:172.21.15.148:6789/0 conn(0x7f848c13d620 legacy=0x7f848c1d3a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 monclient: found mon.b 2024-09-16T04:09:19.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c0a2ac0 con 0x7f848c139b10 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 5 monclient: authenticate success, global_id 5263 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 10 monclient: _renew_subs 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f848c11fda0 con 0x7f848c139b10 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 10 monclient: _renew_subs 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f848c09e270 con 0x7f848c139b10 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8480003940 con 0x7f848c139b10 2024-09-16T04:09:19.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8480003d10 con 0x7f848c139b10 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f848affd640 4 set_mon_vals no callback set 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.781+0000 7f84897fa640 10 monclient: dump: 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:19.803 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8480003f30 con 0x7f848c139b10 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f8492f6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f8492f6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f8492f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846c005080 con 0x7f848c139b10 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(86..86 src has 1..86) ==== 5265+0+0 (unknown 1424395690 0 0) 0x7f8480047520 con 0x7f848c139b10 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.782+0000 7f8490d09640 10 monclient: get_auth_request con 0x7f8478038fa0 auth_method 0 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.786+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8480015ed0 con 0x7f848c139b10 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.786+0000 7f84897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.786+0000 7f84897fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:19.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.954+0000 7f8492f6d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T04:09:19.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.954+0000 7f8492f6d640 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T04:09:19.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.954+0000 7f8492f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:19.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:19.954+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) -- 0x7f846c005370 con 0x7f848c139b10 2024-09-16T04:09:20.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.323+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v87) ==== 85+0+0 (unknown 2494653344 0 0) 0x7f84800157d0 con 0x7f848c139b10 2024-09-16T04:09:20.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.323+0000 7f84897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T04:09:20.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.323+0000 7f84897fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.426+0000 7f8492f6d640 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.426+0000 7f8492f6d640 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.426+0000 7f8492f6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.426+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) -- 0x7f846c002210 con 0x7f848c139b10 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.427+0000 7f84897fa640 1 -- 172.21.15.171:0/2382942254 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v87) ==== 113+0+0 (unknown 3509064181 0 0) 0x7f848001e380 con 0x7f848c139b10 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.427+0000 7f84897fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.427+0000 7f84897fa640 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-16T04:09:20.429 INFO:tasks.workunit.client.0.smithi171.stderr:rule foo-rule does not exist 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 >> v1:172.21.15.171:6812/7327273 conn(0x7f8478038fa0 legacy=0x7f847803b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 10 monclient: shutdown 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 >> v1:172.21.15.171:6789/0 conn(0x7f848c139b10 legacy=0x7f848c1cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 shutdown_connections 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 >> 172.21.15.171:0/2382942254 conn(0x7f848c0a98b0 msgr2=0x7f848c139230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 shutdown_connections 2024-09-16T04:09:20.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.429+0000 7f8492f6d640 1 -- 172.21.15.171:0/2382942254 wait complete. 2024-09-16T04:09:20.440 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm bar 2024-09-16T04:09:20.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.517+0000 7f885adb9640 10 monclient: get_monmap_and_config 2024-09-16T04:09:20.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.517+0000 7f885adb9640 10 monclient: build_initial_monmap 2024-09-16T04:09:20.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.517+0000 7f885adb9640 10 monclient: monmap: 2024-09-16T04:09:20.519 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:20.519 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:20.518370+0000 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:20.518370+0000 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:20.520 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 1 Processor -- start 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 1 -- start start 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient: init 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient(hunting): picked mon.noname-b con 0x7f88541338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient(hunting): picked mon.noname-c con 0x7f885409aa20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient(hunting): picked mon.noname-a con 0x7f885413d240 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88541343d0 con 0x7f885413d240 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8854075db0 con 0x7f885409aa20 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885409b1e0 con 0x7f88541338e0 2024-09-16T04:09:20.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f885adb9640 10 monclient(hunting): authenticate will time out at 936.023854s 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f884bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f885409aa20 0x7f885413bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46192/0 (socket says 172.21.15.171:46192) 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f8859356640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f885413d240 legacy=0x7f885413f630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.518+0000 7f884bfff640 1 -- 172.21.15.171:0/2350183753 learned_addr learned my addr 172.21.15.171:0/2350183753 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185366127 0 0) 0x7f8854075db0 con 0x7f885409aa20 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient(hunting): my global_id is 5266 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f883c003520 con 0x7f885409aa20 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3977954131 0 0) 0x7f883c003520 con 0x7f885409aa20 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 >> v1:172.21.15.148:6790/0 conn(0x7f88541338e0 legacy=0x7f8854133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 >> v1:172.21.15.148:6789/0 conn(0x7f885413d240 legacy=0x7f885413f630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:20.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885409b4c0 con 0x7f885409aa20 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8844001610 con 0x7f885409aa20 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 5 monclient: authenticate success, global_id 5266 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:20.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: dump: 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8844001900 con 0x7f885409aa20 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:20.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 1 -- 172.21.15.171:0/2350183753 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8844001d10 con 0x7f885409aa20 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 4 set_mon_vals no callback set 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f884b7fe640 10 monclient: dump: 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:20.534 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 10 monclient: shutdown 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.519+0000 7f885adb9640 1 -- 172.21.15.171:0/2350183753 >> v1:172.21.15.171:6789/0 conn(0x7f885409aa20 legacy=0x7f885413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 1 -- 172.21.15.171:0/2350183753 shutdown_connections 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 1 -- 172.21.15.171:0/2350183753 >> 172.21.15.171:0/2350183753 conn(0x7f88540a28d0 msgr2=0x7f88540a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 1 -- 172.21.15.171:0/2350183753 shutdown_connections 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 1 -- 172.21.15.171:0/2350183753 wait complete. 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 10 monclient: build_initial_monmap 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 10 monclient: monmap: 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:20.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 1 Processor -- start 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 1 -- start start 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.520+0000 7f885adb9640 10 monclient: init 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient(hunting): picked mon.noname-c con 0x7f885409aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:20.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient(hunting): picked mon.noname-b con 0x7f88541338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient(hunting): picked mon.noname-a con 0x7f885413d240 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885411c710 con 0x7f885413d240 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885411dbc0 con 0x7f88541338e0 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885409d520 con 0x7f885409aa20 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f885adb9640 10 monclient(hunting): authenticate will time out at 936.026519s 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f8859356640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f885413d240 legacy=0x7f88541d3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f884bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f88541338e0 0x7f88541d0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46208/0 (socket says 172.21.15.171:46208) 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f884bfff640 1 -- 172.21.15.171:0/1221955896 learned_addr learned my addr 172.21.15.171:0/1221955896 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240309560 0 0) 0x7f885411dbc0 con 0x7f88541338e0 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:20.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): my global_id is 5269 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8840003520 con 0x7f88541338e0 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329336000 0 0) 0x7f885409d520 con 0x7f885409aa20 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 10 monclient(hunting): my global_id is 5300 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f885411dbc0 con 0x7f885409aa20 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.521+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1060145971 0 0) 0x7f8840003520 con 0x7f88541338e0 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f885409d520 con 0x7f88541338e0 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88440015d0 con 0x7f88541338e0 2024-09-16T04:09:20.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient(hunting): dump: 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:20.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3119735857 0 0) 0x7f885409d520 con 0x7f88541338e0 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 >> v1:172.21.15.148:6790/0 conn(0x7f885409aa20 legacy=0x7f8854132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 >> v1:172.21.15.148:6789/0 conn(0x7f885413d240 legacy=0x7f88541d3fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 monclient: found mon.b 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88540a0140 con 0x7f88541338e0 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 5 monclient: authenticate success, global_id 5269 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 10 monclient: _renew_subs 2024-09-16T04:09:20.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f885411d8e0 con 0x7f88541338e0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 10 monclient: _renew_subs 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f885413b810 con 0x7f88541338e0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8844001900 con 0x7f88541338e0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8844004cc0 con 0x7f88541338e0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f884affd640 4 set_mon_vals no callback set 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 10 monclient: dump: 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:20.541 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.522+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8844004e60 con 0x7f88541338e0 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.523+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(87..87 src has 1..87) ==== 5205+0+0 (unknown 3503061545 0 0) 0x7f88440474a0 con 0x7f88541338e0 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.523+0000 7f8858b55640 10 monclient: get_auth_request con 0x7f8840038fe0 auth_method 0 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.523+0000 7f885adb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.523+0000 7f885adb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:20.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.523+0000 7f885adb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.523+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8834005080 con 0x7f88541338e0 2024-09-16T04:09:20.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.527+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8844015e10 con 0x7f88541338e0 2024-09-16T04:09:20.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.527+0000 7f88497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:20.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.527+0000 7f88497fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:20.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.697+0000 7f885adb9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:20.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.697+0000 7f885adb9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:20.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.697+0000 7f885adb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:20.699 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:20.697+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7f8834005370 con 0x7f88541338e0 2024-09-16T04:09:21.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.334+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v88) ==== 120+0+0 (unknown 1814175515 0 0) 0x7f8844015710 con 0x7f88541338e0 2024-09-16T04:09:21.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.334+0000 7f88497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:21.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.334+0000 7f88497fa640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-16T04:09:21.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.439+0000 7f885adb9640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.439+0000 7f885adb9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.439+0000 7f885adb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.439+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) -- 0x7f8834002210 con 0x7f88541338e0 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.440+0000 7f88497fa640 1 -- 172.21.15.171:0/1221955896 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v88) ==== 120+0+0 (unknown 2237415084 0 0) 0x7f884401e260 con 0x7f88541338e0 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.440+0000 7f88497fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.440+0000 7f88497fa640 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-16T04:09:21.442 INFO:tasks.workunit.client.0.smithi171.stderr:device 'bar' does not appear in the crush map 2024-09-16T04:09:21.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.441+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 >> v1:172.21.15.171:6812/7327273 conn(0x7f8840038fe0 legacy=0x7f884003b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.441+0000 7f885adb9640 10 monclient: shutdown 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.441+0000 7f885adb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.441+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 >> v1:172.21.15.171:6789/0 conn(0x7f88541338e0 legacy=0x7f88541d0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.442+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 shutdown_connections 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.442+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 >> 172.21.15.171:0/1221955896 conn(0x7f88540a28d0 msgr2=0x7f88540a46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.442+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 shutdown_connections 2024-09-16T04:09:21.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.442+0000 7f885adb9640 1 -- 172.21.15.171:0/1221955896 wait complete. 2024-09-16T04:09:21.453 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm foo 2024-09-16T04:09:21.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.528+0000 7fec0bf23640 10 monclient: get_monmap_and_config 2024-09-16T04:09:21.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.528+0000 7fec0bf23640 10 monclient: build_initial_monmap 2024-09-16T04:09:21.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.528+0000 7fec0bf23640 10 monclient: monmap: 2024-09-16T04:09:21.530 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:21.529438+0000 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:21.529438+0000 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:21.531 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:21.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 1 Processor -- start 2024-09-16T04:09:21.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 1 -- start start 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 10 monclient: init 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 10 monclient(hunting): picked mon.noname-b con 0x7fec04072a70 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 10 monclient(hunting): picked mon.noname-a con 0x7fec04070250 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 10 monclient(hunting): picked mon.noname-c con 0x7fec0413afe0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0409ed80 con 0x7fec04070250 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.529+0000 7fec0bf23640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec040a0030 con 0x7fec0413afe0 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec040a0310 con 0x7fec04072a70 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 10 monclient(hunting): authenticate will time out at 937.035029s 2024-09-16T04:09:21.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec094be640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fec04070250 legacy=0x7fec041398c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0a4c0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fec0413afe0 0x7fec0413d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46222/0 (socket says 172.21.15.171:46222) 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0a4c0640 1 -- 172.21.15.171:0/3937657329 learned_addr learned my addr 172.21.15.171:0/3937657329 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237487484 0 0) 0x7fec040a0030 con 0x7fec0413afe0 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): my global_id is 5272 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febf0003520 con 0x7fec0413afe0 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179307801 0 0) 0x7fec040a0310 con 0x7fec04072a70 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:21.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient(hunting): my global_id is 5303 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec040a0030 con 0x7fec04072a70 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4218150710 0 0) 0x7febf0003520 con 0x7fec0413afe0 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 >> v1:172.21.15.148:6790/0 conn(0x7fec04072a70 legacy=0x7fec04072e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 >> v1:172.21.15.148:6789/0 conn(0x7fec04070250 legacy=0x7fec041398c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 monclient: found mon.noname-c 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0409f090 con 0x7fec0413afe0 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: _finish_auth 0 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 5 monclient: authenticate success, global_id 5272 2024-09-16T04:09:21.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:21.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febfc001990 con 0x7fec0413afe0 2024-09-16T04:09:21.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:21.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:21.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: dump: 2024-09-16T04:09:21.544 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:21.544 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febfc001cb0 con 0x7fec0413afe0 2024-09-16T04:09:21.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:21.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 1 -- 172.21.15.171:0/3937657329 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febfc002080 con 0x7fec0413afe0 2024-09-16T04:09:21.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:21.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec0bf23640 4 set_mon_vals no callback set 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.530+0000 7fec08cbd640 10 monclient: dump: 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:21.547 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 10 monclient: shutdown 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 1 -- 172.21.15.171:0/3937657329 >> v1:172.21.15.171:6789/0 conn(0x7fec0413afe0 legacy=0x7fec0413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 1 -- 172.21.15.171:0/3937657329 shutdown_connections 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 1 -- 172.21.15.171:0/3937657329 >> 172.21.15.171:0/3937657329 conn(0x7fec040a7520 msgr2=0x7fec040a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 1 -- 172.21.15.171:0/3937657329 shutdown_connections 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 1 -- 172.21.15.171:0/3937657329 wait complete. 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 10 monclient: build_initial_monmap 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.531+0000 7fec0bf23640 10 monclient: monmap: 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:21.548 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 1 Processor -- start 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 1 -- start start 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient: init 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:21.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient(hunting): picked mon.noname-a con 0x7fec04070250 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient(hunting): picked mon.noname-c con 0x7fec04072a70 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient(hunting): picked mon.noname-b con 0x7fec0413afe0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0413a9e0 con 0x7fec04070250 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0409e0d0 con 0x7fec0413afe0 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0406d860 con 0x7fec04072a70 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0bf23640 10 monclient(hunting): authenticate will time out at 937.037680s 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec09cbf640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fec04070250 legacy=0x7fec04139780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0a4c0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fec0413afe0 0x7fec041d3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46224/0 (socket says 172.21.15.171:46224) 2024-09-16T04:09:21.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7fec0a4c0640 1 -- 172.21.15.171:0/1086332501 learned_addr learned my addr 172.21.15.171:0/1086332501 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3899317857 0 0) 0x7fec0409e0d0 con 0x7fec0413afe0 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7febfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7febfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7febfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7febfa7fc640 10 monclient(hunting): my global_id is 5275 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.532+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe8003520 con 0x7fec0413afe0 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268093635 0 0) 0x7fec0406d860 con 0x7fec04072a70 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:21.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): my global_id is 5306 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec0409e0d0 con 0x7fec04072a70 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 90461222 0 0) 0x7febe8003520 con 0x7fec0413afe0 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec0406d860 con 0x7fec0413afe0 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febfc001950 con 0x7fec0413afe0 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.552 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4094281653 0 0) 0x7fec0406d860 con 0x7fec0413afe0 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 >> v1:172.21.15.148:6790/0 conn(0x7fec04072a70 legacy=0x7fec041d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 >> v1:172.21.15.148:6789/0 conn(0x7fec04070250 legacy=0x7fec04139780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:21.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 monclient: found mon.b 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec041259b0 con 0x7fec0413afe0 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 5 monclient: authenticate success, global_id 5275 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 10 monclient: _renew_subs 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec041cb460 con 0x7fec0413afe0 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 10 monclient: _renew_subs 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fec0413a070 con 0x7fec0413afe0 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febfc004ff0 con 0x7fec0413afe0 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febfc005340 con 0x7fec0413afe0 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfbfff640 4 set_mon_vals no callback set 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:21.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.533+0000 7febfa7fc640 10 monclient: dump: 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7febfc0107f0 con 0x7fec0413afe0 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(88..88 src has 1..88) ==== 5076+0+0 (unknown 1485255269 0 0) 0x7febfc047660 con 0x7fec0413afe0 2024-09-16T04:09:21.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7fec0bf23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:21.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7fec0bf23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:21.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7fec0bf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febec005080 con 0x7fec0413afe0 2024-09-16T04:09:21.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.534+0000 7fec09cbf640 10 monclient: get_auth_request con 0x7febe8038b90 auth_method 0 2024-09-16T04:09:21.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.539+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7febfc0161f0 con 0x7fec0413afe0 2024-09-16T04:09:21.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.539+0000 7febfa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:21.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.539+0000 7febfa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:21.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.708+0000 7fec0bf23640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:21.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.708+0000 7fec0bf23640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:21.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.708+0000 7fec0bf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:21.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:21.708+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7febec005370 con 0x7fec0413afe0 2024-09-16T04:09:22.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.336+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v89) ==== 120+0+0 (unknown 4280242338 0 0) 0x7febfc015af0 con 0x7fec0413afe0 2024-09-16T04:09:22.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.336+0000 7febfa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:22.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.336+0000 7febfa7fc640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T04:09:22.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7fec0bf23640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:22.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7fec0bf23640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:22.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7fec0bf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7febec002210 con 0x7fec0413afe0 2024-09-16T04:09:22.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7febfa7fc640 1 -- 172.21.15.171:0/1086332501 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v89) ==== 120+0+0 (unknown 3724793993 0 0) 0x7febfc012210 con 0x7fec0413afe0 2024-09-16T04:09:22.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7febfa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:22.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.444+0000 7febfa7fc640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T04:09:22.447 INFO:tasks.workunit.client.0.smithi171.stderr:device 'foo' does not appear in the crush map 2024-09-16T04:09:22.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 >> v1:172.21.15.171:6812/7327273 conn(0x7febe8038b90 legacy=0x7febe803b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:22.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 10 monclient: shutdown 2024-09-16T04:09:22.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:22.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 >> v1:172.21.15.171:6789/0 conn(0x7fec0413afe0 legacy=0x7fec041d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:22.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 shutdown_connections 2024-09-16T04:09:22.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 >> 172.21.15.171:0/1086332501 conn(0x7fec040a7520 msgr2=0x7fec040a5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:22.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 shutdown_connections 2024-09-16T04:09:22.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.446+0000 7fec0bf23640 1 -- 172.21.15.171:0/1086332501 wait complete. 2024-09-16T04:09:22.458 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.9 host2 2024-09-16T04:09:22.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.536+0000 7fc1e0ca5640 10 monclient: get_monmap_and_config 2024-09-16T04:09:22.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.536+0000 7fc1e0ca5640 10 monclient: build_initial_monmap 2024-09-16T04:09:22.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.536+0000 7fc1e0ca5640 10 monclient: monmap: 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:22.537440+0000 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:22.537440+0000 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:22.552 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 1 Processor -- start 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 1 -- start start 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient: init 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fc1dc1338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fc1dc09aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fc1dc13d240 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:22.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc1343d0 con 0x7fc1dc13d240 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc075db0 con 0x7fc1dc1338e0 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc09b1e0 con 0x7fc1dc09aa20 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1e0ca5640 10 monclient(hunting): authenticate will time out at 938.042785s 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1da59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc1dc1338e0 0x7fc1dc133cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46238/0 (socket says 172.21.15.171:46238) 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1da59c640 1 -- 172.21.15.171:0/534447634 learned_addr learned my addr 172.21.15.171:0/534447634 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.537+0000 7fc1dad9d640 1 -- 172.21.15.171:0/534447634 >> v1:172.21.15.148:6789/0 conn(0x7fc1dc13d240 legacy=0x7fc1dc13f630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:22.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3084275802 0 0) 0x7fc1dc075db0 con 0x7fc1dc1338e0 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient(hunting): my global_id is 5278 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1c8003520 con 0x7fc1dc1338e0 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3492658831 0 0) 0x7fc1c8003520 con 0x7fc1dc1338e0 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 >> v1:172.21.15.148:6790/0 conn(0x7fc1dc09aa20 legacy=0x7fc1dc13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 >> v1:172.21.15.148:6789/0 conn(0x7fc1dc13d240 legacy=0x7fc1dc13f630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:09:22.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 monclient: found mon.noname-c 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc09b4c0 con 0x7fc1dc1338e0 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: _finish_auth 0 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc1cc001610 con 0x7fc1dc1338e0 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:22.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:22.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: dump: 2024-09-16T04:09:22.557 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:22.557 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:22.557 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.557 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:22.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 5 monclient: authenticate success, global_id 5278 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1cc001880 con 0x7fc1dc1338e0 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 1 -- 172.21.15.171:0/534447634 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc1cc001c90 con 0x7fc1dc1338e0 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 4 set_mon_vals no callback set 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:22.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:22.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1d9d9b640 10 monclient: dump: 2024-09-16T04:09:22.560 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:22.560 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:22.560 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.560 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.560 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 10 monclient: shutdown 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:22.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/534447634 >> v1:172.21.15.171:6789/0 conn(0x7fc1dc1338e0 legacy=0x7fc1dc133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/534447634 shutdown_connections 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/534447634 >> 172.21.15.171:0/534447634 conn(0x7fc1dc0a28d0 msgr2=0x7fc1dc0a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.538+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/534447634 shutdown_connections 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.539+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/534447634 wait complete. 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.539+0000 7fc1e0ca5640 10 monclient: build_initial_monmap 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.539+0000 7fc1e0ca5640 10 monclient: monmap: 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:22.562 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:22.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 1 Processor -- start 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 1 -- start start 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient: init 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fc1dc09aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fc1dc1338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fc1dc13d240 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc11d850 con 0x7fc1dc13d240 2024-09-16T04:09:22.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc11db30 con 0x7fc1dc1338e0 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1dc0a56d0 con 0x7fc1dc09aa20 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1e0ca5640 10 monclient(hunting): authenticate will time out at 938.045673s 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d1d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc1dc1338e0 0x7fc1dc1d0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46248/0 (socket says 172.21.15.171:46248) 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d1d9b640 1 -- 172.21.15.171:0/2472422118 learned_addr learned my addr 172.21.15.171:0/2472422118 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1dad9d640 1 -- 172.21.15.171:0/2472422118 >> v1:172.21.15.148:6789/0 conn(0x7fc1dc13d240 legacy=0x7fc1dc1d3d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3291322944 0 0) 0x7fc1dc11db30 con 0x7fc1dc1338e0 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:22.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d37fe640 10 monclient(hunting): my global_id is 5281 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.540+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1bc003520 con 0x7fc1dc1338e0 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4225651624 0 0) 0x7fc1bc003520 con 0x7fc1dc1338e0 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1dc11db30 con 0x7fc1dc1338e0 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc1c40019b0 con 0x7fc1dc1338e0 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): dump: 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:22.566 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238760340 0 0) 0x7fc1dc0a56d0 con 0x7fc1dc09aa20 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:22.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient(hunting): my global_id is 5312 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1bc003520 con 0x7fc1dc09aa20 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3900542550 0 0) 0x7fc1dc11db30 con 0x7fc1dc1338e0 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 >> v1:172.21.15.148:6790/0 conn(0x7fc1dc09aa20 legacy=0x7fc1dc131e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 >> v1:172.21.15.148:6789/0 conn(0x7fc1dc13d240 legacy=0x7fc1dc1d3d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 monclient: found mon.b 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc13c910 con 0x7fc1dc1338e0 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 5 monclient: authenticate success, global_id 5281 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 10 monclient: _renew_subs 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc11d570 con 0x7fc1dc1338e0 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 10 monclient: _renew_subs 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1dc13be60 con 0x7fc1dc1338e0 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1c4003940 con 0x7fc1dc1338e0 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc1c4003c90 con 0x7fc1dc1338e0 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d959a640 4 set_mon_vals no callback set 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.541+0000 7fc1d37fe640 10 monclient: dump: 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:22.570 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc1c4003eb0 con 0x7fc1dc1338e0 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1e0ca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1e0ca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1e0ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c0005080 con 0x7fc1dc1338e0 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(89..89 src has 1..89) ==== 4947+0+0 (unknown 3149377711 0 0) 0x7fc1c4047420 con 0x7fc1dc1338e0 2024-09-16T04:09:22.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.542+0000 7fc1da59c640 10 monclient: get_auth_request con 0x7fc1bc038f20 auth_method 0 2024-09-16T04:09:22.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.547+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc1c4015eb0 con 0x7fc1dc1338e0 2024-09-16T04:09:22.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.547+0000 7fc1d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:22.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.547+0000 7fc1d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:22.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.720+0000 7fc1e0ca5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T04:09:22.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.720+0000 7fc1e0ca5640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T04:09:22.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.720+0000 7fc1e0ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:22.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:22.720+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) -- 0x7fc1c0005b80 con 0x7fc1dc1338e0 2024-09-16T04:09:23.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.343+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 removed item id 9 name 'osd.9' from crush map v90) ==== 143+0+0 (unknown 1244575519 0 0) 0x7fc1c40157b0 con 0x7fc1dc1338e0 2024-09-16T04:09:23.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.343+0000 7fc1d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T04:09:23.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.343+0000 7fc1d37fe640 10 monclient: _finish_command 2 = system:0 removed item id 9 name 'osd.9' from crush map 2024-09-16T04:09:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1e0ca5640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T04:09:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1e0ca5640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T04:09:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1e0ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"} v 0) -- 0x7fc1c00057a0 con 0x7fc1dc1338e0 2024-09-16T04:09:23.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1d37fe640 1 -- 172.21.15.171:0/2472422118 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}]=0 device 'osd.9' does not appear in the crush map v90) ==== 145+0+0 (unknown 2127492797 0 0) 0x7fc1c4019b00 con 0x7fc1dc1338e0 2024-09-16T04:09:23.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1d37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9", "ancestor": "host2"}] 2024-09-16T04:09:23.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.451+0000 7fc1d37fe640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T04:09:23.454 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 >> v1:172.21.15.171:6812/7327273 conn(0x7fc1bc038f20 legacy=0x7fc1bc03b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 10 monclient: shutdown 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 >> v1:172.21.15.171:6789/0 conn(0x7fc1dc1338e0 legacy=0x7fc1dc1d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 shutdown_connections 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 >> 172.21.15.171:0/2472422118 conn(0x7fc1dc0a28d0 msgr2=0x7fc1dc09e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 shutdown_connections 2024-09-16T04:09:23.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.454+0000 7fc1e0ca5640 1 -- 172.21.15.171:0/2472422118 wait complete. 2024-09-16T04:09:23.465 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm host2 2024-09-16T04:09:23.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.542+0000 7f3d8ae65640 10 monclient: get_monmap_and_config 2024-09-16T04:09:23.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.542+0000 7f3d8ae65640 10 monclient: build_initial_monmap 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.542+0000 7f3d8ae65640 10 monclient: monmap: 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:23.543676+0000 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:23.543676+0000 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:23.545 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:23.546 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:23.546 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.543+0000 7f3d8ae65640 1 Processor -- start 2024-09-16T04:09:23.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.543+0000 7f3d8ae65640 1 -- start start 2024-09-16T04:09:23.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.543+0000 7f3d8ae65640 10 monclient: init 2024-09-16T04:09:23.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:23.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:23.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8409eaa0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient(hunting): picked mon.noname-b con 0x7f3d84134b20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient(hunting): picked mon.noname-a con 0x7f3d84134f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8409d060 con 0x7f3d84134f30 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8409d340 con 0x7f3d8409eaa0 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d84068840 con 0x7f3d84134b20 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient(hunting): authenticate will time out at 939.049201s 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d89402640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f3d84134f30 legacy=0x7f3d8413d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d88c01640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3d8409eaa0 0x7f3d8409ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46262/0 (socket says 172.21.15.171:46262) 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d88c01640 1 -- 172.21.15.171:0/1359186877 learned_addr learned my addr 172.21.15.171:0/1359186877 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1853700834 0 0) 0x7f3d8409d340 con 0x7f3d8409eaa0 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:23.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient(hunting): my global_id is 5284 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d74003520 con 0x7f3d8409eaa0 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1380156763 0 0) 0x7f3d74003520 con 0x7f3d8409eaa0 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 >> v1:172.21.15.148:6790/0 conn(0x7f3d84134b20 legacy=0x7f3d84139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 >> v1:172.21.15.148:6789/0 conn(0x7f3d84134f30 legacy=0x7f3d8413d0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d84141800 con 0x7f3d8409eaa0 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:23.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d78001610 con 0x7f3d8409eaa0 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: dump: 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:23.557 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 5 monclient: authenticate success, global_id 5284 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d78001900 con 0x7f3d8409eaa0 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 1 -- 172.21.15.171:0/1359186877 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d78001cd0 con 0x7f3d8409eaa0 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d8ae65640 4 set_mon_vals no callback set 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.544+0000 7f3d837fe640 10 monclient: dump: 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:23.558 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 10 monclient: shutdown 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1359186877 >> v1:172.21.15.171:6789/0 conn(0x7f3d8409eaa0 legacy=0x7f3d8409ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:23.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1359186877 shutdown_connections 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1359186877 >> 172.21.15.171:0/1359186877 conn(0x7f3d840a1e90 msgr2=0x7f3d840a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1359186877 shutdown_connections 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.545+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1359186877 wait complete. 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.546+0000 7f3d8ae65640 10 monclient: build_initial_monmap 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.546+0000 7f3d8ae65640 10 monclient: monmap: 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:23.560 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.546+0000 7f3d8ae65640 1 Processor -- start 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.546+0000 7f3d8ae65640 1 -- start start 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.546+0000 7f3d8ae65640 10 monclient: init 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.546+0000 7f3d8ae65640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8409eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient(hunting): picked mon.noname-a con 0x7f3d84134b20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient(hunting): picked mon.noname-b con 0x7f3d84134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d84141550 con 0x7f3d84134b20 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d84141190 con 0x7f3d84134f30 2024-09-16T04:09:23.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8409c580 con 0x7f3d8409eaa0 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient(hunting): authenticate will time out at 939.052103s 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d83fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f3d84134b20 legacy=0x7f3d841d0bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d88c01640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f3d8409eaa0 0x7f3d840983e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.171:50460/0 (socket says 172.21.15.171:50460) 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d88c01640 1 -- 172.21.15.171:0/1781571221 learned_addr learned my addr 172.21.15.171:0/1781571221 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078392189 0 0) 0x7f3d8409c580 con 0x7f3d8409eaa0 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): my global_id is 5318 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d68003520 con 0x7f3d8409eaa0 2024-09-16T04:09:23.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735310461 0 0) 0x7f3d84141190 con 0x7f3d84134f30 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): my global_id is 5287 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d8409c580 con 0x7f3d84134f30 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2022123386 0 0) 0x7f3d8409c580 con 0x7f3d84134f30 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d84141190 con 0x7f3d84134f30 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d7c004330 con 0x7f3d84134f30 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): dump: 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:23.563 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1380392954 0 0) 0x7f3d68003520 con 0x7f3d8409eaa0 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d8409c580 con 0x7f3d8409eaa0 2024-09-16T04:09:23.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d780015d0 con 0x7f3d8409eaa0 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient(hunting): dump: 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:23.565 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 616083607 0 0) 0x7f3d84141190 con 0x7f3d84134f30 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 >> v1:172.21.15.148:6790/0 conn(0x7f3d8409eaa0 legacy=0x7f3d840983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 >> v1:172.21.15.148:6789/0 conn(0x7f3d84134b20 legacy=0x7f3d841d0bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:23.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 monclient: found mon.b 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8413bd90 con 0x7f3d84134f30 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 5 monclient: authenticate success, global_id 5287 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient: _renew_subs 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8413c070 con 0x7f3d84134f30 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient: _renew_subs 2024-09-16T04:09:23.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.547+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d8409a3d0 con 0x7f3d84134f30 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d7c004590 con 0x7f3d84134f30 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3d7c0048e0 con 0x7f3d84134f30 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d82ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 10 monclient: dump: 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.568 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3d7c004a80 con 0x7f3d84134f30 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d8ae65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d8ae65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d8ae65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.548+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d64005080 con 0x7f3d84134f30 2024-09-16T04:09:23.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.549+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(90..90 src has 1..90) ==== 4926+0+0 (unknown 585818070 0 0) 0x7f3d7c047450 con 0x7f3d84134f30 2024-09-16T04:09:23.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.549+0000 7f3d88c01640 10 monclient: get_auth_request con 0x7f3d68038ec0 auth_method 0 2024-09-16T04:09:23.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.553+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3d7c019a10 con 0x7f3d84134f30 2024-09-16T04:09:23.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.553+0000 7f3d817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:23.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.553+0000 7f3d817fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:23.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.723+0000 7f3d8ae65640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T04:09:23.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.723+0000 7f3d8ae65640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T04:09:23.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.723+0000 7f3d8ae65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:23.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:23.723+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) -- 0x7f3d64005370 con 0x7f3d84134f30 2024-09-16T04:09:24.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.352+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v91) ==== 124+0+0 (unknown 1113073432 0 0) 0x7f3d7c01e2d0 con 0x7f3d84134f30 2024-09-16T04:09:24.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.352+0000 7f3d817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T04:09:24.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.352+0000 7f3d817fa640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.457+0000 7f3d8ae65640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.457+0000 7f3d8ae65640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.457+0000 7f3d8ae65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.457+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) -- 0x7f3d64002210 con 0x7f3d84134f30 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.458+0000 7f3d817fa640 1 -- 172.21.15.171:0/1781571221 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v91) ==== 124+0+0 (unknown 1391722212 0 0) 0x7f3d7c044d00 con 0x7f3d84134f30 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.458+0000 7f3d817fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.458+0000 7f3d817fa640 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-16T04:09:24.460 INFO:tasks.workunit.client.0.smithi171.stderr:device 'host2' does not appear in the crush map 2024-09-16T04:09:24.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.459+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 >> v1:172.21.15.171:6812/7327273 conn(0x7f3d68038ec0 legacy=0x7f3d6803b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.459+0000 7f3d8ae65640 10 monclient: shutdown 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.459+0000 7f3d8ae65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.459+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 >> v1:172.21.15.171:6789/0 conn(0x7f3d84134f30 legacy=0x7f3d841d4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.460+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 shutdown_connections 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.460+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 >> 172.21.15.171:0/1781571221 conn(0x7f3d840a1e90 msgr2=0x7f3d840a0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.460+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 shutdown_connections 2024-09-16T04:09:24.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.460+0000 7f3d8ae65640 1 -- 172.21.15.171:0/1781571221 wait complete. 2024-09-16T04:09:24.472 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket foo host 2024-09-16T04:09:24.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.548+0000 7ff226274640 10 monclient: get_monmap_and_config 2024-09-16T04:09:24.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.548+0000 7ff226274640 10 monclient: build_initial_monmap 2024-09-16T04:09:24.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.548+0000 7ff226274640 10 monclient: monmap: 2024-09-16T04:09:24.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:24.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:24.552 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:24.549647+0000 2024-09-16T04:09:24.552 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:24.549647+0000 2024-09-16T04:09:24.552 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:24.552 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:24.552 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:24.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:24.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:24.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:24.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.549+0000 7ff226274640 1 Processor -- start 2024-09-16T04:09:24.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.549+0000 7ff226274640 1 -- start start 2024-09-16T04:09:24.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.549+0000 7ff226274640 10 monclient: init 2024-09-16T04:09:24.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient(hunting): picked mon.noname-c con 0x7ff2201316b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient(hunting): picked mon.noname-b con 0x7ff220137800 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient(hunting): picked mon.noname-a con 0x7ff22013b310 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2201321a0 con 0x7ff22013b310 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22009e9e0 con 0x7ff2201316b0 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2200a2120 con 0x7ff220137800 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient(hunting): authenticate will time out at 940.055189s 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff2201316b0 0x7ff220131a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46272/0 (socket says 172.21.15.171:46272) 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff224811640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff22013b310 legacy=0x7ff22013d700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21ffff640 1 -- 172.21.15.171:0/1564568987 learned_addr learned my addr 172.21.15.171:0/1564568987 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:24.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2348625191 0 0) 0x7ff22009e9e0 con 0x7ff2201316b0 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): my global_id is 5290 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff208003520 con 0x7ff2201316b0 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2657251156 0 0) 0x7ff2200a2120 con 0x7ff220137800 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:24.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient(hunting): my global_id is 5321 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff22009e9e0 con 0x7ff220137800 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4163969502 0 0) 0x7ff208003520 con 0x7ff2201316b0 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 >> v1:172.21.15.148:6790/0 conn(0x7ff220137800 legacy=0x7ff220139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 >> v1:172.21.15.148:6789/0 conn(0x7ff22013b310 legacy=0x7ff22013d700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 monclient: found mon.noname-c 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:24.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22009b6d0 con 0x7ff2201316b0 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff210001610 con 0x7ff2201316b0 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 5 monclient: authenticate success, global_id 5290 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: dump: 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:24.570 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:24.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff210001870 con 0x7ff2201316b0 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 1 -- 172.21.15.171:0/1564568987 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff210001c40 con 0x7ff2201316b0 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff226274640 4 set_mon_vals no callback set 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.550+0000 7ff21effd640 10 monclient: dump: 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.572 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 10 monclient: shutdown 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 1 -- 172.21.15.171:0/1564568987 >> v1:172.21.15.171:6789/0 conn(0x7ff2201316b0 legacy=0x7ff220131a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 1 -- 172.21.15.171:0/1564568987 shutdown_connections 2024-09-16T04:09:24.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 1 -- 172.21.15.171:0/1564568987 >> 172.21.15.171:0/1564568987 conn(0x7ff2200a5370 msgr2=0x7ff2200a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 1 -- 172.21.15.171:0/1564568987 shutdown_connections 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 1 -- 172.21.15.171:0/1564568987 wait complete. 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 10 monclient: build_initial_monmap 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.551+0000 7ff226274640 10 monclient: monmap: 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:24.574 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 1 Processor -- start 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 1 -- start start 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient: init 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:24.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient(hunting): picked mon.noname-b con 0x7ff2201316b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient(hunting): picked mon.noname-a con 0x7ff220137800 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient(hunting): picked mon.noname-c con 0x7ff22013b310 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22011b580 con 0x7ff220137800 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22011b860 con 0x7ff2201316b0 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff22006ce80 con 0x7ff22013b310 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff226274640 10 monclient(hunting): authenticate will time out at 940.057889s 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff21f7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff220137800 legacy=0x7ff2201ce6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:24.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff21ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff2201316b0 0x7ff2201c9700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46276/0 (socket says 172.21.15.171:46276) 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.552+0000 7ff21ffff640 1 -- 172.21.15.171:0/15636066 learned_addr learned my addr 172.21.15.171:0/15636066 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833183134 0 0) 0x7ff22011b860 con 0x7ff2201316b0 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): my global_id is 5293 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff20c003520 con 0x7ff2201316b0 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329387090 0 0) 0x7ff22006ce80 con 0x7ff22013b310 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:24.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): my global_id is 5324 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff22011b860 con 0x7ff22013b310 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2281422177 0 0) 0x7ff20c003520 con 0x7ff2201316b0 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff22006ce80 con 0x7ff2201316b0 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff210001510 con 0x7ff2201316b0 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): dump: 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:24.578 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 856327472 0 0) 0x7ff22006ce80 con 0x7ff2201316b0 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 >> v1:172.21.15.148:6790/0 conn(0x7ff22013b310 legacy=0x7ff2201d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 >> v1:172.21.15.148:6789/0 conn(0x7ff220137800 legacy=0x7ff2201ce6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 monclient: found mon.b 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff220136c00 con 0x7ff2201316b0 2024-09-16T04:09:24.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 5 monclient: authenticate success, global_id 5293 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 10 monclient: _renew_subs 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2200a4630 con 0x7ff2201316b0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 10 monclient: _renew_subs 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff220135a60 con 0x7ff2201316b0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff210004bf0 con 0x7ff2201316b0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff210004fc0 con 0x7ff2201316b0 2024-09-16T04:09:24.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21e7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.553+0000 7ff21cff9640 10 monclient: dump: 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:24.581 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff2100103f0 con 0x7ff2201316b0 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff226274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff226274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff226274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff200005080 con 0x7ff2201316b0 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(91..91 src has 1..91) ==== 4791+0+0 (unknown 2198084550 0 0) 0x7ff210047200 con 0x7ff2201316b0 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.554+0000 7ff21f7fe640 10 monclient: get_auth_request con 0x7ff20c038b90 auth_method 0 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.559+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff210015cf0 con 0x7ff2201316b0 2024-09-16T04:09:24.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.559+0000 7ff21cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:24.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.559+0000 7ff21cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:24.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.728+0000 7ff226274640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T04:09:24.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.728+0000 7ff226274640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T04:09:24.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.728+0000 7ff226274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:24.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:24.728+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) -- 0x7ff200005b80 con 0x7ff2201316b0 2024-09-16T04:09:25.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.358+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v92) ==== 138+0+0 (unknown 2515722968 0 0) 0x7ff2100155f0 con 0x7ff2201316b0 2024-09-16T04:09:25.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.358+0000 7ff21cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T04:09:25.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.358+0000 7ff21cff9640 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-16T04:09:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff226274640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T04:09:25.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff226274640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T04:09:25.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff226274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) -- 0x7ff200001090 con 0x7ff2201316b0 2024-09-16T04:09:25.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff21cff9640 1 -- 172.21.15.171:0/15636066 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v92) ==== 126+0+0 (unknown 668897662 0 0) 0x7ff21001e140 con 0x7ff2201316b0 2024-09-16T04:09:25.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff21cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-16T04:09:25.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.460+0000 7ff21cff9640 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-16T04:09:25.463 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'foo' already exists 2024-09-16T04:09:25.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 >> v1:172.21.15.171:6812/7327273 conn(0x7ff20c038b90 legacy=0x7ff20c03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:25.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 10 monclient: shutdown 2024-09-16T04:09:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 >> v1:172.21.15.171:6789/0 conn(0x7ff2201316b0 legacy=0x7ff2201c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 shutdown_connections 2024-09-16T04:09:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 >> 172.21.15.171:0/15636066 conn(0x7ff2200a5370 msgr2=0x7ff2200a29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 shutdown_connections 2024-09-16T04:09:25.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.462+0000 7ff226274640 1 -- 172.21.15.171:0/15636066 wait complete. 2024-09-16T04:09:25.473 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.554+0000 7f90a3b31640 10 monclient: get_monmap_and_config 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.554+0000 7f90a3b31640 10 monclient: build_initial_monmap 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.554+0000 7f90a3b31640 10 monclient: monmap: 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:25.554927+0000 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:25.554927+0000 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:25.556 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:25.557 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:25.557 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.554+0000 7f90a3b31640 1 Processor -- start 2024-09-16T04:09:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.554+0000 7f90a3b31640 1 -- start start 2024-09-16T04:09:25.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.554+0000 7f90a3b31640 10 monclient: init 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient(hunting): picked mon.noname-a con 0x7f909c136980 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient(hunting): picked mon.noname-b con 0x7f909c09c1e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient(hunting): picked mon.noname-c con 0x7f909c137470 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c09aa20 con 0x7f909c136980 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c09af40 con 0x7f909c137470 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c073920 con 0x7f909c09c1e0 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a3b31640 10 monclient(hunting): authenticate will time out at 941.060448s 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a20ce640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f909c137470 0x7f909c13f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46292/0 (socket says 172.21.15.171:46292) 2024-09-16T04:09:25.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a20ce640 1 -- 172.21.15.171:0/3694024564 learned_addr learned my addr 172.21.15.171:0/3694024564 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a18cd640 1 -- 172.21.15.171:0/3694024564 >> v1:172.21.15.148:6789/0 conn(0x7f909c136980 legacy=0x7f909c136d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245206077 0 0) 0x7f909c09af40 con 0x7f909c137470 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 10 monclient(hunting): my global_id is 5296 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9084003520 con 0x7f909c137470 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3992740681 0 0) 0x7f9084003520 con 0x7f909c137470 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 >> v1:172.21.15.148:6790/0 conn(0x7f909c09c1e0 legacy=0x7f909c13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.555+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 >> v1:172.21.15.148:6789/0 conn(0x7f909c136980 legacy=0x7f909c136d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 1 monclient: found mon.noname-c 2024-09-16T04:09:25.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c0a5300 con 0x7f909c137470 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: _finish_auth 0 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f908c001610 con 0x7f909c137470 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: dump: 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.567 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 5 monclient: authenticate success, global_id 5296 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f908c001970 con 0x7f909c137470 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 1 -- 172.21.15.171:0/3694024564 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f908c001d40 con 0x7f909c137470 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:25.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 4 set_mon_vals no callback set 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a08cb640 10 monclient: dump: 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:25.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 10 monclient: shutdown 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 1 -- 172.21.15.171:0/3694024564 >> v1:172.21.15.171:6789/0 conn(0x7f909c137470 legacy=0x7f909c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 1 -- 172.21.15.171:0/3694024564 shutdown_connections 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 1 -- 172.21.15.171:0/3694024564 >> 172.21.15.171:0/3694024564 conn(0x7f909c0a9970 msgr2=0x7f909c0a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 1 -- 172.21.15.171:0/3694024564 shutdown_connections 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.556+0000 7f90a3b31640 1 -- 172.21.15.171:0/3694024564 wait complete. 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 10 monclient: build_initial_monmap 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 10 monclient: monmap: 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:25.570 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 1 Processor -- start 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 1 -- start start 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 10 monclient: init 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.557+0000 7f90a3b31640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient(hunting): picked mon.noname-a con 0x7f909c09c1e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient(hunting): picked mon.noname-c con 0x7f909c136980 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient(hunting): picked mon.noname-b con 0x7f909c137470 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:25.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c134040 con 0x7f909c09c1e0 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c133c20 con 0x7f909c137470 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c0a8480 con 0x7f909c136980 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient(hunting): authenticate will time out at 941.063096s 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a18cd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f909c09c1e0 legacy=0x7f909c130d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a20ce640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f909c137470 0x7f909c1d3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46308/0 (socket says 172.21.15.171:46308) 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a20ce640 1 -- 172.21.15.171:0/1311621108 learned_addr learned my addr 172.21.15.171:0/1311621108 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020663208 0 0) 0x7f909c133c20 con 0x7f909c137470 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): my global_id is 5299 2024-09-16T04:09:25.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9088003520 con 0x7f909c137470 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 499835764 0 0) 0x7f909c0a8480 con 0x7f909c136980 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): my global_id is 5330 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f909c133c20 con 0x7f909c136980 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1696770788 0 0) 0x7f9088003520 con 0x7f909c137470 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f909c0a8480 con 0x7f909c137470 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f908c0015d0 con 0x7f909c137470 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:25.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2842312870 0 0) 0x7f909c0a8480 con 0x7f909c137470 2024-09-16T04:09:25.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 >> v1:172.21.15.148:6790/0 conn(0x7f909c136980 legacy=0x7f909c131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 >> v1:172.21.15.148:6789/0 conn(0x7f909c09c1e0 legacy=0x7f909c130d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 monclient: found mon.b 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c131ba0 con 0x7f909c137470 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 5 monclient: authenticate success, global_id 5299 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient: _renew_subs 2024-09-16T04:09:25.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f909c133940 con 0x7f909c137470 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient: _renew_subs 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f909c132490 con 0x7f909c137470 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f908c004cc0 con 0x7f909c137470 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f908c004e70 con 0x7f909c137470 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909bfff640 4 set_mon_vals no callback set 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.558+0000 7f909a7fc640 10 monclient: dump: 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:25.576 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.559+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f908c0102a0 con 0x7f909c137470 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.559+0000 7f90a3b31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.559+0000 7f90a3b31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.559+0000 7f90a3b31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.559+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f907c005080 con 0x7f909c137470 2024-09-16T04:09:25.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.559+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(92..92 src has 1..92) ==== 4818+0+0 (unknown 1041179440 0 0) 0x7f908c047320 con 0x7f909c137470 2024-09-16T04:09:25.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.560+0000 7f90a18cd640 10 monclient: get_auth_request con 0x7f9088039100 auth_method 0 2024-09-16T04:09:25.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.564+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f908c015e10 con 0x7f909c137470 2024-09-16T04:09:25.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.564+0000 7f909a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:25.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.564+0000 7f909a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:25.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.737+0000 7f90a3b31640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T04:09:25.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.737+0000 7f90a3b31640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T04:09:25.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.737+0000 7f90a3b31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:25.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.737+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) -- 0x7f907c005370 con 0x7f909c137470 2024-09-16T04:09:25.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.988+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v93) ==== 204+0+0 (unknown 4776441 0 0) 0x7f908c015710 con 0x7f909c137470 2024-09-16T04:09:25.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.988+0000 7f909a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T04:09:25.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:25.989+0000 7f909a7fc640 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T04:09:26.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.097+0000 7f90a3b31640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T04:09:26.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.097+0000 7f90a3b31640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T04:09:26.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.097+0000 7f90a3b31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.097+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) -- 0x7f907c005bd0 con 0x7f909c137470 2024-09-16T04:09:26.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.098+0000 7f909a7fc640 1 -- 172.21.15.171:0/1311621108 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v93) ==== 214+0+0 (unknown 959808750 0 0) 0x7f908c01e260 con 0x7f909c137470 2024-09-16T04:09:26.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.098+0000 7f909a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-16T04:09:26.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.098+0000 7f909a7fc640 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T04:09:26.100 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 >> v1:172.21.15.171:6812/7327273 conn(0x7f9088039100 legacy=0x7f908803b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 10 monclient: shutdown 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 >> v1:172.21.15.171:6789/0 conn(0x7f909c137470 legacy=0x7f909c1d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 shutdown_connections 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 >> 172.21.15.171:0/1311621108 conn(0x7f909c0a9970 msgr2=0x7f909c0a6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 shutdown_connections 2024-09-16T04:09:26.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.100+0000 7f90a3b31640 1 -- 172.21.15.171:0/1311621108 wait complete. 2024-09-16T04:09:26.111 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush create-or-move osd.7 1.0 root=default 2024-09-16T04:09:26.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.189+0000 7fc9bbdb9640 10 monclient: get_monmap_and_config 2024-09-16T04:09:26.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.189+0000 7fc9bbdb9640 10 monclient: build_initial_monmap 2024-09-16T04:09:26.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.189+0000 7fc9bbdb9640 10 monclient: monmap: 2024-09-16T04:09:26.191 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:26.190590+0000 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:26.190590+0000 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:26.192 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:26.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.190+0000 7fc9bbdb9640 1 Processor -- start 2024-09-16T04:09:26.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.190+0000 7fc9bbdb9640 1 -- start start 2024-09-16T04:09:26.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.190+0000 7fc9bbdb9640 10 monclient: init 2024-09-16T04:09:26.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:26.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient(hunting): picked mon.noname-b con 0x7fc9b409ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b40a0370 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b40a0800 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b40a0f20 con 0x7fc9b40a0800 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b40a1200 con 0x7fc9b40a0370 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b409bca0 con 0x7fc9b409ff90 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient(hunting): authenticate will time out at 941.696177s 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9ba356640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc9b40a0800 legacy=0x7fc9b413d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b9354640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc9b40a0370 0x7fc9b4139a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46316/0 (socket says 172.21.15.171:46316) 2024-09-16T04:09:26.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b9354640 1 -- 172.21.15.171:0/900323630 learned_addr learned my addr 172.21.15.171:0/900323630 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1597525633 0 0) 0x7fc9b40a1200 con 0x7fc9b40a0370 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): _negotiate 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): my global_id is 5302 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9a0003520 con 0x7fc9b40a0370 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896159675 0 0) 0x7fc9b409bca0 con 0x7fc9b409ff90 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): _negotiate 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient(hunting): my global_id is 5333 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9b40a1200 con 0x7fc9b409ff90 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2821462027 0 0) 0x7fc9a0003520 con 0x7fc9b40a0370 2024-09-16T04:09:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 >> v1:172.21.15.148:6790/0 conn(0x7fc9b409ff90 legacy=0x7fc9b409f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 >> v1:172.21.15.148:6789/0 conn(0x7fc9b40a0800 legacy=0x7fc9b413d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: _finish_hunting 0 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 monclient: found mon.noname-c 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b409e830 con 0x7fc9b40a0370 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: _finish_auth 0 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: _check_auth_tickets 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 5 monclient: authenticate success, global_id 5302 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9a4001610 con 0x7fc9b40a0370 2024-09-16T04:09:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: dump: 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:26.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9a4001930 con 0x7fc9b40a0370 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9bbdb9640 4 set_mon_vals no callback set 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 1 -- 172.21.15.171:0/900323630 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9a4001d00 con 0x7fc9b40a0370 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.191+0000 7fc9b8b53640 10 monclient: dump: 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 10 monclient: shutdown 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/900323630 >> v1:172.21.15.171:6789/0 conn(0x7fc9b40a0370 legacy=0x7fc9b4139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/900323630 shutdown_connections 2024-09-16T04:09:26.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/900323630 >> 172.21.15.171:0/900323630 conn(0x7fc9b40a5450 msgr2=0x7fc9b40a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/900323630 shutdown_connections 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/900323630 wait complete. 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 10 monclient: build_initial_monmap 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.192+0000 7fc9bbdb9640 10 monclient: monmap: 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:26.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 1 Processor -- start 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 1 -- start start 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient: init 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient(hunting): picked mon.noname-b con 0x7fc9b409ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient(hunting): picked mon.noname-a con 0x7fc9b40a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient(hunting): picked mon.noname-c con 0x7fc9b40a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b409c0b0 con 0x7fc9b40a0370 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b4127a90 con 0x7fc9b409ff90 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b4071780 con 0x7fc9b40a0800 2024-09-16T04:09:26.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9bbdb9640 10 monclient(hunting): authenticate will time out at 941.698727s 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9b9b55640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc9b409ff90 0x7fc9b41cbb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:46320/0 (socket says 172.21.15.171:46320) 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9b9b55640 1 -- 172.21.15.171:0/4278474523 learned_addr learned my addr 172.21.15.171:0/4278474523 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9b9354640 1 -- 172.21.15.171:0/4278474523 >> v1:172.21.15.148:6789/0 conn(0x7fc9b40a0370 legacy=0x7fc9b41d0b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896697690 0 0) 0x7fc9b4127a90 con 0x7fc9b409ff90 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9aa7fc640 10 monclient(hunting): my global_id is 5305 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.193+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc998003520 con 0x7fc9b409ff90 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2192563697 0 0) 0x7fc998003520 con 0x7fc9b409ff90 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9b4127a90 con 0x7fc9b409ff90 2024-09-16T04:09:26.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9b00018b0 con 0x7fc9b409ff90 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:26.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 114991404 0 0) 0x7fc9b4127a90 con 0x7fc9b409ff90 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 >> v1:172.21.15.148:6790/0 conn(0x7fc9b40a0800 legacy=0x7fc9b41d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 >> v1:172.21.15.148:6789/0 conn(0x7fc9b40a0370 legacy=0x7fc9b41d0b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 monclient: found mon.b 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b406ec20 con 0x7fc9b409ff90 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:26.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 5 monclient: authenticate success, global_id 5305 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 10 monclient: _renew_subs 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b41274d0 con 0x7fc9b409ff90 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 10 monclient: _renew_subs 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9b4126320 con 0x7fc9b409ff90 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9b0001c90 con 0x7fc9b409ff90 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9b0003a60 con 0x7fc9b409ff90 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:26.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9aa7fc640 10 monclient: dump: 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.194+0000 7fc9abfff640 4 set_mon_vals no callback set 2024-09-16T04:09:26.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc9b0003cc0 con 0x7fc9b409ff90 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9bbdb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9bbdb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9bbdb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4127cd0 con 0x7fc9b409ff90 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(93..93 src has 1..93) ==== 5115+0+0 (unknown 1800706899 0 0) 0x7fc9b00473e0 con 0x7fc9b409ff90 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.195+0000 7fc9b9354640 10 monclient: get_auth_request con 0x7fc998038ee0 auth_method 0 2024-09-16T04:09:26.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.199+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc9b0015cd0 con 0x7fc9b409ff90 2024-09-16T04:09:26.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.200+0000 7fc9aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:26.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.200+0000 7fc9aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:26.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.371+0000 7fc9bbdb9640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T04:09:26.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.371+0000 7fc9bbdb9640 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T04:09:26.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.371+0000 7fc9bbdb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:26.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:26.371+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) -- 0x7fc9b411eb30 con 0x7fc9b409ff90 2024-09-16T04:09:27.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.363+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map v94) ==== 212+0+0 (unknown 2865443337 0 0) 0x7fc9b00155d0 con 0x7fc9b409ff90 2024-09-16T04:09:27.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.363+0000 7fc9aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T04:09:27.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.363+0000 7fc9aa7fc640 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T04:09:27.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9bbdb9640 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T04:09:27.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9bbdb9640 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T04:09:27.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9bbdb9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]} v 0) -- 0x7fc9b406bd60 con 0x7fc9b409ff90 2024-09-16T04:09:27.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9aa7fc640 1 -- 172.21.15.171:0/4278474523 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map v94) ==== 211+0+0 (unknown 643913737 0 0) 0x7fc9b0019920 con 0x7fc9b409ff90 2024-09-16T04:09:27.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9aa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 7, "weight": 1.0, "args": ["root=default"]}] 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.472+0000 7fc9aa7fc640 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:create-or-move updated item name 'osd.7' weight 1 at location {root=default} to crush map 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 >> v1:172.21.15.171:6812/7327273 conn(0x7fc998038ee0 legacy=0x7fc99803b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 10 monclient: shutdown 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 >> v1:172.21.15.171:6789/0 conn(0x7fc9b409ff90 legacy=0x7fc9b41cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 shutdown_connections 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 >> 172.21.15.171:0/4278474523 conn(0x7fc9b40a5450 msgr2=0x7fc9b4071020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 shutdown_connections 2024-09-16T04:09:27.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.474+0000 7fc9bbdb9640 1 -- 172.21.15.171:0/4278474523 wait complete. 2024-09-16T04:09:27.486 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush move osd.7 host=foo 2024-09-16T04:09:27.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.565+0000 7f279f926640 10 monclient: get_monmap_and_config 2024-09-16T04:09:27.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.565+0000 7f279f926640 10 monclient: build_initial_monmap 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.565+0000 7f279f926640 10 monclient: monmap: 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:27.566669+0000 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:27.566669+0000 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:27.568 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.566+0000 7f279f926640 1 Processor -- start 2024-09-16T04:09:27.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.566+0000 7f279f926640 1 -- start start 2024-09-16T04:09:27.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.566+0000 7f279f926640 10 monclient: init 2024-09-16T04:09:27.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:27.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:27.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient(hunting): picked mon.noname-c con 0x7f2798132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:27.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient(hunting): picked mon.noname-b con 0x7f279809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:27.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient(hunting): picked mon.noname-a con 0x7f2798132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:27.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27981332b0 con 0x7f2798132e60 2024-09-16T04:09:27.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2798134310 con 0x7f2798132a80 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279809d980 con 0x7f279809ec20 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient(hunting): authenticate will time out at 943.072201s 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279d6c2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2798132a80 0x7f27981334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42034/0 (socket says 172.21.15.171:42034) 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279d6c2640 1 -- 172.21.15.171:0/255450269 learned_addr learned my addr 172.21.15.171:0/255450269 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279dec3640 1 -- 172.21.15.171:0/255450269 >> v1:172.21.15.148:6789/0 conn(0x7f2798132e60 legacy=0x7f279813c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425923919 0 0) 0x7f2798134310 con 0x7f2798132a80 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:27.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient(hunting): my global_id is 5308 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2780003520 con 0x7f2798132a80 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1418936019 0 0) 0x7f2780003520 con 0x7f2798132a80 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 >> v1:172.21.15.148:6790/0 conn(0x7f279809ec20 legacy=0x7f2798133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 >> v1:172.21.15.148:6789/0 conn(0x7f2798132e60 legacy=0x7f279813c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 monclient: found mon.noname-c 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279809dc60 con 0x7f2798132a80 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:27.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2788001610 con 0x7f2798132a80 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 5 monclient: authenticate success, global_id 5308 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: dump: 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:27.579 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2788001880 con 0x7f2798132a80 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 1 -- 172.21.15.171:0/255450269 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2788001c50 con 0x7f2798132a80 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f279f926640 4 set_mon_vals no callback set 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:27.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.567+0000 7f2797fff640 10 monclient: dump: 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 10 monclient: shutdown 2024-09-16T04:09:27.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 1 -- 172.21.15.171:0/255450269 >> v1:172.21.15.171:6789/0 conn(0x7f2798132a80 legacy=0x7f27981334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 1 -- 172.21.15.171:0/255450269 shutdown_connections 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 1 -- 172.21.15.171:0/255450269 >> 172.21.15.171:0/255450269 conn(0x7f27980a52a0 msgr2=0x7f27980a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 1 -- 172.21.15.171:0/255450269 shutdown_connections 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.568+0000 7f279f926640 1 -- 172.21.15.171:0/255450269 wait complete. 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient: build_initial_monmap 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient: monmap: 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:27.582 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 1 Processor -- start 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 1 -- start start 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient: init 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient(hunting): picked mon.noname-a con 0x7f279809ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient(hunting): picked mon.noname-c con 0x7f2798132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient(hunting): picked mon.noname-b con 0x7f2798132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:27.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279811e4b0 con 0x7f279809ec20 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279811e790 con 0x7f2798132e60 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279811c0b0 con 0x7f2798132a80 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279f926640 10 monclient(hunting): authenticate will time out at 943.074869s 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279d6c2640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f279809ec20 legacy=0x7f279811b290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279dec3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2798132e60 0x7f27981cf630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42046/0 (socket says 172.21.15.171:42046) 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.569+0000 7f279dec3640 1 -- 172.21.15.171:0/3802855267 learned_addr learned my addr 172.21.15.171:0/3802855267 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224257190 0 0) 0x7f279811e790 con 0x7f2798132e60 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:27.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): my global_id is 5311 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2784003520 con 0x7f2798132e60 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974947869 0 0) 0x7f279811c0b0 con 0x7f2798132a80 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): my global_id is 5342 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f279811e790 con 0x7f2798132a80 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4240826329 0 0) 0x7f2784003520 con 0x7f2798132e60 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f279811c0b0 con 0x7f2798132e60 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2790004440 con 0x7f2798132e60 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:27.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): dump: 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:27.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1304000567 0 0) 0x7f279811e790 con 0x7f2798132a80 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2784003520 con 0x7f2798132a80 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f278c001830 con 0x7f2798132a80 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient(hunting): dump: 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:27.587 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 978193038 0 0) 0x7f279811c0b0 con 0x7f2798132e60 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 >> v1:172.21.15.148:6790/0 conn(0x7f2798132a80 legacy=0x7f279811b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 >> v1:172.21.15.148:6789/0 conn(0x7f279809ec20 legacy=0x7f279811b290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 monclient: found mon.b 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279809c810 con 0x7f2798132e60 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:27.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 5 monclient: authenticate success, global_id 5311 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 10 monclient: _renew_subs 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f279806ceb0 con 0x7f2798132e60 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f27900046f0 con 0x7f2798132e60 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2790004ac0 con 0x7f2798132e60 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f2795ffb640 10 monclient: dump: 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:27.589 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 10 monclient: _renew_subs 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f27977fe640 4 set_mon_vals no callback set 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.570+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f279811d1b0 con 0x7f2798132e60 2024-09-16T04:09:27.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2790004d60 con 0x7f2798132e60 2024-09-16T04:09:27.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f279f926640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:27.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(94..94 src has 1..94) ==== 5136+0+0 (unknown 1841918606 0 0) 0x7f2790047720 con 0x7f2798132e60 2024-09-16T04:09:27.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f279f926640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:27.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f279f926640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279806f730 con 0x7f2798132e60 2024-09-16T04:09:27.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.571+0000 7f279d6c2640 10 monclient: get_auth_request con 0x7f2784038ea0 auth_method 0 2024-09-16T04:09:27.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.576+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2790016020 con 0x7f2798132e60 2024-09-16T04:09:27.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.576+0000 7f2795ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:27.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.576+0000 7f2795ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:27.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.746+0000 7f279f926640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T04:09:27.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.746+0000 7f279f926640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T04:09:27.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.746+0000 7f279f926640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:27.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:27.746+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) -- 0x7f279806bcf0 con 0x7f2798132e60 2024-09-16T04:09:28.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.364+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 moved item id 7 name 'osd.7' to location {host=foo} in crush map v95) ==== 165+0+0 (unknown 1684795023 0 0) 0x7f2790015920 con 0x7f2798132e60 2024-09-16T04:09:28.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.364+0000 7f2795ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T04:09:28.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.364+0000 7f2795ffb640 10 monclient: _finish_command 2 = system:0 moved item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T04:09:28.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f279f926640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T04:09:28.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f279f926640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T04:09:28.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f279f926640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]} v 0) -- 0x7f279811ebb0 con 0x7f2798132e60 2024-09-16T04:09:28.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f2795ffb640 1 -- 172.21.15.171:0/3802855267 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}]=0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map v95) ==== 175+0+0 (unknown 656088108 0 0) 0x7f279001e4d0 con 0x7f2798132e60 2024-09-16T04:09:28.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f2795ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.7", "args": ["host=foo"]}] 2024-09-16T04:09:28.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.471+0000 7f2795ffb640 10 monclient: _finish_command 3 = system:0 no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T04:09:28.475 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id 7 name 'osd.7' to location {host=foo} in crush map 2024-09-16T04:09:28.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 >> v1:172.21.15.171:6812/7327273 conn(0x7f2784038ea0 legacy=0x7f278403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 10 monclient: shutdown 2024-09-16T04:09:28.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:28.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 >> v1:172.21.15.171:6789/0 conn(0x7f2798132e60 legacy=0x7f27981cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 shutdown_connections 2024-09-16T04:09:28.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 >> 172.21.15.171:0/3802855267 conn(0x7f27980a52a0 msgr2=0x7f2798131860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:28.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 shutdown_connections 2024-09-16T04:09:28.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.473+0000 7f279f926640 1 -- 172.21.15.171:0/3802855267 wait complete. 2024-09-16T04:09:28.486 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd find osd.7 2024-09-16T04:09:28.486 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep host 2024-09-16T04:09:28.486 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep foo 2024-09-16T04:09:28.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.561+0000 7fbc1d5e8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:28.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.561+0000 7fbc1d5e8640 10 monclient: build_initial_monmap 2024-09-16T04:09:28.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.561+0000 7fbc1d5e8640 10 monclient: monmap: 2024-09-16T04:09:28.564 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:28.564 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:28.563403+0000 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:28.563403+0000 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:28.565 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 1 Processor -- start 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 1 -- start start 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient: init 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient(hunting): picked mon.noname-a con 0x7fbc18136cf0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient(hunting): picked mon.noname-b con 0x7fbc18139ad0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1813d5e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc180a4f90 con 0x7fbc18136cf0 2024-09-16T04:09:28.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc180a0a30 con 0x7fbc1813d5e0 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc180a0d10 con 0x7fbc18139ad0 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc1d5e8640 10 monclient(hunting): authenticate will time out at 944.068868s 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc16ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbc18136cf0 legacy=0x7fbc181370d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc177fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbc1813d5e0 0x7fbc1813f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42052/0 (socket says 172.21.15.171:42052) 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.562+0000 7fbc177fe640 1 -- 172.21.15.171:0/3937912954 learned_addr learned my addr 172.21.15.171:0/3937912954 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617598127 0 0) 0x7fbc180a0a30 con 0x7fbc1813d5e0 2024-09-16T04:09:28.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient(hunting): my global_id is 5314 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc04003520 con 0x7fbc1813d5e0 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3069279193 0 0) 0x7fbc04003520 con 0x7fbc1813d5e0 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 >> v1:172.21.15.148:6790/0 conn(0x7fbc18139ad0 legacy=0x7fbc1813bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 >> v1:172.21.15.148:6789/0 conn(0x7fbc18136cf0 legacy=0x7fbc181370d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:28.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 monclient: found mon.noname-c 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc180a0ff0 con 0x7fbc1813d5e0 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbc10001990 con 0x7fbc1813d5e0 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc1d5e8640 5 monclient: authenticate success, global_id 5314 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc1d5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:28.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: dump: 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc1d5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:28.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc10001c00 con 0x7fbc1813d5e0 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 1 -- 172.21.15.171:0/3937912954 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbc10002010 con 0x7fbc1813d5e0 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc1d5e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc1d5e8640 4 set_mon_vals no callback set 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.563+0000 7fbc15ffb640 10 monclient: dump: 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.578 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 10 monclient: shutdown 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/3937912954 >> v1:172.21.15.171:6789/0 conn(0x7fbc1813d5e0 legacy=0x7fbc1813f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/3937912954 shutdown_connections 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/3937912954 >> 172.21.15.171:0/3937912954 conn(0x7fbc180a9830 msgr2=0x7fbc180a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/3937912954 shutdown_connections 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/3937912954 wait complete. 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 10 monclient: build_initial_monmap 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.564+0000 7fbc1d5e8640 10 monclient: monmap: 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:28.579 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 1 Processor -- start 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 1 -- start start 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient: init 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:28.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient(hunting): picked mon.noname-c con 0x7fbc18136cf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient(hunting): picked mon.noname-b con 0x7fbc18139ad0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1813d5e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc180a8880 con 0x7fbc1813d5e0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc1813acd0 con 0x7fbc18139ad0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc180a01d0 con 0x7fbc18136cf0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc1d5e8640 10 monclient(hunting): authenticate will time out at 944.071627s 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc177fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbc1813d5e0 legacy=0x7fbc181d37b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc167fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbc18139ad0 0x7fbc1813a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42064/0 (socket says 172.21.15.171:42064) 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc167fc640 1 -- 172.21.15.171:0/599078712 learned_addr learned my addr 172.21.15.171:0/599078712 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4154191648 0 0) 0x7fbc1813acd0 con 0x7fbc18139ad0 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:28.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): my global_id is 5317 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbf8003520 con 0x7fbc18139ad0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487311621 0 0) 0x7fbc180a01d0 con 0x7fbc18136cf0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.565+0000 7fbc0f7fe640 10 monclient(hunting): my global_id is 5348 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc1813acd0 con 0x7fbc18136cf0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1762468778 0 0) 0x7fbbf8003520 con 0x7fbc18139ad0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbc180a01d0 con 0x7fbc18139ad0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbc080019b0 con 0x7fbc18139ad0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:28.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): dump: 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 653376403 0 0) 0x7fbc1813acd0 con 0x7fbc18136cf0 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbf8003520 con 0x7fbc18136cf0 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbc00004140 con 0x7fbc18136cf0 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient(hunting): dump: 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.584 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1079594184 0 0) 0x7fbc180a01d0 con 0x7fbc18139ad0 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 >> v1:172.21.15.148:6790/0 conn(0x7fbc18136cf0 legacy=0x7fbc18139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 >> v1:172.21.15.148:6789/0 conn(0x7fbc1813d5e0 legacy=0x7fbc181d37b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 monclient: found mon.b 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:28.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1813aff0 con 0x7fbc18139ad0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 5 monclient: authenticate success, global_id 5317 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 10 monclient: _renew_subs 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1809ae30 con 0x7fbc18139ad0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 10 monclient: _renew_subs 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbc1813b890 con 0x7fbc18139ad0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbc080038c0 con 0x7fbc18139ad0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbc08003c10 con 0x7fbc18139ad0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:28.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc0f7fe640 10 monclient: dump: 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.566+0000 7fbc157fa640 4 set_mon_vals no callback set 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbc08003e30 con 0x7fbc18139ad0 2024-09-16T04:09:28.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc1d5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc1d5e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc1d5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1813bc10 con 0x7fbc18139ad0 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(95..95 src has 1..95) ==== 5310+0+0 (unknown 3936019572 0 0) 0x7fbc08047440 con 0x7fbc18139ad0 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.567+0000 7fbc16ffd640 10 monclient: get_auth_request con 0x7fbbf8038ea0 auth_method 0 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.571+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbc08015d90 con 0x7fbc18139ad0 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.571+0000 7fbc0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:28.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.571+0000 7fbc0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc1d5e8640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc1d5e8640 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc1d5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) -- 0x7fbc1806bcf0 con 0x7fbc18139ad0 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v95) ==== 65+0+226 (unknown 3933836535 0 2157906930) 0x7fbc08015690 con 0x7fbc18139ad0 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 7}] 2024-09-16T04:09:28.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.741+0000 7fbc0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:28.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc1d5e8640 10 start_mon_command cmd=[{"prefix": "osd find", "id": 7}] 2024-09-16T04:09:28.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc1d5e8640 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T04:09:28.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc1d5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd find", "id": 7} v 0) -- 0x7fbc18073930 con 0x7fbc18139ad0 2024-09-16T04:09:28.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc0f7fe640 1 -- 172.21.15.171:0/599078712 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd find", "id": 7}]=0 v95) ==== 65+0+226 (unknown 3933836535 0 2157906930) 0x7fbc18073930 con 0x7fbc18139ad0 2024-09-16T04:09:28.844 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc0f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 7}] 2024-09-16T04:09:28.845 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.841+0000 7fbc0f7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:28.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 >> v1:172.21.15.171:6812/7327273 conn(0x7fbbf8038ea0 legacy=0x7fbbf803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 10 monclient: shutdown 2024-09-16T04:09:28.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:28.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 >> v1:172.21.15.171:6789/0 conn(0x7fbc18139ad0 legacy=0x7fbc1813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 shutdown_connections 2024-09-16T04:09:28.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 >> 172.21.15.171:0/599078712 conn(0x7fbc180a9830 msgr2=0x7fbc180a1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:28.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 shutdown_connections 2024-09-16T04:09:28.847 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.843+0000 7fbc1d5e8640 1 -- 172.21.15.171:0/599078712 wait complete. 2024-09-16T04:09:28.855 INFO:tasks.workunit.client.0.smithi171.stdout: "host": "foo", 2024-09-16T04:09:28.855 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.7 2024-09-16T04:09:28.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.931+0000 7fdf96e5a640 10 monclient: get_monmap_and_config 2024-09-16T04:09:28.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.931+0000 7fdf96e5a640 10 monclient: build_initial_monmap 2024-09-16T04:09:28.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.931+0000 7fdf96e5a640 10 monclient: monmap: 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:28.933486+0000 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:28.933486+0000 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:28.935 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 1 Processor -- start 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 1 -- start start 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient: init 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient(hunting): picked mon.noname-a con 0x7fdf9009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf900a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient(hunting): picked mon.noname-c con 0x7fdf9009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:28.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf900a1260 con 0x7fdf9009f640 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9009d0c0 con 0x7fdf9009e1d0 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9009d3a0 con 0x7fdf900a0130 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.932+0000 7fdf96e5a640 10 monclient(hunting): authenticate will time out at 944.438924s 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf953f7640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdf9009e1d0 0x7fdf9013bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42080/0 (socket says 172.21.15.171:42080) 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf953f7640 1 -- 172.21.15.171:0/3070480678 learned_addr learned my addr 172.21.15.171:0/3070480678 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf94bf6640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdf9009f640 legacy=0x7fdf9009fa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123168032 0 0) 0x7fdf9009d0c0 con 0x7fdf9009e1d0 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient(hunting): my global_id is 5320 2024-09-16T04:09:28.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf80004630 con 0x7fdf9009e1d0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 425148924 0 0) 0x7fdf80004630 con 0x7fdf9009e1d0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 >> v1:172.21.15.148:6790/0 conn(0x7fdf900a0130 legacy=0x7fdf90138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 >> v1:172.21.15.148:6789/0 conn(0x7fdf9009f640 legacy=0x7fdf9009fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9009d680 con 0x7fdf9009e1d0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdf88001990 con 0x7fdf9009e1d0 2024-09-16T04:09:28.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf96e5a640 5 monclient: authenticate success, global_id 5320 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf96e5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: dump: 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.945 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf96e5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf88001c80 con 0x7fdf9009e1d0 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 1 -- 172.21.15.171:0/3070480678 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdf88002090 con 0x7fdf9009e1d0 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf8f7fe640 10 monclient: dump: 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.946 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf96e5a640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.933+0000 7fdf96e5a640 4 set_mon_vals no callback set 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 10 monclient: shutdown 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 1 -- 172.21.15.171:0/3070480678 >> v1:172.21.15.171:6789/0 conn(0x7fdf9009e1d0 legacy=0x7fdf9013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 1 -- 172.21.15.171:0/3070480678 shutdown_connections 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 1 -- 172.21.15.171:0/3070480678 >> 172.21.15.171:0/3070480678 conn(0x7fdf900a52a0 msgr2=0x7fdf900a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:28.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 1 -- 172.21.15.171:0/3070480678 shutdown_connections 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 1 -- 172.21.15.171:0/3070480678 wait complete. 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 10 monclient: build_initial_monmap 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.934+0000 7fdf96e5a640 10 monclient: monmap: 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:28.948 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 1 Processor -- start 2024-09-16T04:09:28.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 1 -- start start 2024-09-16T04:09:28.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient: init 2024-09-16T04:09:28.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:28.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient(hunting): picked mon.noname-a con 0x7fdf9009e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient(hunting): picked mon.noname-c con 0x7fdf9009f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf900a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf901c3760 con 0x7fdf9009e1d0 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf901a7ef0 con 0x7fdf900a0130 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf901aa9b0 con 0x7fdf9009f640 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf96e5a640 10 monclient(hunting): authenticate will time out at 944.441715s 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf94bf6640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdf9009e1d0 legacy=0x7fdf901c3050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf953f7640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdf900a0130 0x7fdf901d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42086/0 (socket says 172.21.15.171:42086) 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf953f7640 1 -- 172.21.15.171:0/1513067202 learned_addr learned my addr 172.21.15.171:0/1513067202 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541719541 0 0) 0x7fdf901a7ef0 con 0x7fdf900a0130 2024-09-16T04:09:28.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.935+0000 7fdf8d7fa640 10 monclient(hunting): my global_id is 5323 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf74003520 con 0x7fdf900a0130 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155052064 0 0) 0x7fdf901aa9b0 con 0x7fdf9009f640 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): my global_id is 5354 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf901a7ef0 con 0x7fdf9009f640 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2804724108 0 0) 0x7fdf74003520 con 0x7fdf900a0130 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf901aa9b0 con 0x7fdf900a0130 2024-09-16T04:09:28.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdf880017b0 con 0x7fdf900a0130 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.953 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 447339926 0 0) 0x7fdf901a7ef0 con 0x7fdf9009f640 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf74003520 con 0x7fdf9009f640 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdf7c001870 con 0x7fdf9009f640 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient(hunting): dump: 2024-09-16T04:09:28.954 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1333358106 0 0) 0x7fdf901aa9b0 con 0x7fdf900a0130 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 >> v1:172.21.15.148:6790/0 conn(0x7fdf9009f640 legacy=0x7fdf901ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 >> v1:172.21.15.148:6789/0 conn(0x7fdf9009e1d0 legacy=0x7fdf901c3050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:28.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 monclient: found mon.b 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf900a4280 con 0x7fdf900a0130 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 5 monclient: authenticate success, global_id 5323 2024-09-16T04:09:28.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 10 monclient: _renew_subs 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9006ceb0 con 0x7fdf900a0130 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 10 monclient: _renew_subs 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf901aa6d0 con 0x7fdf900a0130 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf88004fa0 con 0x7fdf900a0130 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdf88005150 con 0x7fdf900a0130 2024-09-16T04:09:28.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8effd640 4 set_mon_vals no callback set 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.936+0000 7fdf8d7fa640 10 monclient: dump: 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:28.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdf88010580 con 0x7fdf900a0130 2024-09-16T04:09:28.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf96e5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:28.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf96e5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:28.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf96e5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:28.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf70005080 con 0x7fdf900a0130 2024-09-16T04:09:28.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(95..95 src has 1..95) ==== 5310+0+0 (unknown 3936019572 0 0) 0x7fdf880477c0 con 0x7fdf900a0130 2024-09-16T04:09:28.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.937+0000 7fdf94bf6640 10 monclient: get_auth_request con 0x7fdf74039090 auth_method 0 2024-09-16T04:09:28.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.942+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdf88016070 con 0x7fdf900a0130 2024-09-16T04:09:28.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.942+0000 7fdf8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:28.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:28.942+0000 7fdf8d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:29.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.109+0000 7fdf96e5a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T04:09:29.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.109+0000 7fdf96e5a640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T04:09:29.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.109+0000 7fdf96e5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.109+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) -- 0x7fdf70005370 con 0x7fdf900a0130 2024-09-16T04:09:29.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.365+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 removed item id 7 name 'osd.7' from crush map v96) ==== 122+0+0 (unknown 1176158221 0 0) 0x7fdf88015970 con 0x7fdf900a0130 2024-09-16T04:09:29.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.365+0000 7fdf8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T04:09:29.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.365+0000 7fdf8d7fa640 10 monclient: _finish_command 2 = system:0 removed item id 7 name 'osd.7' from crush map 2024-09-16T04:09:29.476 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf96e5a640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T04:09:29.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf96e5a640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T04:09:29.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf96e5a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.7"} v 0) -- 0x7fdf70002210 con 0x7fdf900a0130 2024-09-16T04:09:29.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf8d7fa640 1 -- 172.21.15.171:0/1513067202 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.7"}]=0 device 'osd.7' does not appear in the crush map v96) ==== 124+0+0 (unknown 124837417 0 0) 0x7fdf88043410 con 0x7fdf900a0130 2024-09-16T04:09:29.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf8d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.7"}] 2024-09-16T04:09:29.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.473+0000 7fdf8d7fa640 10 monclient: _finish_command 3 = system:0 device 'osd.7' does not appear in the crush map 2024-09-16T04:09:29.478 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.7' does not appear in the crush map 2024-09-16T04:09:29.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 >> v1:172.21.15.171:6812/7327273 conn(0x7fdf74039090 legacy=0x7fdf7403b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 10 monclient: shutdown 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 >> v1:172.21.15.171:6789/0 conn(0x7fdf900a0130 legacy=0x7fdf901d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 shutdown_connections 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 >> 172.21.15.171:0/1513067202 conn(0x7fdf900a52a0 msgr2=0x7fdf90071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 shutdown_connections 2024-09-16T04:09:29.479 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.475+0000 7fdf96e5a640 1 -- 172.21.15.171:0/1513067202 wait complete. 2024-09-16T04:09:29.487 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.9 2024-09-16T04:09:29.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.563+0000 7f06bdaed640 10 monclient: get_monmap_and_config 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.563+0000 7f06bdaed640 10 monclient: build_initial_monmap 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.563+0000 7f06bdaed640 10 monclient: monmap: 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:29.565711+0000 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:29.565711+0000 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:29.567 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.564+0000 7f06bdaed640 1 Processor -- start 2024-09-16T04:09:29.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.564+0000 7f06bdaed640 1 -- start start 2024-09-16T04:09:29.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.564+0000 7f06bdaed640 10 monclient: init 2024-09-16T04:09:29.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:29.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient(hunting): picked mon.noname-a con 0x7f06b809ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient(hunting): picked mon.noname-c con 0x7f06b80a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient(hunting): picked mon.noname-b con 0x7f06b809e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b809fbf0 con 0x7f06b809ba30 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b80a0e10 con 0x7f06b80a0750 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b809f640 con 0x7f06b809e570 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient(hunting): authenticate will time out at 945.071175s 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b6ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f06b80a0750 0x7f06b81399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42096/0 (socket says 172.21.15.171:42096) 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b6ffd640 1 -- 172.21.15.171:0/149200430 learned_addr learned my addr 172.21.15.171:0/149200430 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b77fe640 1 -- 172.21.15.171:0/149200430 >> v1:172.21.15.148:6789/0 conn(0x7f06b809ba30 legacy=0x7f06b80a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087409714 0 0) 0x7f06b80a0e10 con 0x7f06b80a0750 2024-09-16T04:09:29.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient(hunting): my global_id is 5326 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a0003520 con 0x7f06b80a0750 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1993777842 0 0) 0x7f06a0003520 con 0x7f06b80a0750 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 >> v1:172.21.15.148:6790/0 conn(0x7f06b809e570 legacy=0x7f06b813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 >> v1:172.21.15.148:6789/0 conn(0x7f06b809ba30 legacy=0x7f06b80a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:29.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b809f810 con 0x7f06b80a0750 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f06ac001610 con 0x7f06b80a0750 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 5 monclient: authenticate success, global_id 5326 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: dump: 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.579 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06ac001970 con 0x7f06b80a0750 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:29.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 1 -- 172.21.15.171:0/149200430 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f06ac001d40 con 0x7f06b80a0750 2024-09-16T04:09:29.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:29.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06bdaed640 4 set_mon_vals no callback set 2024-09-16T04:09:29.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:29.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:29.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.565+0000 7f06b67fc640 10 monclient: dump: 2024-09-16T04:09:29.581 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.582 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.582 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.582 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.582 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.582 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.582 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 10 monclient: shutdown 2024-09-16T04:09:29.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 1 -- 172.21.15.171:0/149200430 >> v1:172.21.15.171:6789/0 conn(0x7f06b80a0750 legacy=0x7f06b81399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 1 -- 172.21.15.171:0/149200430 shutdown_connections 2024-09-16T04:09:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 1 -- 172.21.15.171:0/149200430 >> 172.21.15.171:0/149200430 conn(0x7f06b80a52a0 msgr2=0x7f06b80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 1 -- 172.21.15.171:0/149200430 shutdown_connections 2024-09-16T04:09:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 1 -- 172.21.15.171:0/149200430 wait complete. 2024-09-16T04:09:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 10 monclient: build_initial_monmap 2024-09-16T04:09:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.566+0000 7f06bdaed640 10 monclient: monmap: 2024-09-16T04:09:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:29.586 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.586 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:29.586 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:29.586 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:29.586 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 1 Processor -- start 2024-09-16T04:09:29.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 1 -- start start 2024-09-16T04:09:29.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient: init 2024-09-16T04:09:29.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:29.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:29.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient(hunting): picked mon.noname-a con 0x7f06b809ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:29.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient(hunting): picked mon.noname-c con 0x7f06b809e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:29.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient(hunting): picked mon.noname-b con 0x7f06b80a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:29.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b81313c0 con 0x7f06b809ba30 2024-09-16T04:09:29.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b812eb40 con 0x7f06b80a0750 2024-09-16T04:09:29.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b81306f0 con 0x7f06b809e570 2024-09-16T04:09:29.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:29.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06bdaed640 10 monclient(hunting): authenticate will time out at 945.073940s 2024-09-16T04:09:29.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.567+0000 7f06b77fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f06b809ba30 legacy=0x7f06b812dd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:29.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06b7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f06b80a0750 0x7f06b81d1ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42106/0 (socket says 172.21.15.171:42106) 2024-09-16T04:09:29.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06b7fff640 1 -- 172.21.15.171:0/962406767 learned_addr learned my addr 172.21.15.171:0/962406767 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:29.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967811112 0 0) 0x7f06b812eb40 con 0x7f06b80a0750 2024-09-16T04:09:29.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): my global_id is 5329 2024-09-16T04:09:29.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a4003520 con 0x7f06b80a0750 2024-09-16T04:09:29.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055868087 0 0) 0x7f06b81306f0 con 0x7f06b809e570 2024-09-16T04:09:29.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): my global_id is 5360 2024-09-16T04:09:29.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06b812eb40 con 0x7f06b809e570 2024-09-16T04:09:29.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3813653036 0 0) 0x7f06a4003520 con 0x7f06b80a0750 2024-09-16T04:09:29.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f06b81306f0 con 0x7f06b80a0750 2024-09-16T04:09:29.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f06b0001c40 con 0x7f06b80a0750 2024-09-16T04:09:29.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:29.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:29.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): dump: 2024-09-16T04:09:29.593 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.593 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.593 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.595 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1090140528 0 0) 0x7f06b81306f0 con 0x7f06b80a0750 2024-09-16T04:09:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 >> v1:172.21.15.148:6790/0 conn(0x7f06b809e570 legacy=0x7f06b812e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 >> v1:172.21.15.148:6789/0 conn(0x7f06b809ba30 legacy=0x7f06b812dd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 monclient: found mon.b 2024-09-16T04:09:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b812fe10 con 0x7f06b80a0750 2024-09-16T04:09:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: _finish_auth 0 2024-09-16T04:09:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 5 monclient: authenticate success, global_id 5329 2024-09-16T04:09:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 10 monclient: _renew_subs 2024-09-16T04:09:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f06b8132b60 con 0x7f06b80a0750 2024-09-16T04:09:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 10 monclient: _renew_subs 2024-09-16T04:09:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f06b8130250 con 0x7f06b80a0750 2024-09-16T04:09:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f06b0003990 con 0x7f06b80a0750 2024-09-16T04:09:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f06b0003ce0 con 0x7f06b80a0750 2024-09-16T04:09:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f06b5ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.568+0000 7f069ffff640 10 monclient: dump: 2024-09-16T04:09:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.601 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f06b0003e80 con 0x7f06b80a0750 2024-09-16T04:09:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f06bdaed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f06bdaed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f06bdaed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0698005080 con 0x7f06b80a0750 2024-09-16T04:09:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(96..96 src has 1..96) ==== 5260+0+0 (unknown 3638113495 0 0) 0x7f06b00476e0 con 0x7f06b80a0750 2024-09-16T04:09:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.569+0000 7f06b77fe640 10 monclient: get_auth_request con 0x7f06a4039000 auth_method 0 2024-09-16T04:09:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.574+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f06b0016000 con 0x7f06b80a0750 2024-09-16T04:09:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.574+0000 7f069ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.574+0000 7f069ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:29.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f06bdaed640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T04:09:29.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f06bdaed640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T04:09:29.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f06bdaed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) -- 0x7f0698005370 con 0x7f06b80a0750 2024-09-16T04:09:29.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v96) ==== 124+0+0 (unknown 1405139665 0 0) 0x7f06b0015900 con 0x7f06b80a0750 2024-09-16T04:09:29.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f069ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T04:09:29.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.746+0000 7f069ffff640 10 monclient: _finish_command 2 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T04:09:29.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f06bdaed640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T04:09:29.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f06bdaed640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T04:09:29.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f06bdaed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.855 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.9"} v 0) -- 0x7f0698002210 con 0x7f06b80a0750 2024-09-16T04:09:29.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f069ffff640 1 -- 172.21.15.171:0/962406767 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.9"}]=0 device 'osd.9' does not appear in the crush map v96) ==== 124+0+0 (unknown 1405139665 0 0) 0x7f06b001e4b0 con 0x7f06b80a0750 2024-09-16T04:09:29.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f069ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.9"}] 2024-09-16T04:09:29.856 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.852+0000 7f069ffff640 10 monclient: _finish_command 3 = system:0 device 'osd.9' does not appear in the crush map 2024-09-16T04:09:29.856 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.9' does not appear in the crush map 2024-09-16T04:09:29.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 >> v1:172.21.15.171:6812/7327273 conn(0x7f06a4039000 legacy=0x7f06a403b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 10 monclient: shutdown 2024-09-16T04:09:29.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:29.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 >> v1:172.21.15.171:6789/0 conn(0x7f06b80a0750 legacy=0x7f06b81d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 shutdown_connections 2024-09-16T04:09:29.857 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 >> 172.21.15.171:0/962406767 conn(0x7f06b80a52a0 msgr2=0x7f06b8132e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:29.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 shutdown_connections 2024-09-16T04:09:29.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.854+0000 7f06bdaed640 1 -- 172.21.15.171:0/962406767 wait complete. 2024-09-16T04:09:29.866 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm foo 2024-09-16T04:09:29.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.942+0000 7f7c5992c640 10 monclient: get_monmap_and_config 2024-09-16T04:09:29.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.942+0000 7f7c5992c640 10 monclient: build_initial_monmap 2024-09-16T04:09:29.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.942+0000 7f7c5992c640 10 monclient: monmap: 2024-09-16T04:09:29.945 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:29.945 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:29.944344+0000 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:29.944344+0000 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:29.946 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 1 Processor -- start 2024-09-16T04:09:29.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 1 -- start start 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient: init 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c54136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c54139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c5413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c540a1340 con 0x7f7c54136b60 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c54137650 con 0x7f7c54139b10 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c540a0aa0 con 0x7f7c5413d620 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c5992c640 10 monclient(hunting): authenticate will time out at 945.449829s 2024-09-16T04:09:29.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c4bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7c54139b10 0x7f7c5413bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42122/0 (socket says 172.21.15.171:42122) 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c52ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7c54136b60 legacy=0x7f7c54136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.943+0000 7f7c4bfff640 1 -- 172.21.15.171:0/2599929615 learned_addr learned my addr 172.21.15.171:0/2599929615 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425649721 0 0) 0x7f7c54137650 con 0x7f7c54139b10 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): my global_id is 5332 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c40003520 con 0x7f7c54139b10 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728243327 0 0) 0x7f7c540a0aa0 con 0x7f7c5413d620 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient(hunting): my global_id is 5363 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c54137650 con 0x7f7c5413d620 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 63001438 0 0) 0x7f7c40003520 con 0x7f7c54139b10 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 >> v1:172.21.15.148:6790/0 conn(0x7f7c5413d620 legacy=0x7f7c5413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 >> v1:172.21.15.148:6789/0 conn(0x7f7c54136b60 legacy=0x7f7c54136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c540a0550 con 0x7f7c54139b10 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:29.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7c3c001610 con 0x7f7c54139b10 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c5992c640 5 monclient: authenticate success, global_id 5332 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c5992c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: dump: 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.957 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c5992c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c3c001930 con 0x7f7c54139b10 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 1 -- 172.21.15.171:0/2599929615 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7c3c001d00 con 0x7f7c54139b10 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:29.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c527fc640 10 monclient: dump: 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c5992c640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:29.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.944+0000 7f7c5992c640 4 set_mon_vals no callback set 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 10 monclient: shutdown 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 1 -- 172.21.15.171:0/2599929615 >> v1:172.21.15.171:6789/0 conn(0x7f7c54139b10 legacy=0x7f7c5413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 1 -- 172.21.15.171:0/2599929615 shutdown_connections 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 1 -- 172.21.15.171:0/2599929615 >> 172.21.15.171:0/2599929615 conn(0x7f7c540a98b0 msgr2=0x7f7c540a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 1 -- 172.21.15.171:0/2599929615 shutdown_connections 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 1 -- 172.21.15.171:0/2599929615 wait complete. 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 10 monclient: build_initial_monmap 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 10 monclient: monmap: 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:29.960 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.945+0000 7f7c5992c640 1 Processor -- start 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 1 -- start start 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient: init 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:29.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c54136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c54139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c5413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c541207c0 con 0x7f7c54136b60 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c540a0240 con 0x7f7c54139b10 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c540a6f50 con 0x7f7c5413d620 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c5992c640 10 monclient(hunting): authenticate will time out at 945.452478s 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c52ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7c54136b60 legacy=0x7f7c541cba10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c4bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7c54139b10 0x7f7c541d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42132/0 (socket says 172.21.15.171:42132) 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c4bfff640 1 -- 172.21.15.171:0/2490367490 learned_addr learned my addr 172.21.15.171:0/2490367490 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835921585 0 0) 0x7f7c540a0240 con 0x7f7c54139b10 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): my global_id is 5335 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c34003520 con 0x7f7c54139b10 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561959889 0 0) 0x7f7c540a6f50 con 0x7f7c5413d620 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): my global_id is 5366 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c540a0240 con 0x7f7c5413d620 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2992710753 0 0) 0x7f7c34003520 con 0x7f7c54139b10 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c540a6f50 con 0x7f7c54139b10 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7c3c0015d0 con 0x7f7c54139b10 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.946+0000 7f7c50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:29.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:29.964 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2025279687 0 0) 0x7f7c540a0240 con 0x7f7c5413d620 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c34003520 con 0x7f7c5413d620 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7c4c001c30 con 0x7f7c5413d620 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.965 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2971052104 0 0) 0x7f7c540a6f50 con 0x7f7c54139b10 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 >> v1:172.21.15.148:6790/0 conn(0x7f7c5413d620 legacy=0x7f7c541d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 >> v1:172.21.15.148:6789/0 conn(0x7f7c54136b60 legacy=0x7f7c541cba10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 monclient: found mon.b 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c540a29d0 con 0x7f7c54139b10 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:29.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 5 monclient: authenticate success, global_id 5335 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 10 monclient: _renew_subs 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c54120aa0 con 0x7f7c54139b10 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 10 monclient: _renew_subs 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c54136640 con 0x7f7c54139b10 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c3c001930 con 0x7f7c54139b10 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7c3c004e60 con 0x7f7c54139b10 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c51ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.947+0000 7f7c50ff9640 10 monclient: dump: 2024-09-16T04:09:29.967 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7c3c010290 con 0x7f7c54139b10 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c5992c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:29.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c5992c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:29.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c5992c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:29.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c38005080 con 0x7f7c54139b10 2024-09-16T04:09:29.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(96..96 src has 1..96) ==== 5260+0+0 (unknown 3638113495 0 0) 0x7f7c3c047490 con 0x7f7c54139b10 2024-09-16T04:09:29.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.948+0000 7f7c52ffd640 10 monclient: get_auth_request con 0x7f7c34038f10 auth_method 0 2024-09-16T04:09:29.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.952+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7c3c015dd0 con 0x7f7c54139b10 2024-09-16T04:09:29.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.952+0000 7f7c50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:29.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:29.952+0000 7f7c50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:30.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.122+0000 7f7c5992c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:30.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.122+0000 7f7c5992c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:30.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.122+0000 7f7c5992c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.122+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f7c38005370 con 0x7f7c54139b10 2024-09-16T04:09:30.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.367+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v97) ==== 120+0+0 (unknown 3891411363 0 0) 0x7f7c3c01e220 con 0x7f7c54139b10 2024-09-16T04:09:30.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.367+0000 7f7c50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:30.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.367+0000 7f7c50ff9640 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-16T04:09:30.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c5992c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:30.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c5992c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:30.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c5992c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) -- 0x7f7c38002210 con 0x7f7c54139b10 2024-09-16T04:09:30.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c50ff9640 1 -- 172.21.15.171:0/2490367490 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v97) ==== 120+0+0 (unknown 3337531784 0 0) 0x7f7c3c001160 con 0x7f7c54139b10 2024-09-16T04:09:30.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c50ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-16T04:09:30.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.477+0000 7f7c50ff9640 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-16T04:09:30.481 INFO:tasks.workunit.client.0.smithi171.stderr:device 'foo' does not appear in the crush map 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 >> v1:172.21.15.171:6812/7327273 conn(0x7f7c34038f10 legacy=0x7f7c3403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 10 monclient: shutdown 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 >> v1:172.21.15.171:6789/0 conn(0x7f7c54139b10 legacy=0x7f7c541d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 shutdown_connections 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 >> 172.21.15.171:0/2490367490 conn(0x7f7c540a98b0 msgr2=0x7f7c54139070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.479+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 shutdown_connections 2024-09-16T04:09:30.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.480+0000 7f7c5992c640 1 -- 172.21.15.171:0/2490367490 wait complete. 2024-09-16T04:09:30.492 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.566+0000 7fed755e8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.566+0000 7fed755e8640 10 monclient: build_initial_monmap 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.567+0000 7fed755e8640 10 monclient: monmap: 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:30.568788+0000 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:30.568788+0000 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:30.570 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:30.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:30.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.567+0000 7fed755e8640 1 Processor -- start 2024-09-16T04:09:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.567+0000 7fed755e8640 1 -- start start 2024-09-16T04:09:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.567+0000 7fed755e8640 10 monclient: init 2024-09-16T04:09:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:30.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient(hunting): picked mon.noname-b con 0x7fed700729d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient(hunting): picked mon.noname-a con 0x7fed70070250 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient(hunting): picked mon.noname-c con 0x7fed7013b000 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed700a0310 con 0x7fed70070250 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed700a0030 con 0x7fed7013b000 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed7009f000 con 0x7fed700729d0 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient(hunting): authenticate will time out at 946.074222s 2024-09-16T04:09:30.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6e7fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fed70070250 legacy=0x7fed701398e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fed7013b000 0x7fed7013d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42142/0 (socket says 172.21.15.171:42142) 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6f7fe640 1 -- 172.21.15.171:0/2770507988 learned_addr learned my addr 172.21.15.171:0/2770507988 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282565347 0 0) 0x7fed700a0030 con 0x7fed7013b000 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): my global_id is 5338 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed58003520 con 0x7fed7013b000 2024-09-16T04:09:30.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 426843180 0 0) 0x7fed7009f000 con 0x7fed700729d0 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient(hunting): my global_id is 5369 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed700a0030 con 0x7fed700729d0 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1999013468 0 0) 0x7fed58003520 con 0x7fed7013b000 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 >> v1:172.21.15.148:6790/0 conn(0x7fed700729d0 legacy=0x7fed70072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 >> v1:172.21.15.148:6789/0 conn(0x7fed70070250 legacy=0x7fed701398e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 monclient: found mon.noname-c 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:30.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed700a42f0 con 0x7fed7013b000 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 5 monclient: authenticate success, global_id 5338 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed68001990 con 0x7fed7013b000 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: dump: 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.582 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed68001cb0 con 0x7fed7013b000 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 1 -- 172.21.15.171:0/2770507988 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed68002080 con 0x7fed7013b000 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:30.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed6dffb640 10 monclient: dump: 2024-09-16T04:09:30.584 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.568+0000 7fed755e8640 4 set_mon_vals no callback set 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 10 monclient: shutdown 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:30.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 1 -- 172.21.15.171:0/2770507988 >> v1:172.21.15.171:6789/0 conn(0x7fed7013b000 legacy=0x7fed7013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 1 -- 172.21.15.171:0/2770507988 shutdown_connections 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 1 -- 172.21.15.171:0/2770507988 >> 172.21.15.171:0/2770507988 conn(0x7fed700a7470 msgr2=0x7fed700a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 1 -- 172.21.15.171:0/2770507988 shutdown_connections 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.569+0000 7fed755e8640 1 -- 172.21.15.171:0/2770507988 wait complete. 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient: build_initial_monmap 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient: monmap: 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.586 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 1 Processor -- start 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 1 -- start start 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient: init 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient(hunting): picked mon.noname-a con 0x7fed70070250 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.570+0000 7fed755e8640 10 monclient(hunting): picked mon.noname-b con 0x7fed700729d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient(hunting): picked mon.noname-c con 0x7fed7013b000 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed70124f70 con 0x7fed70070250 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed70123560 con 0x7fed700729d0 2024-09-16T04:09:30.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed700a58b0 con 0x7fed7013b000 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient(hunting): authenticate will time out at 946.077057s 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed6effd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fed70070250 legacy=0x7fed701c9640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed6e7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fed700729d0 0x7fed701ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42156/0 (socket says 172.21.15.171:42156) 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed6e7fc640 1 -- 172.21.15.171:0/3494849249 learned_addr learned my addr 172.21.15.171:0/3494849249 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386585717 0 0) 0x7fed70123560 con 0x7fed700729d0 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): my global_id is 5341 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed5c003520 con 0x7fed700729d0 2024-09-16T04:09:30.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 226375889 0 0) 0x7fed700a58b0 con 0x7fed7013b000 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): my global_id is 5372 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed70123560 con 0x7fed7013b000 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 847734318 0 0) 0x7fed5c003520 con 0x7fed700729d0 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed700a58b0 con 0x7fed700729d0 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed640041d0 con 0x7fed700729d0 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): dump: 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:30.589 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:30.590 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:30.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:30.590 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3304675613 0 0) 0x7fed70123560 con 0x7fed7013b000 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed5c003520 con 0x7fed7013b000 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed680017b0 con 0x7fed7013b000 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient(hunting): dump: 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.591 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3291872744 0 0) 0x7fed700a58b0 con 0x7fed700729d0 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 >> v1:172.21.15.148:6790/0 conn(0x7fed7013b000 legacy=0x7fed701d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 >> v1:172.21.15.148:6789/0 conn(0x7fed70070250 legacy=0x7fed701c9640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 monclient: found mon.b 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:30.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7009e0d0 con 0x7fed700729d0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 5 monclient: authenticate success, global_id 5341 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient: _renew_subs 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed70122c80 con 0x7fed700729d0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient: _renew_subs 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fed7013a8a0 con 0x7fed700729d0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed64004480 con 0x7fed700729d0 2024-09-16T04:09:30.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed64004850 con 0x7fed700729d0 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed6d7fa640 4 set_mon_vals no callback set 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.571+0000 7fed577fe640 10 monclient: dump: 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:30.594 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fed64004ab0 con 0x7fed700729d0 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(97..97 src has 1..97) ==== 4962+0+0 (unknown 602358094 0 0) 0x7fed64047290 con 0x7fed700729d0 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed6effd640 10 monclient: get_auth_request con 0x7fed5c038e90 auth_method 0 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed755e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed755e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed755e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.572+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed50005080 con 0x7fed700729d0 2024-09-16T04:09:30.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.577+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fed64015d70 con 0x7fed700729d0 2024-09-16T04:09:30.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.577+0000 7fed577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:30.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.577+0000 7fed577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:30.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.758+0000 7fed755e8640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:09:30.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.758+0000 7fed755e8640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T04:09:30.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.758+0000 7fed755e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:30.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:30.758+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7fed50005370 con 0x7fed700729d0 2024-09-16T04:09:31.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.369+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v98) ==== 58+0+3 (unknown 1761221957 0 520826459) 0x7fed64015670 con 0x7fed700729d0 2024-09-16T04:09:31.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.369+0000 7fed577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T04:09:31.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.369+0000 7fed577fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:31.485 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.483+0000 7fed755e8640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:09:31.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.483+0000 7fed755e8640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T04:09:31.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.483+0000 7fed755e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:31.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:31.483+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7fed50000bf0 con 0x7fed700729d0 2024-09-16T04:09:32.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.388+0000 7fed577fe640 1 -- 172.21.15.171:0/3494849249 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v99) ==== 58+0+3 (unknown 3717186481 0 212448812) 0x7fed6401e220 con 0x7fed700729d0 2024-09-16T04:09:32.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.388+0000 7fed577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T04:09:32.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.388+0000 7fed577fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:32.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 >> v1:172.21.15.171:6812/7327273 conn(0x7fed5c038e90 legacy=0x7fed5c03b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:32.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 10 monclient: shutdown 2024-09-16T04:09:32.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:32.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 >> v1:172.21.15.171:6789/0 conn(0x7fed700729d0 legacy=0x7fed701ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:32.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 shutdown_connections 2024-09-16T04:09:32.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 >> 172.21.15.171:0/3494849249 conn(0x7fed700a7470 msgr2=0x7fed700a2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:32.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 shutdown_connections 2024-09-16T04:09:32.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.391+0000 7fed755e8640 1 -- 172.21.15.171:0/3494849249 wait complete. 2024-09-16T04:09:32.406 INFO:tasks.workunit.client.0.smithi171.stderr:+ o3=11 2024-09-16T04:09:32.406 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 11 123 root=default 2024-09-16T04:09:32.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.489+0000 7fdff268d640 10 monclient: get_monmap_and_config 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.489+0000 7fdff268d640 10 monclient: build_initial_monmap 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.489+0000 7fdff268d640 10 monclient: monmap: 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:32.490887+0000 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:32.490887+0000 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:32.492 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:32.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 1 Processor -- start 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 1 -- start start 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient: init 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfec136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfec137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfec1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec09b2c0 con 0x7fdfec137140 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec09b5a0 con 0x7fdfec136d60 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec0a6640 con 0x7fdfec1375a0 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:32.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdff268d640 10 monclient(hunting): authenticate will time out at 947.996633s 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfebfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdfec136d60 0x7fdfec09bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42166/0 (socket says 172.21.15.171:42166) 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfebfff640 1 -- 172.21.15.171:0/928998429 learned_addr learned my addr 172.21.15.171:0/928998429 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfe3fff640 1 -- 172.21.15.171:0/928998429 >> v1:172.21.15.148:6789/0 conn(0x7fdfec137140 legacy=0x7fdfec13bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961868927 0 0) 0x7fdfec09b5a0 con 0x7fdfec136d60 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfeb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.490+0000 7fdfeb7fe640 10 monclient(hunting): my global_id is 5344 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfd8003520 con 0x7fdfec136d60 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2368970335 0 0) 0x7fdfd8003520 con 0x7fdfec136d60 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 >> v1:172.21.15.148:6790/0 conn(0x7fdfec1375a0 legacy=0x7fdfec13f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 >> v1:172.21.15.148:6789/0 conn(0x7fdfec137140 legacy=0x7fdfec13bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:32.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfec143b20 con 0x7fdfec136d60 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdfdc001610 con 0x7fdfec136d60 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 5 monclient: authenticate success, global_id 5344 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:32.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: dump: 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:32.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdfdc001900 con 0x7fdfec136d60 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 1 -- 172.21.15.171:0/928998429 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdfdc001cd0 con 0x7fdfec136d60 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 4 set_mon_vals no callback set 2024-09-16T04:09:32.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdfeb7fe640 10 monclient: dump: 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 10 monclient: shutdown 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:32.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 1 -- 172.21.15.171:0/928998429 >> v1:172.21.15.171:6789/0 conn(0x7fdfec136d60 legacy=0x7fdfec09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 1 -- 172.21.15.171:0/928998429 shutdown_connections 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 1 -- 172.21.15.171:0/928998429 >> 172.21.15.171:0/928998429 conn(0x7fdfec0a9ae0 msgr2=0x7fdfec0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 1 -- 172.21.15.171:0/928998429 shutdown_connections 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.491+0000 7fdff268d640 1 -- 172.21.15.171:0/928998429 wait complete. 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.492+0000 7fdff268d640 10 monclient: build_initial_monmap 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.492+0000 7fdff268d640 10 monclient: monmap: 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:32.506 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.492+0000 7fdff268d640 1 Processor -- start 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.492+0000 7fdff268d640 1 -- start start 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.492+0000 7fdff268d640 10 monclient: init 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:32.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient(hunting): picked mon.noname-c con 0x7fdfec136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient(hunting): picked mon.noname-b con 0x7fdfec137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient(hunting): picked mon.noname-a con 0x7fdfec1375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec13a310 con 0x7fdfec1375a0 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec13a8d0 con 0x7fdfec137140 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfec13a030 con 0x7fdfec136d60 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient(hunting): authenticate will time out at 947.999288s 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdfec137140 0x7fdfec138180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42176/0 (socket says 172.21.15.171:42176) 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe3fff640 1 -- 172.21.15.171:0/4287255115 learned_addr learned my addr 172.21.15.171:0/4287255115 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff0c2a640 1 -- 172.21.15.171:0/4287255115 >> v1:172.21.15.148:6789/0 conn(0x7fdfec1375a0 legacy=0x7fdfec1d3a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578025603 0 0) 0x7fdfec13a8d0 con 0x7fdfec137140 2024-09-16T04:09:32.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): my global_id is 5347 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc003520 con 0x7fdfec137140 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1946367522 0 0) 0x7fdfcc003520 con 0x7fdfec137140 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfec13a8d0 con 0x7fdfec137140 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdfd4004180 con 0x7fdfec137140 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): dump: 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:32.509 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218195696 0 0) 0x7fdfec13a030 con 0x7fdfec136d60 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:32.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient(hunting): my global_id is 5378 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc003520 con 0x7fdfec136d60 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1117511513 0 0) 0x7fdfec13a8d0 con 0x7fdfec137140 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 >> v1:172.21.15.148:6790/0 conn(0x7fdfec136d60 legacy=0x7fdfec137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 >> v1:172.21.15.148:6789/0 conn(0x7fdfec1375a0 legacy=0x7fdfec1d3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 monclient: found mon.b 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfec13a5f0 con 0x7fdfec137140 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdfe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:32.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 5 monclient: authenticate success, global_id 5347 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient: _renew_subs 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.493+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdfec0a2990 con 0x7fdfec137140 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 10 monclient: _renew_subs 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdfec138a70 con 0x7fdfec137140 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdfd4004470 con 0x7fdfec137140 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdfd4004840 con 0x7fdfec137140 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 10 monclient: dump: 2024-09-16T04:09:32.512 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfeaffd640 4 set_mon_vals no callback set 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdfd4004aa0 con 0x7fdfec137140 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:32.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:32.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.494+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd0005080 con 0x7fdfec137140 2024-09-16T04:09:32.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.495+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(99..99 src has 1..99) ==== 5192+0+0 (unknown 1547194125 0 0) 0x7fdfd40474e0 con 0x7fdfec137140 2024-09-16T04:09:32.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.495+0000 7fdfebfff640 10 monclient: get_auth_request con 0x7fdfcc038f70 auth_method 0 2024-09-16T04:09:32.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.499+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdfd4015d80 con 0x7fdfec137140 2024-09-16T04:09:32.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.499+0000 7fdfe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:32.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.499+0000 7fdfe97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:32.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.668+0000 7fdff268d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T04:09:32.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.668+0000 7fdff268d640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T04:09:32.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.668+0000 7fdff268d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:32.671 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:32.668+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) -- 0x7fdfd0005370 con 0x7fdfec137140 2024-09-16T04:09:33.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.390+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map v100) ==== 193+0+0 (unknown 2674872195 0 0) 0x7fdfd4015680 con 0x7fdfec137140 2024-09-16T04:09:33.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.390+0000 7fdfe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T04:09:33.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.390+0000 7fdfe97fa640 10 monclient: _finish_command 2 = system:0 add item id 11 name 'osd.11' weight 123 at location {root=default} to crush map 2024-09-16T04:09:33.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdff268d640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdff268d640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdff268d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]} v 0) -- 0x7fdfd0005bd0 con 0x7fdfec137140 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdfe97fa640 1 -- 172.21.15.171:0/4287255115 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}]=0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change v100) ==== 191+0+0 (unknown 199084981 0 0) 0x7fdfd40199d0 con 0x7fdfec137140 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdfe97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 11, "weight": 123.0, "args": ["root=default"]}] 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.496+0000 7fdfe97fa640 10 monclient: _finish_command 3 = system:0 set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T04:09:33.500 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 11 name 'osd.11' weight 123 at location {root=default}: no change 2024-09-16T04:09:33.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.498+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 >> v1:172.21.15.171:6812/7327273 conn(0x7fdfcc038f70 legacy=0x7fdfcc03b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.498+0000 7fdff268d640 10 monclient: shutdown 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.498+0000 7fdff268d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.498+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 >> v1:172.21.15.171:6789/0 conn(0x7fdfec137140 legacy=0x7fdfec138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.498+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 shutdown_connections 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.498+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 >> 172.21.15.171:0/4287255115 conn(0x7fdfec0a9ae0 msgr2=0x7fdfec072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.499+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 shutdown_connections 2024-09-16T04:09:33.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.499+0000 7fdff268d640 1 -- 172.21.15.171:0/4287255115 wait complete. 2024-09-16T04:09:33.511 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:33.511 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep osd.11 2024-09-16T04:09:33.511 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 123 2024-09-16T04:09:33.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.586+0000 7f42b125e640 10 monclient: get_monmap_and_config 2024-09-16T04:09:33.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.586+0000 7f42b125e640 10 monclient: build_initial_monmap 2024-09-16T04:09:33.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.586+0000 7f42b125e640 10 monclient: monmap: 2024-09-16T04:09:33.589 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:33.589 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:33.588113+0000 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:33.588113+0000 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.586+0000 7f42b125e640 1 Processor -- start 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.586+0000 7f42b125e640 1 -- start start 2024-09-16T04:09:33.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.586+0000 7f42b125e640 10 monclient: init 2024-09-16T04:09:33.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:33.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:33.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient(hunting): picked mon.noname-c con 0x7f42ac130930 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:33.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient(hunting): picked mon.noname-a con 0x7f42ac09eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:33.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient(hunting): picked mon.noname-b con 0x7f42ac130d10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:33.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac1343a0 con 0x7f42ac09eb60 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac09c1b0 con 0x7f42ac130930 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac09d8e0 con 0x7f42ac130d10 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42b125e640 10 monclient(hunting): authenticate will time out at 949.093474s 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42aad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42ac130930 0x7f42ac133580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42192/0 (socket says 172.21.15.171:42192) 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42aad9d640 1 -- 172.21.15.171:0/3093644807 learned_addr learned my addr 172.21.15.171:0/3093644807 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42aa59c640 1 -- 172.21.15.171:0/3093644807 >> v1:172.21.15.148:6789/0 conn(0x7f42ac09eb60 legacy=0x7f42ac133c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884089519 0 0) 0x7f42ac09c1b0 con 0x7f42ac130930 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): my global_id is 5350 2024-09-16T04:09:33.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4294003520 con 0x7f42ac130930 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574375170 0 0) 0x7f42ac09d8e0 con 0x7f42ac130d10 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 10 monclient(hunting): my global_id is 5381 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac09c1b0 con 0x7f42ac130d10 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.587+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2578827176 0 0) 0x7f4294003520 con 0x7f42ac130930 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 >> v1:172.21.15.148:6790/0 conn(0x7f42ac130d10 legacy=0x7f42ac13c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 >> v1:172.21.15.148:6789/0 conn(0x7f42ac09eb60 legacy=0x7f42ac133c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 monclient: found mon.noname-c 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:33.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac09dbc0 con 0x7f42ac130930 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: _finish_auth 0 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42a0001610 con 0x7f42ac130930 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 5 monclient: authenticate success, global_id 5350 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: dump: 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.600 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a0001870 con 0x7f42ac130930 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:33.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 1 -- 172.21.15.171:0/3093644807 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42a0001bc0 con 0x7f42ac130930 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 4 set_mon_vals no callback set 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42a9d9b640 10 monclient: dump: 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.602 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 10 monclient: shutdown 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 1 -- 172.21.15.171:0/3093644807 >> v1:172.21.15.171:6789/0 conn(0x7f42ac130930 legacy=0x7f42ac133580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 1 -- 172.21.15.171:0/3093644807 shutdown_connections 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 1 -- 172.21.15.171:0/3093644807 >> 172.21.15.171:0/3093644807 conn(0x7f42ac0a52a0 msgr2=0x7f42ac0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 1 -- 172.21.15.171:0/3093644807 shutdown_connections 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.588+0000 7f42b125e640 1 -- 172.21.15.171:0/3093644807 wait complete. 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.589+0000 7f42b125e640 10 monclient: build_initial_monmap 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.589+0000 7f42b125e640 10 monclient: monmap: 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:33.603 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.589+0000 7f42b125e640 1 Processor -- start 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.589+0000 7f42b125e640 1 -- start start 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.589+0000 7f42b125e640 10 monclient: init 2024-09-16T04:09:33.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient(hunting): picked mon.noname-a con 0x7f42ac09eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient(hunting): picked mon.noname-c con 0x7f42ac130930 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient(hunting): picked mon.noname-b con 0x7f42ac130d10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac1219a0 con 0x7f42ac09eb60 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac120fd0 con 0x7f42ac130d10 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac120110 con 0x7f42ac130930 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42b125e640 10 monclient(hunting): authenticate will time out at 949.096417s 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42aad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f42ac09eb60 legacy=0x7f42ac12f550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42ab59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f42ac130d10 0x7f42ac1d1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42202/0 (socket says 172.21.15.171:42202) 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42ab59e640 1 -- 172.21.15.171:0/130011306 learned_addr learned my addr 172.21.15.171:0/130011306 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648501343 0 0) 0x7f42ac120fd0 con 0x7f42ac130d10 2024-09-16T04:09:33.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): my global_id is 5353 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4298003550 con 0x7f42ac130d10 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465429542 0 0) 0x7f42ac120110 con 0x7f42ac130930 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): my global_id is 5384 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac120fd0 con 0x7f42ac130930 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 512731940 0 0) 0x7f4298003550 con 0x7f42ac130d10 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42ac120110 con 0x7f42ac130d10 2024-09-16T04:09:33.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42a4001d00 con 0x7f42ac130d10 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): dump: 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.590+0000 7f42937fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1358360064 0 0) 0x7f42ac120fd0 con 0x7f42ac130930 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4298003550 con 0x7f42ac130930 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f429c0040f0 con 0x7f42ac130930 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient(hunting): dump: 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.608 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1275882974 0 0) 0x7f42ac120110 con 0x7f42ac130d10 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 >> v1:172.21.15.148:6790/0 conn(0x7f42ac130930 legacy=0x7f42ac1ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 >> v1:172.21.15.148:6789/0 conn(0x7f42ac09eb60 legacy=0x7f42ac12f550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 monclient: found mon.b 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac121c80 con 0x7f42ac130d10 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:33.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 5 monclient: authenticate success, global_id 5353 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 10 monclient: _renew_subs 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42ac133ba0 con 0x7f42ac130d10 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 10 monclient: _renew_subs 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42ac1216c0 con 0x7f42ac130d10 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42a4002020 con 0x7f42ac130d10 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42a4003de0 con 0x7f42ac130d10 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:33.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42a959a640 4 set_mon_vals no callback set 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.591+0000 7f42937fe640 10 monclient: dump: 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f42a4003f80 con 0x7f42ac130d10 2024-09-16T04:09:33.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42b125e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42b125e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42b125e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c005080 con 0x7f42ac130d10 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(100..100 src has 1..100) ==== 5214+0+0 (unknown 1514978075 0 0) 0x7f42a40476a0 con 0x7f42ac130d10 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.592+0000 7f42aad9d640 10 monclient: get_auth_request con 0x7f4298038f70 auth_method 0 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.597+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f42a4015f70 con 0x7f42ac130d10 2024-09-16T04:09:33.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.597+0000 7f42937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:33.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.597+0000 7f42937fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:33.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42b125e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:33.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42b125e640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:33.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42b125e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f428c005b80 con 0x7f42ac130d10 2024-09-16T04:09:33.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v100) ==== 56+0+1173 (unknown 1723643170 0 281299908) 0x7f42a4015870 con 0x7f42ac130d10 2024-09-16T04:09:33.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:33.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.762+0000 7f42937fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.862+0000 7f42b125e640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.863+0000 7f42b125e640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.863+0000 7f42b125e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.863+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f428c005d50 con 0x7f42ac130d10 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.863+0000 7f42937fe640 1 -- 172.21.15.171:0/130011306 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v100) ==== 56+0+1173 (unknown 1723643170 0 281299908) 0x7f42a4019bc0 con 0x7f42ac130d10 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.863+0000 7f42937fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:33.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.863+0000 7f42937fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:33.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 >> v1:172.21.15.171:6812/7327273 conn(0x7f4298038f70 legacy=0x7f429803b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 10 monclient: shutdown 2024-09-16T04:09:33.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:33.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 >> v1:172.21.15.171:6789/0 conn(0x7f42ac130d10 legacy=0x7f42ac1d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 shutdown_connections 2024-09-16T04:09:33.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 >> 172.21.15.171:0/130011306 conn(0x7f42ac0a52a0 msgr2=0x7f42ac09f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:33.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 shutdown_connections 2024-09-16T04:09:33.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.865+0000 7f42b125e640 1 -- 172.21.15.171:0/130011306 wait complete. 2024-09-16T04:09:33.877 INFO:tasks.workunit.client.0.smithi171.stdout: 11 123.00000 osd.11 down 1.00000 1.00000 2024-09-16T04:09:33.877 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush reweight osd.11 113 2024-09-16T04:09:33.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.955+0000 7f63b5c53640 10 monclient: get_monmap_and_config 2024-09-16T04:09:33.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.955+0000 7f63b5c53640 10 monclient: build_initial_monmap 2024-09-16T04:09:33.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.955+0000 7f63b5c53640 10 monclient: monmap: 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:33.957582+0000 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:33.957582+0000 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:33.959 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.956+0000 7f63b5c53640 1 Processor -- start 2024-09-16T04:09:33.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.956+0000 7f63b5c53640 1 -- start start 2024-09-16T04:09:33.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.956+0000 7f63b5c53640 10 monclient: init 2024-09-16T04:09:33.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:33.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:33.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient(hunting): picked mon.noname-b con 0x7f63b009e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient(hunting): picked mon.noname-c con 0x7f63b0136530 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient(hunting): picked mon.noname-a con 0x7f63b013a040 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b00a1260 con 0x7f63b013a040 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b009d0c0 con 0x7f63b0136530 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b009d3a0 con 0x7f63b009e1d0 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient(hunting): authenticate will time out at 949.463247s 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63affff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f63b013a040 legacy=0x7f63b013c430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63a7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f63b0136530 0x7f63b0138920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42218/0 (socket says 172.21.15.171:42218) 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63a7fff640 1 -- 172.21.15.171:0/10504047 learned_addr learned my addr 172.21.15.171:0/10504047 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3436187549 0 0) 0x7f63b009d0c0 con 0x7f63b0136530 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): my global_id is 5356 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f639c003520 con 0x7f63b0136530 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575195720 0 0) 0x7f63b009d3a0 con 0x7f63b009e1d0 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient(hunting): my global_id is 5387 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63b009d0c0 con 0x7f63b009e1d0 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1625065052 0 0) 0x7f639c003520 con 0x7f63b0136530 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 >> v1:172.21.15.148:6790/0 conn(0x7f63b009e1d0 legacy=0x7f63b00a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 >> v1:172.21.15.148:6789/0 conn(0x7f63b013a040 legacy=0x7f63b013c430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:33.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b009d680 con 0x7f63b0136530 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6398001610 con 0x7f63b0136530 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 5 monclient: authenticate success, global_id 5356 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: dump: 2024-09-16T04:09:33.969 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6398001870 con 0x7f63b0136530 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:33.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 1 -- 172.21.15.171:0/10504047 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6398001bc0 con 0x7f63b0136530 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63b5c53640 4 set_mon_vals no callback set 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.957+0000 7f63aeffd640 10 monclient: dump: 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.971 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 10 monclient: shutdown 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 1 -- 172.21.15.171:0/10504047 >> v1:172.21.15.171:6789/0 conn(0x7f63b0136530 legacy=0x7f63b0138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 1 -- 172.21.15.171:0/10504047 shutdown_connections 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 1 -- 172.21.15.171:0/10504047 >> 172.21.15.171:0/10504047 conn(0x7f63b00a52a0 msgr2=0x7f63b00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 1 -- 172.21.15.171:0/10504047 shutdown_connections 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.958+0000 7f63b5c53640 1 -- 172.21.15.171:0/10504047 wait complete. 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient: build_initial_monmap 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient: monmap: 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:33.972 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 1 Processor -- start 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 1 -- start start 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient: init 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient(hunting): picked mon.noname-b con 0x7f63b009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:33.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient(hunting): picked mon.noname-c con 0x7f63b0136530 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient(hunting): picked mon.noname-a con 0x7f63b013a040 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b00a4060 con 0x7f63b013a040 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b01205b0 con 0x7f63b009e1d0 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63b0121a60 con 0x7f63b0136530 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63b5c53640 10 monclient(hunting): authenticate will time out at 949.465911s 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63af7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f63b009e1d0 0x7f63b0134f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42232/0 (socket says 172.21.15.171:42232) 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63affff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f63b013a040 legacy=0x7f63b01d1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.959+0000 7f63af7fe640 1 -- 172.21.15.171:0/659132416 learned_addr learned my addr 172.21.15.171:0/659132416 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618436079 0 0) 0x7f63b01205b0 con 0x7f63b009e1d0 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): my global_id is 5359 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6390003520 con 0x7f63b009e1d0 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2348398518 0 0) 0x7f63b0121a60 con 0x7f63b0136530 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): my global_id is 5390 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63b01205b0 con 0x7f63b0136530 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1311347919 0 0) 0x7f6390003520 con 0x7f63b009e1d0 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63b0121a60 con 0x7f63b009e1d0 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f63a0001a00 con 0x7f63b009e1d0 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): dump: 2024-09-16T04:09:33.975 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2315434210 0 0) 0x7f63b01205b0 con 0x7f63b0136530 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6390003520 con 0x7f63b0136530 2024-09-16T04:09:33.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6398001510 con 0x7f63b0136530 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient(hunting): dump: 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.977 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2490281174 0 0) 0x7f63b0121a60 con 0x7f63b009e1d0 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 >> v1:172.21.15.148:6790/0 conn(0x7f63b0136530 legacy=0x7f63b01ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 >> v1:172.21.15.148:6789/0 conn(0x7f63b013a040 legacy=0x7f63b01d1d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 monclient: found mon.b 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b0121780 con 0x7f63b009e1d0 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 5 monclient: authenticate success, global_id 5359 2024-09-16T04:09:33.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 10 monclient: _renew_subs 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63b0120330 con 0x7f63b009e1d0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 10 monclient: _renew_subs 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63b011eea0 con 0x7f63b009e1d0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f63a00038c0 con 0x7f63b009e1d0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f63a0003c90 con 0x7f63b009e1d0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63acff9640 10 monclient: dump: 2024-09-16T04:09:33.979 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.960+0000 7f63ae7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f63a0003f30 con 0x7f63b009e1d0 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63b5c53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:33.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63b5c53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63b5c53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b0071760 con 0x7f63b009e1d0 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(100..100 src has 1..100) ==== 5214+0+0 (unknown 1514978075 0 0) 0x7f63a0047460 con 0x7f63b009e1d0 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.961+0000 7f63a7fff640 10 monclient: get_auth_request con 0x7f6390038ea0 auth_method 0 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.966+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f63a0015db0 con 0x7f63b009e1d0 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.966+0000 7f63acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:33.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:33.966+0000 7f63acff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:34.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.139+0000 7f63b5c53640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T04:09:34.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.139+0000 7f63b5c53640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T04:09:34.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.139+0000 7f63b5c53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:34.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.139+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) -- 0x7f63b006bcf0 con 0x7f63b009e1d0 2024-09-16T04:09:34.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.400+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v101) ==== 156+0+0 (unknown 1361214950 0 0) 0x7f63a0019a60 con 0x7f63b009e1d0 2024-09-16T04:09:34.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.400+0000 7f63acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T04:09:34.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.400+0000 7f63acff9640 10 monclient: _finish_command 2 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T04:09:34.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.506+0000 7f63b5c53640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T04:09:34.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.506+0000 7f63b5c53640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T04:09:34.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.506+0000 7f63b5c53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:34.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:34.506+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0} v 0) -- 0x7f63b011f130 con 0x7f63b009e1d0 2024-09-16T04:09:35.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.407+0000 7f63acff9640 1 -- 172.21.15.171:0/659132416 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}]=0 reweighted item id 11 name 'osd.11' to 113 in crush map v102) ==== 156+0+0 (unknown 291619180 0 0) 0x7f63a0011e00 con 0x7f63b009e1d0 2024-09-16T04:09:35.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.407+0000 7f63acff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 113.0}] 2024-09-16T04:09:35.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.407+0000 7f63acff9640 10 monclient: _finish_command 3 = system:0 reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T04:09:35.410 INFO:tasks.workunit.client.0.smithi171.stderr:reweighted item id 11 name 'osd.11' to 113 in crush map 2024-09-16T04:09:35.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 >> v1:172.21.15.171:6812/7327273 conn(0x7f6390038ea0 legacy=0x7f639003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 10 monclient: shutdown 2024-09-16T04:09:35.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:35.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 >> v1:172.21.15.171:6789/0 conn(0x7f63b009e1d0 legacy=0x7f63b0134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 shutdown_connections 2024-09-16T04:09:35.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 >> 172.21.15.171:0/659132416 conn(0x7f63b00a52a0 msgr2=0x7f63b0072b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:35.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 shutdown_connections 2024-09-16T04:09:35.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.409+0000 7f63b5c53640 1 -- 172.21.15.171:0/659132416 wait complete. 2024-09-16T04:09:35.424 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush reweight osd.11 123456 2024-09-16T04:09:35.424 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:35.425 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush reweight osd.11 123456 2024-09-16T04:09:35.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.506+0000 7fe5c2711640 10 monclient: get_monmap_and_config 2024-09-16T04:09:35.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.506+0000 7fe5c2711640 10 monclient: build_initial_monmap 2024-09-16T04:09:35.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.506+0000 7fe5c2711640 10 monclient: monmap: 2024-09-16T04:09:35.509 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:35.509 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:35.508443+0000 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:35.508443+0000 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:35.510 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 1 Processor -- start 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 1 -- start start 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 10 monclient: init 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 10 monclient(hunting): picked mon.noname-b con 0x7fe5bc132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:35.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 10 monclient(hunting): picked mon.noname-a con 0x7fe5bc09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 10 monclient(hunting): picked mon.noname-c con 0x7fe5bc132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc1332b0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.507+0000 7fe5c2711640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc134310 con 0x7fe5bc132e60 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc09d980 con 0x7fe5bc132a80 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 10 monclient(hunting): authenticate will time out at 951.014021s 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5bb7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fe5bc09ec20 legacy=0x7fe5bc133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c0cae640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe5bc132e60 0x7fe5bc13c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42234/0 (socket says 172.21.15.171:42234) 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c0cae640 1 -- 172.21.15.171:0/1768254294 learned_addr learned my addr 172.21.15.171:0/1768254294 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 204662998 0 0) 0x7fe5bc134310 con 0x7fe5bc132e60 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): my global_id is 5362 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a4003520 con 0x7fe5bc132e60 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4230566461 0 0) 0x7fe5bc09d980 con 0x7fe5bc132a80 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient(hunting): my global_id is 5393 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5bc134310 con 0x7fe5bc132a80 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1685695478 0 0) 0x7fe5a4003520 con 0x7fe5bc132e60 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 >> v1:172.21.15.148:6790/0 conn(0x7fe5bc132a80 legacy=0x7fe5bc1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 >> v1:172.21.15.148:6789/0 conn(0x7fe5bc09ec20 legacy=0x7fe5bc133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:35.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5bc09dc60 con 0x7fe5bc132e60 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 5 monclient: authenticate success, global_id 5362 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5b4001990 con 0x7fe5bc132e60 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:35.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: dump: 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:35.521 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5b4001c40 con 0x7fe5bc132e60 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 1 -- 172.21.15.171:0/1768254294 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5b4002010 con 0x7fe5bc132e60 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5c2711640 4 set_mon_vals no callback set 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.508+0000 7fe5baffd640 10 monclient: dump: 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.522 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 10 monclient: shutdown 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 1 -- 172.21.15.171:0/1768254294 >> v1:172.21.15.171:6789/0 conn(0x7fe5bc132e60 legacy=0x7fe5bc13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 1 -- 172.21.15.171:0/1768254294 shutdown_connections 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 1 -- 172.21.15.171:0/1768254294 >> 172.21.15.171:0/1768254294 conn(0x7fe5bc0a52a0 msgr2=0x7fe5bc0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 1 -- 172.21.15.171:0/1768254294 shutdown_connections 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 1 -- 172.21.15.171:0/1768254294 wait complete. 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 10 monclient: build_initial_monmap 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.509+0000 7fe5c2711640 10 monclient: monmap: 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:35.523 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 1 Processor -- start 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 1 -- start start 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient: init 2024-09-16T04:09:35.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient(hunting): picked mon.noname-b con 0x7fe5bc09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient(hunting): picked mon.noname-c con 0x7fe5bc132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient(hunting): picked mon.noname-a con 0x7fe5bc132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc1211e0 con 0x7fe5bc132e60 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc11fee0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5bc06ceb0 con 0x7fe5bc132a80 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5bbfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe5bc09ec20 0x7fe5bc1c7440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42238/0 (socket says 172.21.15.171:42238) 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c2711640 10 monclient(hunting): authenticate will time out at 951.016946s 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5bbfff640 1 -- 172.21.15.171:0/2687326162 learned_addr learned my addr 172.21.15.171:0/2687326162 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:35.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.510+0000 7fe5c0cae640 1 -- 172.21.15.171:0/2687326162 >> v1:172.21.15.148:6789/0 conn(0x7fe5bc132e60 legacy=0x7fe5bc1cf780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1480692973 0 0) 0x7fe5bc11fee0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): my global_id is 5365 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a8003520 con 0x7fe5bc09ec20 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 953375452 0 0) 0x7fe5bc06ceb0 con 0x7fe5bc132a80 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): my global_id is 5396 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5bc11fee0 con 0x7fe5bc132a80 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 901139287 0 0) 0x7fe5a8003520 con 0x7fe5bc09ec20 2024-09-16T04:09:35.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5bc06ceb0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5ac001970 con 0x7fe5bc09ec20 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.527 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3075164222 0 0) 0x7fe5bc06ceb0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 >> v1:172.21.15.148:6790/0 conn(0x7fe5bc132a80 legacy=0x7fe5bc130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 >> v1:172.21.15.148:6789/0 conn(0x7fe5bc132e60 legacy=0x7fe5bc1cf780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 monclient: found mon.b 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5bc0a3d20 con 0x7fe5bc09ec20 2024-09-16T04:09:35.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 5 monclient: authenticate success, global_id 5365 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 10 monclient: _renew_subs 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5bc09c910 con 0x7fe5bc09ec20 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 10 monclient: _renew_subs 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5c2711640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5bc1311e0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5ac001d20 con 0x7fe5bc09ec20 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5ac003a60 con 0x7fe5bc09ec20 2024-09-16T04:09:35.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5ba7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.511+0000 7fe5b8ff9640 10 monclient: dump: 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe5ac003c00 con 0x7fe5bc09ec20 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5c2711640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5c2711640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5c2711640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5c2711640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59c005080 con 0x7fe5bc09ec20 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(102..102 src has 1..102) ==== 5243+0+0 (unknown 4175377118 0 0) 0x7fe5ac047370 con 0x7fe5bc09ec20 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.512+0000 7fe5bb7fe640 10 monclient: get_auth_request con 0x7fe5a8038be0 auth_method 0 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.517+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe5ac015ce0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.517+0000 7fe5b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:35.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.517+0000 7fe5b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:35.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5c2711640 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T04:09:35.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5c2711640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T04:09:35.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5c2711640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5c2711640 1 -- 172.21.15.171:0/2687326162 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0} v 0) -- 0x7fe59c005b80 con 0x7fe5bc09ec20 2024-09-16T04:09:35.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5b8ff9640 1 -- 172.21.15.171:0/2687326162 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}]=-75 (75) Value too large for defined data type v102) ==== 146+0+0 (unknown 1119774799 0 0) 0x7fe5ac0155e0 con 0x7fe5bc09ec20 2024-09-16T04:09:35.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.11", "weight": 123456.0}] 2024-09-16T04:09:35.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.686+0000 7fe5b8ff9640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T04:09:35.690 INFO:tasks.workunit.client.0.smithi171.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T04:09:35.693 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:35.693 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:35.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep osd.11 2024-09-16T04:09:35.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 113 2024-09-16T04:09:35.769 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.766+0000 7ff00df31640 10 monclient: get_monmap_and_config 2024-09-16T04:09:35.769 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.766+0000 7ff00df31640 10 monclient: build_initial_monmap 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.766+0000 7ff00df31640 10 monclient: monmap: 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:35.768656+0000 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:35.768656+0000 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:35.770 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.767+0000 7ff00df31640 1 Processor -- start 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.767+0000 7ff00df31640 1 -- start start 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.767+0000 7ff00df31640 10 monclient: init 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient(hunting): picked mon.noname-c con 0x7ff00809ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient(hunting): picked mon.noname-b con 0x7ff008073570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:35.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient(hunting): picked mon.noname-a con 0x7ff00809dcb0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0080a0460 con 0x7ff00809dcb0 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0080a0db0 con 0x7ff00809ba30 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0080a11b0 con 0x7ff008073570 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff00df31640 10 monclient(hunting): authenticate will time out at 951.274387s 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff007fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff00809dcb0 legacy=0x7ff00813b460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff0077fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff00809ba30 0x7ff00809f640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42252/0 (socket says 172.21.15.171:42252) 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff0077fe640 1 -- 172.21.15.171:0/2367328269 learned_addr learned my addr 172.21.15.171:0/2367328269 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151464220 0 0) 0x7ff0080a0db0 con 0x7ff00809ba30 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.779 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): my global_id is 5368 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feff4003520 con 0x7ff00809ba30 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3858995748 0 0) 0x7ff0080a11b0 con 0x7ff008073570 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient(hunting): my global_id is 5399 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0080a0db0 con 0x7ff008073570 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3132726384 0 0) 0x7feff4003520 con 0x7ff00809ba30 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 >> v1:172.21.15.148:6790/0 conn(0x7ff008073570 legacy=0x7ff00809fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 >> v1:172.21.15.148:6789/0 conn(0x7ff00809dcb0 legacy=0x7ff00813b460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 monclient: found mon.noname-c 2024-09-16T04:09:35.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00809cff0 con 0x7ff00809ba30 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.768+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feff8001610 con 0x7ff00809ba30 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 5 monclient: authenticate success, global_id 5368 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: dump: 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.781 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feff8001930 con 0x7ff00809ba30 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 1 -- 172.21.15.171:0/2367328269 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feff8001d00 con 0x7ff00809ba30 2024-09-16T04:09:35.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 4 set_mon_vals no callback set 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff006ffd640 10 monclient: dump: 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.783 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 10 monclient: shutdown 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 1 -- 172.21.15.171:0/2367328269 >> v1:172.21.15.171:6789/0 conn(0x7ff00809ba30 legacy=0x7ff00809f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 1 -- 172.21.15.171:0/2367328269 shutdown_connections 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 1 -- 172.21.15.171:0/2367328269 >> 172.21.15.171:0/2367328269 conn(0x7ff0080a52a0 msgr2=0x7ff0080a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 1 -- 172.21.15.171:0/2367328269 shutdown_connections 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.769+0000 7ff00df31640 1 -- 172.21.15.171:0/2367328269 wait complete. 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient: build_initial_monmap 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient: monmap: 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:35.784 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 1 Processor -- start 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 1 -- start start 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient: init 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:35.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient(hunting): picked mon.noname-c con 0x7ff008073570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.770+0000 7ff00df31640 10 monclient(hunting): picked mon.noname-a con 0x7ff00809ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient(hunting): picked mon.noname-b con 0x7ff00809dcb0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0081a8cd0 con 0x7ff00809ba30 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0081a6170 con 0x7ff00809dcb0 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0081a7440 con 0x7ff008073570 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient(hunting): authenticate will time out at 951.277047s 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7feffffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff00809ba30 legacy=0x7ff0081a5a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff007fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff00809dcb0 0x7ff0081d3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42258/0 (socket says 172.21.15.171:42258) 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff007fff640 1 -- 172.21.15.171:0/225647832 learned_addr learned my addr 172.21.15.171:0/225647832 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166550414 0 0) 0x7ff0081a6170 con 0x7ff00809dcb0 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): my global_id is 5371 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe8003520 con 0x7ff00809dcb0 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773438299 0 0) 0x7ff0081a7440 con 0x7ff008073570 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): my global_id is 5402 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0081a6170 con 0x7ff008073570 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 470316359 0 0) 0x7fefe8003520 con 0x7ff00809dcb0 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0081a7440 con 0x7ff00809dcb0 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0000043f0 con 0x7ff00809dcb0 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:35.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2369676804 0 0) 0x7ff0081a6170 con 0x7ff008073570 2024-09-16T04:09:35.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefe8003520 con 0x7ff008073570 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feff8001430 con 0x7ff008073570 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient(hunting): dump: 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.789 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1918111787 0 0) 0x7ff0081a7440 con 0x7ff00809dcb0 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 >> v1:172.21.15.148:6790/0 conn(0x7ff008073570 legacy=0x7ff0081a5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 >> v1:172.21.15.148:6789/0 conn(0x7ff00809ba30 legacy=0x7ff0081a5a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 monclient: found mon.b 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0081a9290 con 0x7ff00809dcb0 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:35.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 5 monclient: authenticate success, global_id 5371 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient: _renew_subs 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff0080a4270 con 0x7ff00809dcb0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient: _renew_subs 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff0081a8fb0 con 0x7ff00809dcb0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0000046a0 con 0x7ff00809dcb0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff000004a70 con 0x7ff00809dcb0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff004ff9640 10 monclient: dump: 2024-09-16T04:09:35.791 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.771+0000 7ff0067fc640 4 set_mon_vals no callback set 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.772+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff000004d10 con 0x7ff00809dcb0 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.772+0000 7ff00df31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:35.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.772+0000 7ff00df31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.772+0000 7ff00df31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.772+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff008072180 con 0x7ff00809dcb0 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.772+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(102..102 src has 1..102) ==== 5243+0+0 (unknown 4175377118 0 0) 0x7ff0000475d0 con 0x7ff00809dcb0 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.773+0000 7ff0077fe640 10 monclient: get_auth_request con 0x7fefe8038ea0 auth_method 0 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.777+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff000015f40 con 0x7ff00809dcb0 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.777+0000 7ff004ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:35.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.777+0000 7ff004ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:35.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff00df31640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:35.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff00df31640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:35.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff00df31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:35.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7ff00806bcf0 con 0x7ff00809dcb0 2024-09-16T04:09:35.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v102) ==== 56+0+1173 (unknown 1606776867 0 1462146485) 0x7ff000015840 con 0x7ff00809dcb0 2024-09-16T04:09:35.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff004ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:35.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:35.942+0000 7ff004ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.042+0000 7ff00df31640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.042+0000 7ff00df31640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.042+0000 7ff00df31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.042+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7ff0081b7b50 con 0x7ff00809dcb0 2024-09-16T04:09:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.043+0000 7ff004ff9640 1 -- 172.21.15.171:0/225647832 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v103) ==== 56+0+1173 (unknown 3247416795 0 1462146485) 0x7ff000019b90 con 0x7ff00809dcb0 2024-09-16T04:09:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.043+0000 7ff004ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.043+0000 7ff004ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 >> v1:172.21.15.171:6812/7327273 conn(0x7fefe8038ea0 legacy=0x7fefe803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 10 monclient: shutdown 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 >> v1:172.21.15.171:6789/0 conn(0x7ff00809dcb0 legacy=0x7ff0081d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 shutdown_connections 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 >> 172.21.15.171:0/225647832 conn(0x7ff0080a52a0 msgr2=0x7ff0080a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 shutdown_connections 2024-09-16T04:09:36.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.045+0000 7ff00df31640 1 -- 172.21.15.171:0/225647832 wait complete. 2024-09-16T04:09:36.057 INFO:tasks.workunit.client.0.smithi171.stdout: 11 113.00000 osd.11 down 1.00000 1.00000 2024-09-16T04:09:36.057 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.11 2024-09-16T04:09:36.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.132+0000 7fedcd978640 10 monclient: get_monmap_and_config 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.132+0000 7fedcd978640 10 monclient: build_initial_monmap 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.132+0000 7fedcd978640 10 monclient: monmap: 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:36.134744+0000 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:36.134744+0000 2024-09-16T04:09:36.136 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:36.137 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:36.137 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:36.137 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:36.137 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:36.137 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:36.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.133+0000 7fedcd978640 1 Processor -- start 2024-09-16T04:09:36.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.133+0000 7fedcd978640 1 -- start start 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.133+0000 7fedcd978640 10 monclient: init 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient(hunting): picked mon.noname-a con 0x7fedc80a4040 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient(hunting): picked mon.noname-b con 0x7fedc809fc20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient(hunting): picked mon.noname-c con 0x7fedc80a0050 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc80737d0 con 0x7fedc80a4040 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc806c630 con 0x7fedc80a0050 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc806d8e0 con 0x7fedc809fc20 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:36.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient(hunting): authenticate will time out at 951.640135s 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fedc80a0050 0x7fedc813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42274/0 (socket says 172.21.15.171:42274) 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc77fe640 1 -- 172.21.15.171:0/3284277831 learned_addr learned my addr 172.21.15.171:0/3284277831 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc6ffd640 1 -- 172.21.15.171:0/3284277831 >> v1:172.21.15.148:6789/0 conn(0x7fedc80a4040 legacy=0x7fedc80a0bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547054870 0 0) 0x7fedc806c630 con 0x7fedc80a0050 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient(hunting): my global_id is 5374 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedb4003520 con 0x7fedc80a0050 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1395387721 0 0) 0x7fedb4003520 con 0x7fedc80a0050 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 >> v1:172.21.15.148:6790/0 conn(0x7fedc809fc20 legacy=0x7fedc813b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:36.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 >> v1:172.21.15.148:6789/0 conn(0x7fedc80a4040 legacy=0x7fedc80a0bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 monclient: found mon.noname-c 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc8143840 con 0x7fedc80a0050 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedc0001990 con 0x7fedc80a0050 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:36.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: dump: 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:36.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 5 monclient: authenticate success, global_id 5374 2024-09-16T04:09:36.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:36.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedc0001c80 con 0x7fedc80a0050 2024-09-16T04:09:36.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:36.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:36.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 1 -- 172.21.15.171:0/3284277831 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedc0002090 con 0x7fedc80a0050 2024-09-16T04:09:36.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:36.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedcd978640 4 set_mon_vals no callback set 2024-09-16T04:09:36.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.134+0000 7fedc5ffb640 10 monclient: dump: 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 10 monclient: shutdown 2024-09-16T04:09:36.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 1 -- 172.21.15.171:0/3284277831 >> v1:172.21.15.171:6789/0 conn(0x7fedc80a0050 legacy=0x7fedc813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 1 -- 172.21.15.171:0/3284277831 shutdown_connections 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 1 -- 172.21.15.171:0/3284277831 >> 172.21.15.171:0/3284277831 conn(0x7fedc80a7520 msgr2=0x7fedc80a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 1 -- 172.21.15.171:0/3284277831 shutdown_connections 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 1 -- 172.21.15.171:0/3284277831 wait complete. 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 10 monclient: build_initial_monmap 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.135+0000 7fedcd978640 10 monclient: monmap: 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:36.152 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 1 Processor -- start 2024-09-16T04:09:36.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 1 -- start start 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient: init 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient(hunting): picked mon.noname-a con 0x7fedc809fc20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient(hunting): picked mon.noname-b con 0x7fedc80a0050 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient(hunting): picked mon.noname-c con 0x7fedc80a4040 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc8072990 con 0x7fedc809fc20 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc8072c70 con 0x7fedc80a0050 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc8070f70 con 0x7fedc80a4040 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:36.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedcd978640 10 monclient(hunting): authenticate will time out at 951.642791s 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedc67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fedc80a0050 0x7fedc8070860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42280/0 (socket says 172.21.15.171:42280) 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedc67fc640 1 -- 172.21.15.171:0/1344147192 learned_addr learned my addr 172.21.15.171:0/1344147192 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.136+0000 7fedc6ffd640 1 -- 172.21.15.171:0/1344147192 >> v1:172.21.15.148:6789/0 conn(0x7fedc809fc20 legacy=0x7fedc8070150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297097165 0 0) 0x7fedc8072c70 con 0x7fedc80a0050 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): my global_id is 5377 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda8003520 con 0x7fedc80a0050 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353739341 0 0) 0x7fedc8070f70 con 0x7fedc80a4040 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:36.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): my global_id is 5408 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc8072c70 con 0x7fedc80a4040 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3619687866 0 0) 0x7feda8003520 con 0x7fedc80a0050 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedc8070f70 con 0x7fedc80a0050 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedb8001930 con 0x7fedc80a0050 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): dump: 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:36.156 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1617810440 0 0) 0x7fedc8070f70 con 0x7fedc80a0050 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 >> v1:172.21.15.148:6790/0 conn(0x7fedc80a4040 legacy=0x7fedc81d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 >> v1:172.21.15.148:6789/0 conn(0x7fedc809fc20 legacy=0x7fedc8070150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:36.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 monclient: found mon.b 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc8137ed0 con 0x7fedc80a0050 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: _finish_auth 0 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 5 monclient: authenticate success, global_id 5377 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 10 monclient: _renew_subs 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedc8142a40 con 0x7fedc80a0050 2024-09-16T04:09:36.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 10 monclient: _renew_subs 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fedc8072f50 con 0x7fedc80a0050 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedb8001d00 con 0x7fedc80a0050 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedb8003a60 con 0x7fedc80a0050 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedc57fa640 4 set_mon_vals no callback set 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.137+0000 7fedbffff640 10 monclient: dump: 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:36.159 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fedb8003c00 con 0x7fedc80a0050 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(103..103 src has 1..103) ==== 5243+0+0 (unknown 1906996657 0 0) 0x7fedb8047410 con 0x7fedc80a0050 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedcd978640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:36.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedcd978640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:36.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedcd978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedac005080 con 0x7fedc80a0050 2024-09-16T04:09:36.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.138+0000 7fedc6ffd640 10 monclient: get_auth_request con 0x7feda8038f60 auth_method 0 2024-09-16T04:09:36.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.143+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fedb8015d80 con 0x7fedc80a0050 2024-09-16T04:09:36.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.143+0000 7fedbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:36.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.143+0000 7fedbffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:36.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.312+0000 7fedcd978640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T04:09:36.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.312+0000 7fedcd978640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T04:09:36.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.312+0000 7fedcd978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:36.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.312+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) -- 0x7fedac005370 con 0x7fedc80a0050 2024-09-16T04:09:37.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.998+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 removed item id 11 name 'osd.11' from crush map v104) ==== 125+0+0 (unknown 1935494135 0 0) 0x7fedb8015680 con 0x7fedc80a0050 2024-09-16T04:09:37.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.998+0000 7fedbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T04:09:37.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:36.998+0000 7fedbffff640 10 monclient: _finish_command 2 = system:0 removed item id 11 name 'osd.11' from crush map 2024-09-16T04:09:37.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedcd978640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T04:09:37.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedcd978640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T04:09:37.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedcd978640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.11"} v 0) -- 0x7fedac002210 con 0x7fedc80a0050 2024-09-16T04:09:37.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedbffff640 1 -- 172.21.15.171:0/1344147192 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.11"}]=0 device 'osd.11' does not appear in the crush map v104) ==== 126+0+0 (unknown 4154262679 0 0) 0x7fedb80199d0 con 0x7fedc80a0050 2024-09-16T04:09:37.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedbffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.11"}] 2024-09-16T04:09:37.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.107+0000 7fedbffff640 10 monclient: _finish_command 3 = system:0 device 'osd.11' does not appear in the crush map 2024-09-16T04:09:37.111 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.11' does not appear in the crush map 2024-09-16T04:09:37.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 >> v1:172.21.15.171:6812/7327273 conn(0x7feda8038f60 legacy=0x7feda803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:37.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 10 monclient: shutdown 2024-09-16T04:09:37.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:37.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 >> v1:172.21.15.171:6789/0 conn(0x7fedc80a0050 legacy=0x7fedc8070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:37.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 shutdown_connections 2024-09-16T04:09:37.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 >> 172.21.15.171:0/1344147192 conn(0x7fedc80a7520 msgr2=0x7fedc80a2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:37.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 shutdown_connections 2024-09-16T04:09:37.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.109+0000 7fedcd978640 1 -- 172.21.15.171:0/1344147192 wait complete. 2024-09-16T04:09:37.122 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm osd.11 2024-09-16T04:09:37.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.197+0000 7fee04b86640 10 monclient: get_monmap_and_config 2024-09-16T04:09:37.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.197+0000 7fee04b86640 10 monclient: build_initial_monmap 2024-09-16T04:09:37.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.197+0000 7fee04b86640 10 monclient: monmap: 2024-09-16T04:09:37.201 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:37.201 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:37.201 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:37.199555+0000 2024-09-16T04:09:37.201 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:37.199555+0000 2024-09-16T04:09:37.201 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:37.201 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:37.202 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:37.202 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:37.202 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:37.202 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:37.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 1 Processor -- start 2024-09-16T04:09:37.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 1 -- start start 2024-09-16T04:09:37.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 10 monclient: init 2024-09-16T04:09:37.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:37.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 10 monclient(hunting): picked mon.noname-b con 0x7fee001338e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 10 monclient(hunting): picked mon.noname-a con 0x7fee0009aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 10 monclient(hunting): picked mon.noname-c con 0x7fee0013d240 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.198+0000 7fee04b86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee001343d0 con 0x7fee0009aa20 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee00075db0 con 0x7fee0013d240 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0009b1e0 con 0x7fee001338e0 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 10 monclient(hunting): authenticate will time out at 952.705024s 2024-09-16T04:09:37.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fee0013d240 0x7fee0013f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42288/0 (socket says 172.21.15.171:42288) 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfed9d640 1 -- 172.21.15.171:0/2252782599 learned_addr learned my addr 172.21.15.171:0/2252782599 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfdd9b640 1 -- 172.21.15.171:0/2252782599 >> v1:172.21.15.148:6789/0 conn(0x7fee0009aa20 legacy=0x7fee0013bb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3184070073 0 0) 0x7fee00075db0 con 0x7fee0013d240 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient(hunting): my global_id is 5380 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fede8003520 con 0x7fee0013d240 2024-09-16T04:09:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3826720941 0 0) 0x7fede8003520 con 0x7fee0013d240 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 >> v1:172.21.15.148:6790/0 conn(0x7fee001338e0 legacy=0x7fee00133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 >> v1:172.21.15.148:6789/0 conn(0x7fee0009aa20 legacy=0x7fee0013bb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: _finish_hunting 0 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 monclient: found mon.noname-c 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0009b4c0 con 0x7fee0013d240 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: _finish_auth 0 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedf8001990 con 0x7fee0013d240 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 5 monclient: authenticate success, global_id 5380 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: dump: 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedf8001c80 con 0x7fee0013d240 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 1 -- 172.21.15.171:0/2252782599 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedf8002090 con 0x7fee0013d240 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fee04b86640 4 set_mon_vals no callback set 2024-09-16T04:09:37.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:37.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.199+0000 7fedfd59a640 10 monclient: dump: 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:37.218 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:37.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 10 monclient: shutdown 2024-09-16T04:09:37.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:37.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 1 -- 172.21.15.171:0/2252782599 >> v1:172.21.15.171:6789/0 conn(0x7fee0013d240 legacy=0x7fee0013f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:37.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 1 -- 172.21.15.171:0/2252782599 shutdown_connections 2024-09-16T04:09:37.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 1 -- 172.21.15.171:0/2252782599 >> 172.21.15.171:0/2252782599 conn(0x7fee000a28d0 msgr2=0x7fee000a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 1 -- 172.21.15.171:0/2252782599 shutdown_connections 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.200+0000 7fee04b86640 1 -- 172.21.15.171:0/2252782599 wait complete. 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient: build_initial_monmap 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient: monmap: 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:37.219 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 1 Processor -- start 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 1 -- start start 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient: init 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:37.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient(hunting): picked mon.noname-a con 0x7fee0009aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient(hunting): picked mon.noname-b con 0x7fee001338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient(hunting): picked mon.noname-c con 0x7fee0013d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0011d590 con 0x7fee0009aa20 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0011db50 con 0x7fee001338e0 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0011d2c0 con 0x7fee0013d240 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fee04b86640 10 monclient(hunting): authenticate will time out at 952.707943s 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fedfe59c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fee0009aa20 legacy=0x7fee00132000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fedfdd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fee001338e0 0x7fee001d0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42300/0 (socket says 172.21.15.171:42300) 2024-09-16T04:09:37.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.201+0000 7fedfdd9b640 1 -- 172.21.15.171:0/2761542368 learned_addr learned my addr 172.21.15.171:0/2761542368 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040033092 0 0) 0x7fee0011db50 con 0x7fee001338e0 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): my global_id is 5383 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fede0003520 con 0x7fee001338e0 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3643295218 0 0) 0x7fee0011d2c0 con 0x7fee0013d240 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:37.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): my global_id is 5414 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee0011db50 con 0x7fee0013d240 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3942403514 0 0) 0x7fede0003520 con 0x7fee001338e0 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee0011d2c0 con 0x7fee001338e0 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedf4004180 con 0x7fee001338e0 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): dump: 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:37.223 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:37.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2129324579 0 0) 0x7fee0011d2c0 con 0x7fee001338e0 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 >> v1:172.21.15.148:6790/0 conn(0x7fee0013d240 legacy=0x7fee001d3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 >> v1:172.21.15.148:6789/0 conn(0x7fee0009aa20 legacy=0x7fee00132000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 monclient: found mon.b 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0011c420 con 0x7fee001338e0 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 5 monclient: authenticate success, global_id 5383 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 10 monclient: _renew_subs 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee0013b490 con 0x7fee001338e0 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 10 monclient: _renew_subs 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee00132710 con 0x7fee001338e0 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedf4004430 con 0x7fee001338e0 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:37.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fedf4004800 con 0x7fee001338e0 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedfcd99640 4 set_mon_vals no callback set 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.202+0000 7fedeeffd640 10 monclient: dump: 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:37.227 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fedf40049a0 con 0x7fee001338e0 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fee04b86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fee04b86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fee04b86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede4005080 con 0x7fee001338e0 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(104..104 src has 1..104) ==== 5192+0+0 (unknown 3090637794 0 0) 0x7fedf40473a0 con 0x7fee001338e0 2024-09-16T04:09:37.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.203+0000 7fedfe59c640 10 monclient: get_auth_request con 0x7fede0038f40 auth_method 0 2024-09-16T04:09:37.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.208+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fedf4015c60 con 0x7fee001338e0 2024-09-16T04:09:37.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.208+0000 7fedeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:37.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.208+0000 7fedeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:37.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.382+0000 7fee04b86640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T04:09:37.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.382+0000 7fee04b86640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T04:09:37.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.382+0000 7fee04b86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:37.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:37.382+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) -- 0x7fede4005370 con 0x7fee001338e0 2024-09-16T04:09:37.386 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-09-16T04:09:37.383+0000 7f1fb544d640 -1 mon.b@1(leader).osd e104 definitely_dead 0 2024-09-16T04:09:38.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.002+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 removed osd.11 v105) ==== 87+0+0 (unknown 702218926 0 0) 0x7fedf4015560 con 0x7fee001338e0 2024-09-16T04:09:38.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.002+0000 7fedeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T04:09:38.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.002+0000 7fedeeffd640 10 monclient: _finish_command 2 = system:0 removed osd.11 2024-09-16T04:09:38.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.114+0000 7fee04b86640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T04:09:38.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.114+0000 7fee04b86640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T04:09:38.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.114+0000 7fee04b86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.114+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.11"]} v 0) -- 0x7fede4000bc0 con 0x7fee001338e0 2024-09-16T04:09:38.118 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-09-16T04:09:38.115+0000 7f1fb544d640 -1 mon.b@1(leader).osd e105 definitely_dead 0 2024-09-16T04:09:38.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.115+0000 7fedeeffd640 1 -- 172.21.15.171:0/2761542368 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.11"]}]=0 osd.11 does not exist. v105) ==== 96+0+0 (unknown 2949963300 0 0) 0x7fedf40198b0 con 0x7fee001338e0 2024-09-16T04:09:38.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.115+0000 7fedeeffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.11"]}] 2024-09-16T04:09:38.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.115+0000 7fedeeffd640 10 monclient: _finish_command 3 = system:0 osd.11 does not exist. 2024-09-16T04:09:38.119 INFO:tasks.workunit.client.0.smithi171.stderr:osd.11 does not exist. 2024-09-16T04:09:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 >> v1:172.21.15.171:6812/7327273 conn(0x7fede0038f40 legacy=0x7fede003b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 10 monclient: shutdown 2024-09-16T04:09:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 >> v1:172.21.15.171:6789/0 conn(0x7fee001338e0 legacy=0x7fee001d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 shutdown_connections 2024-09-16T04:09:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 >> 172.21.15.171:0/2761542368 conn(0x7fee000a28d0 msgr2=0x7fee0013ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:38.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 shutdown_connections 2024-09-16T04:09:38.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.117+0000 7fee04b86640 1 -- 172.21.15.171:0/2761542368 wait complete. 2024-09-16T04:09:38.129 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 2024-09-16T04:09:38.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.203+0000 7f2c9396d640 10 monclient: get_monmap_and_config 2024-09-16T04:09:38.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.203+0000 7f2c9396d640 10 monclient: build_initial_monmap 2024-09-16T04:09:38.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.203+0000 7f2c9396d640 10 monclient: monmap: 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:38.205423+0000 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:38.205423+0000 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:38.207 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:38.208 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 1 Processor -- start 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 1 -- start start 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient: init 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient(hunting): picked mon.noname-b con 0x7f2c8c133560 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c8c09d2e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:38.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c8c134050 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8c09dbc0 con 0x7f2c8c134050 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8c09fc70 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8c09d8e0 con 0x7f2c8c133560 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c9396d640 10 monclient(hunting): authenticate will time out at 953.710884s 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c90f08640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2c8c09d2e0 0x7f2c8c138880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33050/0 (socket says 172.21.15.171:33050) 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.204+0000 7f2c90f08640 1 -- 172.21.15.171:0/1163727725 learned_addr learned my addr 172.21.15.171:0/1163727725 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c91f0a640 1 -- 172.21.15.171:0/1163727725 >> v1:172.21.15.148:6789/0 conn(0x7f2c8c134050 legacy=0x7f2c8c13bfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25248881 0 0) 0x7f2c8c09fc70 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:38.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient(hunting): my global_id is 5386 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c78003520 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2486176354 0 0) 0x7f2c78003520 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 >> v1:172.21.15.148:6790/0 conn(0x7f2c8c133560 legacy=0x7f2c8c133940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 >> v1:172.21.15.148:6789/0 conn(0x7f2c8c134050 legacy=0x7f2c8c13bfb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 monclient: found mon.noname-c 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8c09f6b0 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c9396d640 5 monclient: authenticate success, global_id 5386 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c88001610 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c9396d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: dump: 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.218 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c9396d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c88001970 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c9396d640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 1 -- 172.21.15.171:0/1163727725 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c88001d40 con 0x7f2c8c09d2e0 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c9396d640 4 set_mon_vals no callback set 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:38.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.205+0000 7f2c83fff640 10 monclient: dump: 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 10 monclient: shutdown 2024-09-16T04:09:38.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 1 -- 172.21.15.171:0/1163727725 >> v1:172.21.15.171:6789/0 conn(0x7f2c8c09d2e0 legacy=0x7f2c8c138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 1 -- 172.21.15.171:0/1163727725 shutdown_connections 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 1 -- 172.21.15.171:0/1163727725 >> 172.21.15.171:0/1163727725 conn(0x7f2c8c0a17b0 msgr2=0x7f2c8c0a1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 1 -- 172.21.15.171:0/1163727725 shutdown_connections 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 1 -- 172.21.15.171:0/1163727725 wait complete. 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 10 monclient: build_initial_monmap 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.206+0000 7f2c9396d640 10 monclient: monmap: 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:38.221 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 1 Processor -- start 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 1 -- start start 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient: init 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:38.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c8c133560 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient(hunting): picked mon.noname-b con 0x7f2c8c134050 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c8c124040 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8c1269b0 con 0x7f2c8c124040 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8c1257e0 con 0x7f2c8c134050 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c8c126c90 con 0x7f2c8c133560 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c9396d640 10 monclient(hunting): authenticate will time out at 953.713582s 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c91f0a640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2c8c124040 legacy=0x7f2c8c1d3f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c90f08640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2c8c134050 0x7f2c8c1d0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33060/0 (socket says 172.21.15.171:33060) 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c90f08640 1 -- 172.21.15.171:0/2757926247 learned_addr learned my addr 172.21.15.171:0/2757926247 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486025184 0 0) 0x7f2c8c1257e0 con 0x7f2c8c134050 2024-09-16T04:09:38.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): my global_id is 5389 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c70003520 con 0x7f2c8c134050 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727434989 0 0) 0x7f2c8c126c90 con 0x7f2c8c133560 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): my global_id is 5420 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c8c1257e0 con 0x7f2c8c133560 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1102440209 0 0) 0x7f2c70003520 con 0x7f2c8c134050 2024-09-16T04:09:38.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c8c126c90 con 0x7f2c8c134050 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c880015d0 con 0x7f2c8c134050 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.207+0000 7f2c81ffb640 10 monclient(hunting): dump: 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 343444072 0 0) 0x7f2c8c126c90 con 0x7f2c8c134050 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 >> v1:172.21.15.148:6790/0 conn(0x7f2c8c133560 legacy=0x7f2c8c1cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 >> v1:172.21.15.148:6789/0 conn(0x7f2c8c124040 legacy=0x7f2c8c1d3f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 monclient: found mon.b 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8c072110 con 0x7f2c8c134050 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 5 monclient: authenticate success, global_id 5389 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 10 monclient: _renew_subs 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8c1266d0 con 0x7f2c8c134050 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c88004cc0 con 0x7f2c8c134050 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c88005010 con 0x7f2c8c134050 2024-09-16T04:09:38.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c81ffb640 10 monclient: dump: 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c837fe640 4 set_mon_vals no callback set 2024-09-16T04:09:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 10 monclient: _renew_subs 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.208+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c8c1cbd20 con 0x7f2c8c134050 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2c88010440 con 0x7f2c8c134050 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c9396d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c9396d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c9396d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c8c09b370 con 0x7f2c8c134050 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(105..105 src has 1..105) ==== 5240+0+0 (unknown 2123597176 0 0) 0x7f2c88047520 con 0x7f2c8c134050 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.209+0000 7f2c91709640 10 monclient: get_auth_request con 0x7f2c70038f60 auth_method 0 2024-09-16T04:09:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.213+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2c8c09b370 con 0x7f2c8c134050 2024-09-16T04:09:38.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.213+0000 7f2c81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:38.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.213+0000 7f2c81ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:38.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.389+0000 7f2c9396d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:09:38.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.389+0000 7f2c9396d640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T04:09:38.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.389+0000 7f2c9396d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:38.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:38.389+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f2c8c0a2f40 con 0x7f2c8c134050 2024-09-16T04:09:39.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.003+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v106) ==== 58+0+3 (unknown 3507046352 0 212448812) 0x7f2c8c0a2f40 con 0x7f2c8c134050 2024-09-16T04:09:39.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.003+0000 7f2c81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T04:09:39.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.003+0000 7f2c81ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:39.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.114+0000 7f2c9396d640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:09:39.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.114+0000 7f2c9396d640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T04:09:39.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.114+0000 7f2c9396d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:39.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:39.114+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f2c8c0008d0 con 0x7f2c8c134050 2024-09-16T04:09:40.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.019+0000 7f2c81ffb640 1 -- 172.21.15.171:0/2757926247 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v107) ==== 58+0+3 (unknown 1685887268 0 944643765) 0x7f2c88015e50 con 0x7f2c8c134050 2024-09-16T04:09:40.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.019+0000 7f2c81ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T04:09:40.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.019+0000 7f2c81ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:40.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 >> v1:172.21.15.171:6812/7327273 conn(0x7f2c70038f60 legacy=0x7f2c7003b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 10 monclient: shutdown 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 >> v1:172.21.15.171:6789/0 conn(0x7f2c8c134050 legacy=0x7f2c8c1d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 shutdown_connections 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 >> 172.21.15.171:0/2757926247 conn(0x7f2c8c0a17b0 msgr2=0x7f2c8c0a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 shutdown_connections 2024-09-16T04:09:40.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.022+0000 7f2c9396d640 1 -- 172.21.15.171:0/2757926247 wait complete. 2024-09-16T04:09:40.035 INFO:tasks.workunit.client.0.smithi171.stderr:+ o4=12 2024-09-16T04:09:40.035 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 2024-09-16T04:09:40.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.111+0000 7f9ca5f33640 10 monclient: get_monmap_and_config 2024-09-16T04:09:40.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.111+0000 7f9ca5f33640 10 monclient: build_initial_monmap 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.111+0000 7f9ca5f33640 10 monclient: monmap: 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:40.112655+0000 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:40.112655+0000 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:40.114 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.112+0000 7f9ca5f33640 1 Processor -- start 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.112+0000 7f9ca5f33640 1 -- start start 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.112+0000 7f9ca5f33640 10 monclient: init 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca009fa10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:40.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca0068f80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca009e720 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca00731a0 con 0x7f9ca0068f80 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca009bef0 con 0x7f9ca009fa10 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca00a16e0 con 0x7f9ca009e720 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient(hunting): authenticate will time out at 955.618219s 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9ca009fa10 0x7f9ca009fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33062/0 (socket says 172.21.15.171:33062) 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9f7fe640 1 -- 172.21.15.171:0/2035174294 learned_addr learned my addr 172.21.15.171:0/2035174294 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9effd640 1 -- 172.21.15.171:0/2035174294 >> v1:172.21.15.148:6789/0 conn(0x7f9ca0068f80 legacy=0x7f9ca0139cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581192025 0 0) 0x7f9ca009bef0 con 0x7f9ca009fa10 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:40.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): my global_id is 5392 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c88003520 con 0x7f9ca009fa10 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721024098 0 0) 0x7f9ca00a16e0 con 0x7f9ca009e720 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient(hunting): my global_id is 5423 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ca009bef0 con 0x7f9ca009e720 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 146096851 0 0) 0x7f9c88003520 con 0x7f9ca009fa10 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 >> v1:172.21.15.148:6790/0 conn(0x7f9ca009e720 legacy=0x7f9ca013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 >> v1:172.21.15.148:6789/0 conn(0x7f9ca0068f80 legacy=0x7f9ca0139cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:40.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca00a0500 con 0x7f9ca009fa10 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9c90001610 con 0x7f9ca009fa10 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 5 monclient: authenticate success, global_id 5392 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:40.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: dump: 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c90001930 con 0x7f9ca009fa10 2024-09-16T04:09:40.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 1 -- 172.21.15.171:0/2035174294 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9c90001d00 con 0x7f9ca009fa10 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9ca5f33640 4 set_mon_vals no callback set 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.113+0000 7f9c9e7fc640 10 monclient: dump: 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.128 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 10 monclient: shutdown 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 1 -- 172.21.15.171:0/2035174294 >> v1:172.21.15.171:6789/0 conn(0x7f9ca009fa10 legacy=0x7f9ca009fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 1 -- 172.21.15.171:0/2035174294 shutdown_connections 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 1 -- 172.21.15.171:0/2035174294 >> 172.21.15.171:0/2035174294 conn(0x7f9ca00a1b10 msgr2=0x7f9ca00a1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 1 -- 172.21.15.171:0/2035174294 shutdown_connections 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 1 -- 172.21.15.171:0/2035174294 wait complete. 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 10 monclient: build_initial_monmap 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.114+0000 7f9ca5f33640 10 monclient: monmap: 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:40.129 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 1 Processor -- start 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 1 -- start start 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient: init 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:40.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca0068f80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca009e720 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca009fa10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca0121110 con 0x7f9ca009e720 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca01213f0 con 0x7f9ca009fa10 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca006efb0 con 0x7f9ca0068f80 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9ca5f33640 10 monclient(hunting): authenticate will time out at 955.620882s 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9c9effd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f9ca009e720 legacy=0x7f9ca01ce260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9c9ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9ca009fa10 0x7f9ca01d1990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33074/0 (socket says 172.21.15.171:33074) 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.115+0000 7f9c9ffff640 1 -- 172.21.15.171:0/1732674521 learned_addr learned my addr 172.21.15.171:0/1732674521 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374137466 0 0) 0x7f9ca01213f0 con 0x7f9ca009fa10 2024-09-16T04:09:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): my global_id is 5395 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c8c003520 con 0x7f9ca009fa10 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838638463 0 0) 0x7f9ca006efb0 con 0x7f9ca0068f80 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): my global_id is 5426 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ca01213f0 con 0x7f9ca0068f80 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 694782139 0 0) 0x7f9c8c003520 con 0x7f9ca009fa10 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ca006efb0 con 0x7f9ca009fa10 2024-09-16T04:09:40.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9c98001c30 con 0x7f9ca009fa10 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): dump: 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:40.133 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3408510870 0 0) 0x7f9ca01213f0 con 0x7f9ca0068f80 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c8c003520 con 0x7f9ca0068f80 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9c90001430 con 0x7f9ca0068f80 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient(hunting): dump: 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:40.134 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 888269413 0 0) 0x7f9ca006efb0 con 0x7f9ca009fa10 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 >> v1:172.21.15.148:6790/0 conn(0x7f9ca0068f80 legacy=0x7f9ca0133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 >> v1:172.21.15.148:6789/0 conn(0x7f9ca009e720 legacy=0x7f9ca01ce260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 monclient: found mon.b 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca009c200 con 0x7f9ca009fa10 2024-09-16T04:09:40.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 5 monclient: authenticate success, global_id 5395 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 10 monclient: _renew_subs 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca011f100 con 0x7f9ca009fa10 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 10 monclient: _renew_subs 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ca011f9e0 con 0x7f9ca009fa10 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c98003990 con 0x7f9ca009fa10 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9c98003d60 con 0x7f9ca009fa10 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:40.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c87fff640 10 monclient: dump: 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.116+0000 7f9c9dffb640 4 set_mon_vals no callback set 2024-09-16T04:09:40.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9c98004000 con 0x7f9ca009fa10 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9ca5f33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9ca5f33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9ca5f33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca01bc0b0 con 0x7f9ca009fa10 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(107..107 src has 1..107) ==== 5359+0+0 (unknown 703706567 0 0) 0x7f9c980476e0 con 0x7f9ca009fa10 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.117+0000 7f9c9f7fe640 10 monclient: get_auth_request con 0x7f9c8c038ec0 auth_method 0 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.122+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9c98016040 con 0x7f9ca009fa10 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.122+0000 7f9c87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:40.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.122+0000 7f9c87fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:40.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.300+0000 7f9ca5f33640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:09:40.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.301+0000 7f9ca5f33640 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-16T04:09:40.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.301+0000 7f9ca5f33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:40.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:40.301+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f9ca006c140 con 0x7f9ca009fa10 2024-09-16T04:09:41.024 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.022+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create"}]=0 v108) ==== 58+0+3 (unknown 1610685179 0 736921282) 0x7f9c980434c0 con 0x7f9ca009fa10 2024-09-16T04:09:41.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.022+0000 7f9c87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-16T04:09:41.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.022+0000 7f9c87fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:41.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.137+0000 7f9ca5f33640 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-16T04:09:41.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.137+0000 7f9ca5f33640 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-16T04:09:41.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.137+0000 7f9ca5f33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:41.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:41.137+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create"} v 0) -- 0x7f9ca00a0890 con 0x7f9ca009fa10 2024-09-16T04:09:42.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.031+0000 7f9c87fff640 1 -- 172.21.15.171:0/1732674521 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create"}]=0 v109) ==== 58+0+3 (unknown 3581199375 0 1367428999) 0x7f9ca00a0890 con 0x7f9ca009fa10 2024-09-16T04:09:42.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.031+0000 7f9c87fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-16T04:09:42.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.031+0000 7f9c87fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:42.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 >> v1:172.21.15.171:6812/7327273 conn(0x7f9c8c038ec0 legacy=0x7f9c8c03b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:42.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 10 monclient: shutdown 2024-09-16T04:09:42.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:42.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 >> v1:172.21.15.171:6789/0 conn(0x7f9ca009fa10 legacy=0x7f9ca01d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:42.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 shutdown_connections 2024-09-16T04:09:42.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 >> 172.21.15.171:0/1732674521 conn(0x7f9ca00a1b10 msgr2=0x7f9ca0072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:42.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.034+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 shutdown_connections 2024-09-16T04:09:42.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.035+0000 7f9ca5f33640 1 -- 172.21.15.171:0/1732674521 wait complete. 2024-09-16T04:09:42.049 INFO:tasks.workunit.client.0.smithi171.stderr:+ o5=14 2024-09-16T04:09:42.049 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 12 123 root=default host=foobaz 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.134+0000 7f41a445e640 10 monclient: get_monmap_and_config 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.134+0000 7f41a445e640 10 monclient: build_initial_monmap 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.134+0000 7f41a445e640 10 monclient: monmap: 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:42.134849+0000 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:42.134849+0000 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:42.136 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:42.137 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:42.137 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:42.137 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:42.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.134+0000 7f41a445e640 1 Processor -- start 2024-09-16T04:09:42.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.134+0000 7f41a445e640 1 -- start start 2024-09-16T04:09:42.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.134+0000 7f41a445e640 10 monclient: init 2024-09-16T04:09:42.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:42.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient(hunting): picked mon.noname-b con 0x7f419c136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient(hunting): picked mon.noname-c con 0x7f419c137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient(hunting): picked mon.noname-a con 0x7f419c1375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c09b2c0 con 0x7f419c1375a0 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c09b5a0 con 0x7f419c137140 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c0a6640 con 0x7f419c136d60 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a445e640 10 monclient(hunting): authenticate will time out at 957.640567s 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a29fb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f419c1375a0 legacy=0x7f419c13f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a19f9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f419c137140 0x7f419c13bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33080/0 (socket says 172.21.15.171:33080) 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a19f9640 1 -- 172.21.15.171:0/1984219473 learned_addr learned my addr 172.21.15.171:0/1984219473 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075972168 0 0) 0x7f419c09b5a0 con 0x7f419c137140 2024-09-16T04:09:42.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a11f8640 10 monclient(hunting): _negotiate 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a11f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a11f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a11f8640 10 monclient(hunting): my global_id is 5398 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.135+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4184003520 con 0x7f419c137140 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 502215257 0 0) 0x7f4184003520 con 0x7f419c137140 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 >> v1:172.21.15.148:6790/0 conn(0x7f419c136d60 legacy=0x7f419c09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 >> v1:172.21.15.148:6789/0 conn(0x7f419c1375a0 legacy=0x7f419c13f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: _finish_hunting 0 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 monclient: found mon.noname-c 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c143b20 con 0x7f419c137140 2024-09-16T04:09:42.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: _finish_auth 0 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: _check_auth_tickets 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4190001990 con 0x7f419c137140 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 5 monclient: authenticate success, global_id 5398 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: dump: 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:42.146 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4190001c80 con 0x7f419c137140 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:42.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 4 set_mon_vals no callback set 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 1 -- 172.21.15.171:0/1984219473 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4190002050 con 0x7f419c137140 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a11f8640 10 monclient: dump: 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.148 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 10 monclient: shutdown 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 1 -- 172.21.15.171:0/1984219473 >> v1:172.21.15.171:6789/0 conn(0x7f419c137140 legacy=0x7f419c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 1 -- 172.21.15.171:0/1984219473 shutdown_connections 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 1 -- 172.21.15.171:0/1984219473 >> 172.21.15.171:0/1984219473 conn(0x7f419c0a9ae0 msgr2=0x7f419c0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 1 -- 172.21.15.171:0/1984219473 shutdown_connections 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.136+0000 7f41a445e640 1 -- 172.21.15.171:0/1984219473 wait complete. 2024-09-16T04:09:42.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.137+0000 7f41a445e640 10 monclient: build_initial_monmap 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.137+0000 7f41a445e640 10 monclient: monmap: 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.137+0000 7f41a445e640 1 Processor -- start 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.137+0000 7f41a445e640 1 -- start start 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.137+0000 7f41a445e640 10 monclient: init 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:42.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient(hunting): picked mon.noname-a con 0x7f419c136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient(hunting): picked mon.noname-c con 0x7f419c137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient(hunting): picked mon.noname-b con 0x7f419c1375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c138890 con 0x7f419c136d60 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c0a8720 con 0x7f419c1375a0 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c0a81f0 con 0x7f419c137140 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a445e640 10 monclient(hunting): authenticate will time out at 957.643329s 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a21fa640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f419c136d60 legacy=0x7f419c137a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a29fb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f419c1375a0 0x7f419c1d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33090/0 (socket says 172.21.15.171:33090) 2024-09-16T04:09:42.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f41a29fb640 1 -- 172.21.15.171:0/1863466494 learned_addr learned my addr 172.21.15.171:0/1863466494 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398083697 0 0) 0x7f419c0a8720 con 0x7f419c1375a0 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): my global_id is 5401 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4188003520 con 0x7f419c1375a0 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3749266935 0 0) 0x7f4188003520 con 0x7f419c1375a0 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f419c0a8720 con 0x7f419c1375a0 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f418c004030 con 0x7f419c1375a0 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): dump: 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:42.152 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095251710 0 0) 0x7f419c0a81f0 con 0x7f419c137140 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient(hunting): my global_id is 5432 2024-09-16T04:09:42.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4188003520 con 0x7f419c137140 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 792277590 0 0) 0x7f419c0a8720 con 0x7f419c1375a0 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 >> v1:172.21.15.148:6790/0 conn(0x7f419c137140 legacy=0x7f419c138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 >> v1:172.21.15.148:6789/0 conn(0x7f419c136d60 legacy=0x7f419c137a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 1 monclient: found mon.b 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.138+0000 7f419affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c0a03b0 con 0x7f419c1375a0 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 10 monclient: _finish_auth 0 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 5 monclient: authenticate success, global_id 5401 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 monclient: _renew_subs 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:42.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f419c13a8c0 con 0x7f419c1375a0 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 monclient: _renew_subs 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f419c139470 con 0x7f419c1375a0 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f418c004320 con 0x7f419c1375a0 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f418c0046f0 con 0x7f419c1375a0 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a09f7640 4 set_mon_vals no callback set 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 10 monclient: dump: 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.155 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f418c004890 con 0x7f419c1375a0 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.139+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f417c005080 con 0x7f419c1375a0 2024-09-16T04:09:42.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.140+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(109..109 src has 1..109) ==== 5597+0+0 (unknown 1135549016 0 0) 0x7f418c047490 con 0x7f419c1375a0 2024-09-16T04:09:42.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.140+0000 7f41a21fa640 10 monclient: get_auth_request con 0x7f4188038fe0 auth_method 0 2024-09-16T04:09:42.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.144+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f418c015b60 con 0x7f419c1375a0 2024-09-16T04:09:42.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.144+0000 7f419affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:42.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.144+0000 7f419affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:42.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.314+0000 7f41a445e640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:42.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.314+0000 7f41a445e640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:42.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.314+0000 7f41a445e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:42.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:42.314+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7f417c005370 con 0x7f419c1375a0 2024-09-16T04:09:43.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.033+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map v110) ==== 220+0+0 (unknown 1155089183 0 0) 0x7f418c019810 con 0x7f419c1375a0 2024-09-16T04:09:43.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.033+0000 7f419affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:43.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.033+0000 7f419affd640 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T04:09:43.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.135+0000 7f41a445e640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:43.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.135+0000 7f41a445e640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:43.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.135+0000 7f41a445e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.135+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7f417c005c40 con 0x7f419c1375a0 2024-09-16T04:09:43.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.136+0000 7f419affd640 1 -- 172.21.15.171:0/1863466494 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change v110) ==== 218+0+0 (unknown 3560193959 0 0) 0x7f418c013830 con 0x7f419c1375a0 2024-09-16T04:09:43.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.136+0000 7f419affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:43.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.136+0000 7f419affd640 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T04:09:43.138 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 12 name 'osd.12' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.137+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 >> v1:172.21.15.171:6812/7327273 conn(0x7f4188038fe0 legacy=0x7f418803b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.137+0000 7f41a445e640 10 monclient: shutdown 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.137+0000 7f41a445e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.137+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 >> v1:172.21.15.171:6789/0 conn(0x7f419c1375a0 legacy=0x7f419c1d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.137+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 shutdown_connections 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.137+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 >> 172.21.15.171:0/1863466494 conn(0x7f419c0a9ae0 msgr2=0x7f419c13ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.138+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 shutdown_connections 2024-09-16T04:09:43.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.138+0000 7f41a445e640 1 -- 172.21.15.171:0/1863466494 wait complete. 2024-09-16T04:09:43.149 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add 14 123 root=default host=foobaz 2024-09-16T04:09:43.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.229+0000 7f2e98786640 10 monclient: get_monmap_and_config 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.229+0000 7f2e98786640 10 monclient: build_initial_monmap 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.229+0000 7f2e98786640 10 monclient: monmap: 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:43.229810+0000 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:43.229810+0000 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:43.231 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.229+0000 7f2e98786640 1 Processor -- start 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.229+0000 7f2e98786640 1 -- start start 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.229+0000 7f2e98786640 10 monclient: init 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient(hunting): picked mon.noname-b con 0x7f2e900a0ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9009f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9013d2c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e900737b0 con 0x7f2e9009f320 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e9006c6d0 con 0x7f2e9013d2c0 2024-09-16T04:09:43.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e9006d980 con 0x7f2e900a0ae0 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient(hunting): authenticate will time out at 958.735313s 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e96d23640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2e9013d2c0 0x7f2e9013f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33102/0 (socket says 172.21.15.171:33102) 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95d21640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2e9009f320 legacy=0x7f2e9013bba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e96d23640 1 -- 172.21.15.171:0/1046550654 learned_addr learned my addr 172.21.15.171:0/1046550654 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471154843 0 0) 0x7f2e9006c6d0 con 0x7f2e9013d2c0 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient(hunting): _negotiate 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient(hunting): my global_id is 5404 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e78003520 con 0x7f2e9013d2c0 2024-09-16T04:09:43.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3877807623 0 0) 0x7f2e78003520 con 0x7f2e9013d2c0 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 >> v1:172.21.15.148:6790/0 conn(0x7f2e900a0ae0 legacy=0x7f2e900a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 >> v1:172.21.15.148:6789/0 conn(0x7f2e9009f320 legacy=0x7f2e9013bba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: _finish_hunting 0 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 monclient: found mon.noname-c 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9006c950 con 0x7f2e9013d2c0 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: _finish_auth 0 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: _check_auth_tickets 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2e80001990 con 0x7f2e9013d2c0 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 5 monclient: authenticate success, global_id 5404 2024-09-16T04:09:43.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: dump: 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:43.242 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e80001cf0 con 0x7f2e9013d2c0 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 1 -- 172.21.15.171:0/1046550654 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2e80004c20 con 0x7f2e9013d2c0 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e98786640 4 set_mon_vals no callback set 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.230+0000 7f2e95520640 10 monclient: dump: 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.243 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 10 monclient: shutdown 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 1 -- 172.21.15.171:0/1046550654 >> v1:172.21.15.171:6789/0 conn(0x7f2e9013d2c0 legacy=0x7f2e9013f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 1 -- 172.21.15.171:0/1046550654 shutdown_connections 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 1 -- 172.21.15.171:0/1046550654 >> 172.21.15.171:0/1046550654 conn(0x7f2e900a77f0 msgr2=0x7f2e900a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 1 -- 172.21.15.171:0/1046550654 shutdown_connections 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.231+0000 7f2e98786640 1 -- 172.21.15.171:0/1046550654 wait complete. 2024-09-16T04:09:43.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient: build_initial_monmap 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient: monmap: 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 1 Processor -- start 2024-09-16T04:09:43.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 1 -- start start 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient: init 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9009f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient(hunting): picked mon.noname-b con 0x7f2e900a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9013d2c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e901271b0 con 0x7f2e9009f320 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e90127490 con 0x7f2e900a0ae0 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e900a2110 con 0x7f2e9013d2c0 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e98786640 10 monclient(hunting): authenticate will time out at 958.737874s 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e95d21640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2e900a0ae0 0x7f2e901d0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33112/0 (socket says 172.21.15.171:33112) 2024-09-16T04:09:43.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e95d21640 1 -- 172.21.15.171:0/1235792681 learned_addr learned my addr 172.21.15.171:0/1235792681 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.232+0000 7f2e96522640 1 -- 172.21.15.171:0/1235792681 >> v1:172.21.15.148:6789/0 conn(0x7f2e9009f320 legacy=0x7f2e901cb900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877638516 0 0) 0x7f2e90127490 con 0x7f2e900a0ae0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): my global_id is 5407 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e7c003520 con 0x7f2e900a0ae0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 519729 0 0) 0x7f2e7c003520 con 0x7f2e900a0ae0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2e90127490 con 0x7f2e900a0ae0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2e84004180 con 0x7f2e900a0ae0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:43.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): dump: 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3529323684 0 0) 0x7f2e900a2110 con 0x7f2e9013d2c0 2024-09-16T04:09:43.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient(hunting): my global_id is 5438 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e7c003520 con 0x7f2e9013d2c0 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1018951018 0 0) 0x7f2e90127490 con 0x7f2e900a0ae0 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 >> v1:172.21.15.148:6790/0 conn(0x7f2e9013d2c0 legacy=0x7f2e901d4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 >> v1:172.21.15.148:6789/0 conn(0x7f2e9009f320 legacy=0x7f2e901cb900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 monclient: found mon.b 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e900a1bc0 con 0x7f2e900a0ae0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 5 monclient: authenticate success, global_id 5407 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 10 monclient: _renew_subs 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9009e1b0 con 0x7f2e900a0ae0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 10 monclient: _renew_subs 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2e90125a80 con 0x7f2e900a0ae0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2e840044e0 con 0x7f2e900a0ae0 2024-09-16T04:09:43.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2e840048b0 con 0x7f2e900a0ae0 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e94d1f640 4 set_mon_vals no callback set 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.233+0000 7f2e8effd640 10 monclient: dump: 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:43.251 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2e84004b10 con 0x7f2e900a0ae0 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e98786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e98786640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e98786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e70005080 con 0x7f2e900a0ae0 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(110..110 src has 1..110) ==== 5781+0+0 (unknown 325310165 0 0) 0x7f2e84047640 con 0x7f2e900a0ae0 2024-09-16T04:09:43.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.234+0000 7f2e96522640 10 monclient: get_auth_request con 0x7f2e7c038f90 auth_method 0 2024-09-16T04:09:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.239+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2e84015df0 con 0x7f2e900a0ae0 2024-09-16T04:09:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.239+0000 7f2e8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.239+0000 7f2e8effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:43.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.406+0000 7f2e98786640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:43.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.406+0000 7f2e98786640 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:43.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.406+0000 7f2e98786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:43.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:43.406+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7f2e70005370 con 0x7f2e900a0ae0 2024-09-16T04:09:44.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.043+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map v111) ==== 220+0+0 (unknown 1590490178 0 0) 0x7f2e840156f0 con 0x7f2e900a0ae0 2024-09-16T04:09:44.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.043+0000 7f2e8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:44.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.043+0000 7f2e8effd640 10 monclient: _finish_command 2 = system:0 add item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-16T04:09:44.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e98786640 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:44.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e98786640 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e98786640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) -- 0x7f2e70005c40 con 0x7f2e900a0ae0 2024-09-16T04:09:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e8effd640 1 -- 172.21.15.171:0/1235792681 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change v111) ==== 218+0+0 (unknown 1673024617 0 0) 0x7f2e8401e2a0 con 0x7f2e900a0ae0 2024-09-16T04:09:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e8effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 14, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-16T04:09:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.150+0000 7f2e8effd640 10 monclient: _finish_command 3 = system:0 set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T04:09:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:set item id 14 name 'osd.14' weight 123 at location {host=foobaz,root=default}: no change 2024-09-16T04:09:44.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 >> v1:172.21.15.171:6812/7327273 conn(0x7f2e7c038f90 legacy=0x7f2e7c03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 10 monclient: shutdown 2024-09-16T04:09:44.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:44.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 >> v1:172.21.15.171:6789/0 conn(0x7f2e900a0ae0 legacy=0x7f2e901d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 shutdown_connections 2024-09-16T04:09:44.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 >> 172.21.15.171:0/1235792681 conn(0x7f2e900a77f0 msgr2=0x7f2e9009ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:44.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 shutdown_connections 2024-09-16T04:09:44.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.152+0000 7f2e98786640 1 -- 172.21.15.171:0/1235792681 wait complete. 2024-09-16T04:09:44.164 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:44.164 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep osd.12 2024-09-16T04:09:44.165 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 123 2024-09-16T04:09:44.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.238+0000 7f87f5eac640 10 monclient: get_monmap_and_config 2024-09-16T04:09:44.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.238+0000 7f87f5eac640 10 monclient: build_initial_monmap 2024-09-16T04:09:44.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.238+0000 7f87f5eac640 10 monclient: monmap: 2024-09-16T04:09:44.240 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:44.239475+0000 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:44.239475+0000 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:44.241 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 1 Processor -- start 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 1 -- start start 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient: init 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient(hunting): picked mon.noname-c con 0x7f87f0130930 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient(hunting): picked mon.noname-a con 0x7f87f009eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient(hunting): picked mon.noname-b con 0x7f87f0130d10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f01343a0 con 0x7f87f009eb60 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f009c1b0 con 0x7f87f0130930 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f009d8e0 con 0x7f87f0130d10 2024-09-16T04:09:44.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.239+0000 7f87f5eac640 10 monclient(hunting): authenticate will time out at 959.744989s 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ef7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f87f0130930 0x7f87f0133580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33128/0 (socket says 172.21.15.171:33128) 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ef7fe640 1 -- 172.21.15.171:0/3621833562 learned_addr learned my addr 172.21.15.171:0/3621833562 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87eeffd640 1 -- 172.21.15.171:0/3621833562 >> v1:172.21.15.148:6789/0 conn(0x7f87f009eb60 legacy=0x7f87f0133c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599811576 0 0) 0x7f87f009c1b0 con 0x7f87f0130930 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient(hunting): my global_id is 5410 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87d8003520 con 0x7f87f0130930 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2002009934 0 0) 0x7f87d8003520 con 0x7f87f0130930 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 >> v1:172.21.15.148:6790/0 conn(0x7f87f0130d10 legacy=0x7f87f013c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 >> v1:172.21.15.148:6789/0 conn(0x7f87f009eb60 legacy=0x7f87f0133c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:44.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f009dbc0 con 0x7f87f0130930 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87f5eac640 5 monclient: authenticate success, global_id 5410 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f87e0001610 con 0x7f87f0130930 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87f5eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:44.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: dump: 2024-09-16T04:09:44.252 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.252 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.252 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87f5eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87e0001900 con 0x7f87f0130930 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:44.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 1 -- 172.21.15.171:0/3621833562 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f87e0001cd0 con 0x7f87f0130930 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87f5eac640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87f5eac640 4 set_mon_vals no callback set 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.240+0000 7f87ee7fc640 10 monclient: dump: 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.254 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 10 monclient: shutdown 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 1 -- 172.21.15.171:0/3621833562 >> v1:172.21.15.171:6789/0 conn(0x7f87f0130930 legacy=0x7f87f0133580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 1 -- 172.21.15.171:0/3621833562 shutdown_connections 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 1 -- 172.21.15.171:0/3621833562 >> 172.21.15.171:0/3621833562 conn(0x7f87f00a52a0 msgr2=0x7f87f00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 1 -- 172.21.15.171:0/3621833562 shutdown_connections 2024-09-16T04:09:44.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 1 -- 172.21.15.171:0/3621833562 wait complete. 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 10 monclient: build_initial_monmap 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.241+0000 7f87f5eac640 10 monclient: monmap: 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 1 Processor -- start 2024-09-16T04:09:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 1 -- start start 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient: init 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient(hunting): picked mon.noname-b con 0x7f87f009eb60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient(hunting): picked mon.noname-a con 0x7f87f0130930 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient(hunting): picked mon.noname-c con 0x7f87f0130d10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f0123e30 con 0x7f87f0130930 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f00a4590 con 0x7f87f009eb60 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f00727e0 con 0x7f87f0130d10 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87f5eac640 10 monclient(hunting): authenticate will time out at 959.747787s 2024-09-16T04:09:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87ef7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f87f009eb60 0x7f87f01cb530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33138/0 (socket says 172.21.15.171:33138) 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87ef7fe640 1 -- 172.21.15.171:0/2615023097 learned_addr learned my addr 172.21.15.171:0/2615023097 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.242+0000 7f87eeffd640 1 -- 172.21.15.171:0/2615023097 >> v1:172.21.15.148:6789/0 conn(0x7f87f0130930 legacy=0x7f87f012f6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070727292 0 0) 0x7f87f00a4590 con 0x7f87f009eb60 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): my global_id is 5413 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87dc003520 con 0x7f87f009eb60 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1069455472 0 0) 0x7f87dc003520 con 0x7f87f009eb60 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87f00a4590 con 0x7f87f009eb60 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f87e00015d0 con 0x7f87f009eb60 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): dump: 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668861020 0 0) 0x7f87f00727e0 con 0x7f87f0130d10 2024-09-16T04:09:44.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient(hunting): my global_id is 5444 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87dc003520 con 0x7f87f0130d10 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4129688410 0 0) 0x7f87f00a4590 con 0x7f87f009eb60 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 >> v1:172.21.15.148:6790/0 conn(0x7f87f0130d10 legacy=0x7f87f01d3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 >> v1:172.21.15.148:6789/0 conn(0x7f87f0130930 legacy=0x7f87f012f6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 monclient: found mon.b 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f0123b50 con 0x7f87f009eb60 2024-09-16T04:09:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 5 monclient: authenticate success, global_id 5413 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 10 monclient: _renew_subs 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87f0122700 con 0x7f87f009eb60 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 10 monclient: _renew_subs 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87f012ffd0 con 0x7f87f009eb60 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87e0004cf0 con 0x7f87f009eb60 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f87e00050c0 con 0x7f87f009eb60 2024-09-16T04:09:44.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87edffb640 4 set_mon_vals no callback set 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.243+0000 7f87d7fff640 10 monclient: dump: 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.262 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f87e0005260 con 0x7f87f009eb60 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87f5eac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87f5eac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87f5eac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d0005080 con 0x7f87f009eb60 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(111..111 src has 1..111) ==== 5803+0+0 (unknown 4115951566 0 0) 0x7f87e0047760 con 0x7f87f009eb60 2024-09-16T04:09:44.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.244+0000 7f87eeffd640 10 monclient: get_auth_request con 0x7f87dc038fc0 auth_method 0 2024-09-16T04:09:44.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.249+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f87e0015e20 con 0x7f87f009eb60 2024-09-16T04:09:44.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.249+0000 7f87d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:44.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.249+0000 7f87d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87f5eac640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87f5eac640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87f5eac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f87d0005b80 con 0x7f87f009eb60 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) ==== 56+0+1449 (unknown 610832863 0 3821503211) 0x7f87e001e2d0 con 0x7f87f009eb60 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.419+0000 7f87d7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:44.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87f5eac640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:44.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87f5eac640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87f5eac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f87d0005d50 con 0x7f87f009eb60 2024-09-16T04:09:44.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87d7fff640 1 -- 172.21.15.171:0/2615023097 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) ==== 56+0+1449 (unknown 610832863 0 3821503211) 0x7f87e0044d00 con 0x7f87f009eb60 2024-09-16T04:09:44.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87d7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.522+0000 7f87d7fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:44.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 >> v1:172.21.15.171:6812/7327273 conn(0x7f87dc038fc0 legacy=0x7f87dc03b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 10 monclient: shutdown 2024-09-16T04:09:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 >> v1:172.21.15.171:6789/0 conn(0x7f87f009eb60 legacy=0x7f87f01cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 shutdown_connections 2024-09-16T04:09:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 >> 172.21.15.171:0/2615023097 conn(0x7f87f00a52a0 msgr2=0x7f87f009f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 shutdown_connections 2024-09-16T04:09:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.524+0000 7f87f5eac640 1 -- 172.21.15.171:0/2615023097 wait complete. 2024-09-16T04:09:44.535 INFO:tasks.workunit.client.0.smithi171.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-16T04:09:44.536 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:44.536 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep osd.14 2024-09-16T04:09:44.536 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 123 2024-09-16T04:09:44.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.611+0000 7fd996e3c640 10 monclient: get_monmap_and_config 2024-09-16T04:09:44.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.611+0000 7fd996e3c640 10 monclient: build_initial_monmap 2024-09-16T04:09:44.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.611+0000 7fd996e3c640 10 monclient: monmap: 2024-09-16T04:09:44.613 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:44.613 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:44.612414+0000 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:44.612414+0000 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:44.614 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 1 Processor -- start 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 1 -- start start 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient: init 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient(hunting): picked mon.noname-a con 0x7fd99009fd70 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient(hunting): picked mon.noname-c con 0x7fd9901379b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:44.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient(hunting): picked mon.noname-b con 0x7fd99013b4c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99009ea60 con 0x7fd99009fd70 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9900a0d70 con 0x7fd9901379b0 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9900a1050 con 0x7fd99013b4c0 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd996e3c640 10 monclient(hunting): authenticate will time out at 960.117894s 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.612+0000 7fd994bd8640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd99009fd70 legacy=0x7fd9900a0150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd987fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd9901379b0 0x7fd990139da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33146/0 (socket says 172.21.15.171:33146) 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd987fff640 1 -- 172.21.15.171:0/1364756332 learned_addr learned my addr 172.21.15.171:0/1364756332 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957816586 0 0) 0x7fd9900a0d70 con 0x7fd9901379b0 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): my global_id is 5416 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd978003520 con 0x7fd9901379b0 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2875421043 0 0) 0x7fd9900a1050 con 0x7fd99013b4c0 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient(hunting): my global_id is 5447 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9900a0d70 con 0x7fd99013b4c0 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2375806580 0 0) 0x7fd978003520 con 0x7fd9901379b0 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 >> v1:172.21.15.148:6790/0 conn(0x7fd99013b4c0 legacy=0x7fd99013d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 >> v1:172.21.15.148:6789/0 conn(0x7fd99009fd70 legacy=0x7fd9900a0150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:44.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99009fb00 con 0x7fd9901379b0 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd980001610 con 0x7fd9901379b0 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: dump: 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.627 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd996e3c640 5 monclient: authenticate success, global_id 5416 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd996e3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd980001930 con 0x7fd9901379b0 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:44.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 1 -- 172.21.15.171:0/1364756332 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd980001d00 con 0x7fd9901379b0 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd996e3c640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd996e3c640 4 set_mon_vals no callback set 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.613+0000 7fd9877fe640 10 monclient: dump: 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.629 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.630 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 10 monclient: shutdown 2024-09-16T04:09:44.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:44.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 1 -- 172.21.15.171:0/1364756332 >> v1:172.21.15.171:6789/0 conn(0x7fd9901379b0 legacy=0x7fd990139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 1 -- 172.21.15.171:0/1364756332 shutdown_connections 2024-09-16T04:09:44.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 1 -- 172.21.15.171:0/1364756332 >> 172.21.15.171:0/1364756332 conn(0x7fd9900a52a0 msgr2=0x7fd9900a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 1 -- 172.21.15.171:0/1364756332 shutdown_connections 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 1 -- 172.21.15.171:0/1364756332 wait complete. 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 10 monclient: build_initial_monmap 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.614+0000 7fd996e3c640 10 monclient: monmap: 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:44.631 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.632 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:44.632 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:44.632 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:44.632 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.632 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 1 Processor -- start 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 1 -- start start 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient: init 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient(hunting): picked mon.noname-a con 0x7fd99009fd70 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient(hunting): picked mon.noname-c con 0x7fd9901379b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient(hunting): picked mon.noname-b con 0x7fd99013b4c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99011fed0 con 0x7fd99009fd70 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9901ca460 con 0x7fd99013b4c0 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9900a47c0 con 0x7fd9901379b0 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd996e3c640 10 monclient(hunting): authenticate will time out at 960.120721s 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd994bd8640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fd99009fd70 legacy=0x7fd9901c9640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:44.633 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9953d9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd99013b4c0 0x7fd9901d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33158/0 (socket says 172.21.15.171:33158) 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9953d9640 1 -- 172.21.15.171:0/1485534208 learned_addr learned my addr 172.21.15.171:0/1485534208 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598693507 0 0) 0x7fd9901ca460 con 0x7fd99013b4c0 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9857fa640 10 monclient(hunting): my global_id is 5419 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.615+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd97c003520 con 0x7fd99013b4c0 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515070263 0 0) 0x7fd9900a47c0 con 0x7fd9901379b0 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): my global_id is 5450 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9901ca460 con 0x7fd9901379b0 2024-09-16T04:09:44.634 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2213537211 0 0) 0x7fd97c003520 con 0x7fd99013b4c0 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9900a47c0 con 0x7fd99013b4c0 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd988001c30 con 0x7fd99013b4c0 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): dump: 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.635 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:44.636 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2946777511 0 0) 0x7fd9900a47c0 con 0x7fd99013b4c0 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 >> v1:172.21.15.148:6790/0 conn(0x7fd9901379b0 legacy=0x7fd9901c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 >> v1:172.21.15.148:6789/0 conn(0x7fd99009fd70 legacy=0x7fd9901c9640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 monclient: found mon.b 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd990138090 con 0x7fd99013b4c0 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 5 monclient: authenticate success, global_id 5419 2024-09-16T04:09:44.637 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 10 monclient: _renew_subs 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd99006cea0 con 0x7fd99013b4c0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 10 monclient: _renew_subs 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd990121600 con 0x7fd99013b4c0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd988001fe0 con 0x7fd99013b4c0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd988003de0 con 0x7fd99013b4c0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd986ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:44.638 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.616+0000 7fd9857fa640 10 monclient: dump: 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd988003f80 con 0x7fd99013b4c0 2024-09-16T04:09:44.639 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(111..111 src has 1..111) ==== 5803+0+0 (unknown 4115951566 0 0) 0x7fd9880479f0 con 0x7fd99013b4c0 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd996e3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd996e3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd996e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd970005080 con 0x7fd99013b4c0 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.617+0000 7fd994bd8640 10 monclient: get_auth_request con 0x7fd97c038bf0 auth_method 0 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.622+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd988016050 con 0x7fd99013b4c0 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.622+0000 7fd9857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:44.640 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.622+0000 7fd9857fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:44.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.786+0000 7fd996e3c640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:44.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.786+0000 7fd996e3c640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.786+0000 7fd996e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.786+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fd970005b80 con 0x7fd99013b4c0 2024-09-16T04:09:44.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.787+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) ==== 56+0+1449 (unknown 610832863 0 3821503211) 0x7fd988015950 con 0x7fd99013b4c0 2024-09-16T04:09:44.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.787+0000 7fd9857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.787+0000 7fd9857fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:44.887 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd996e3c640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:44.887 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd996e3c640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.887 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd996e3c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.887 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7fd970006730 con 0x7fd99013b4c0 2024-09-16T04:09:44.887 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd9857fa640 1 -- 172.21.15.171:0/1485534208 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v111) ==== 56+0+1449 (unknown 610832863 0 3821503211) 0x7fd98801e500 con 0x7fd99013b4c0 2024-09-16T04:09:44.887 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd9857fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:44.888 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.885+0000 7fd9857fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 >> v1:172.21.15.171:6812/7327273 conn(0x7fd97c038bf0 legacy=0x7fd97c03b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 10 monclient: shutdown 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 >> v1:172.21.15.171:6789/0 conn(0x7fd99013b4c0 legacy=0x7fd9901d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 shutdown_connections 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 >> 172.21.15.171:0/1485534208 conn(0x7fd9900a52a0 msgr2=0x7fd990138fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:44.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 shutdown_connections 2024-09-16T04:09:44.890 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.887+0000 7fd996e3c640 1 -- 172.21.15.171:0/1485534208 wait complete. 2024-09-16T04:09:44.898 INFO:tasks.workunit.client.0.smithi171.stdout: 14 123.00000 osd.14 down 1.00000 1.00000 2024-09-16T04:09:44.898 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.977+0000 7f41d8fe8640 10 monclient: get_monmap_and_config 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.977+0000 7f41d8fe8640 10 monclient: build_initial_monmap 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.977+0000 7f41d8fe8640 10 monclient: monmap: 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:44.977951+0000 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:44.977951+0000 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:44.979 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:44.980 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.977+0000 7f41d8fe8640 1 Processor -- start 2024-09-16T04:09:44.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.977+0000 7f41d8fe8640 1 -- start start 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.977+0000 7f41d8fe8640 10 monclient: init 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient(hunting): picked mon.noname-c con 0x7f41d4136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient(hunting): picked mon.noname-a con 0x7f41d4139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient(hunting): picked mon.noname-b con 0x7f41d413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d40a1340 con 0x7f41d4139b10 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d4137650 con 0x7f41d4136b60 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d40a0aa0 con 0x7f41d413d620 2024-09-16T04:09:44.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d8fe8640 10 monclient(hunting): authenticate will time out at 960.483482s 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f41d4136b60 0x7f41d4136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33162/0 (socket says 172.21.15.171:33162) 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d259c640 1 -- 172.21.15.171:0/2740092599 learned_addr learned my addr 172.21.15.171:0/2740092599 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d1d9b640 1 -- 172.21.15.171:0/2740092599 >> v1:172.21.15.148:6789/0 conn(0x7f41d4139b10 legacy=0x7f41d413bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585808621 0 0) 0x7f41d4137650 con 0x7f41d4136b60 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): my global_id is 5422 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41bc003520 con 0x7f41d4136b60 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012908637 0 0) 0x7f41d40a0aa0 con 0x7f41d413d620 2024-09-16T04:09:44.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 10 monclient(hunting): my global_id is 5453 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.978+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41d4137650 con 0x7f41d413d620 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1147703272 0 0) 0x7f41bc003520 con 0x7f41d4136b60 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 >> v1:172.21.15.148:6790/0 conn(0x7f41d413d620 legacy=0x7f41d413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 >> v1:172.21.15.148:6789/0 conn(0x7f41d4139b10 legacy=0x7f41d413bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: _finish_hunting 0 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 monclient: found mon.noname-c 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d40a0550 con 0x7f41d4136b60 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: _finish_auth 0 2024-09-16T04:09:44.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41c4001990 con 0x7f41d4136b60 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 5 monclient: authenticate success, global_id 5422 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: dump: 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.991 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41c4001cb0 con 0x7f41d4136b60 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 1 -- 172.21.15.171:0/2740092599 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41c4002080 con 0x7f41d4136b60 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d159a640 10 monclient: dump: 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.992 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 4 set_mon_vals no callback set 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 10 monclient: shutdown 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2740092599 >> v1:172.21.15.171:6789/0 conn(0x7f41d4136b60 legacy=0x7f41d4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2740092599 shutdown_connections 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2740092599 >> 172.21.15.171:0/2740092599 conn(0x7f41d40a98b0 msgr2=0x7f41d40a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2740092599 shutdown_connections 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.979+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2740092599 wait complete. 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.980+0000 7f41d8fe8640 10 monclient: build_initial_monmap 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.980+0000 7f41d8fe8640 10 monclient: monmap: 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:44.994 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.980+0000 7f41d8fe8640 1 Processor -- start 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.980+0000 7f41d8fe8640 1 -- start start 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.980+0000 7f41d8fe8640 10 monclient: init 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient(hunting): picked mon.noname-a con 0x7f41d4136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:44.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient(hunting): picked mon.noname-c con 0x7f41d4139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient(hunting): picked mon.noname-b con 0x7f41d413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d409b080 con 0x7f41d4136b60 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d40a6f50 con 0x7f41d413d620 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d40a0240 con 0x7f41d4139b10 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient(hunting): authenticate will time out at 960.486297s 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d259c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f41d4136b60 legacy=0x7f41d4139ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d2d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f41d413d620 0x7f41d413ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33172/0 (socket says 172.21.15.171:33172) 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d2d9d640 1 -- 172.21.15.171:0/2290889181 learned_addr learned my addr 172.21.15.171:0/2290889181 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675944464 0 0) 0x7f41d40a6f50 con 0x7f41d413d620 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): my global_id is 5425 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41b4003520 con 0x7f41d413d620 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1897113087 0 0) 0x7f41b4003520 con 0x7f41d413d620 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41d40a6f50 con 0x7f41d413d620 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41cc0018d0 con 0x7f41d413d620 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:44.997 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592447557 0 0) 0x7f41d40a0240 con 0x7f41d4139b10 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient(hunting): my global_id is 5456 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41b4003520 con 0x7f41d4139b10 2024-09-16T04:09:44.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 823912268 0 0) 0x7f41d40a6f50 con 0x7f41d413d620 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 >> v1:172.21.15.148:6790/0 conn(0x7f41d4139b10 legacy=0x7f41d413a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 >> v1:172.21.15.148:6789/0 conn(0x7f41d4136b60 legacy=0x7f41d4139ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 monclient: found mon.b 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d413c720 con 0x7f41d413d620 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 5 monclient: authenticate success, global_id 5425 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient: _renew_subs 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:44.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.981+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f41d409ada0 con 0x7f41d413d620 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 10 monclient: _renew_subs 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f41d41d4800 con 0x7f41d413d620 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41cc001ce0 con 0x7f41d413d620 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41cc003ae0 con 0x7f41d413d620 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d0d99640 4 set_mon_vals no callback set 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 10 monclient: dump: 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:45.000 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f41cc003c80 con 0x7f41d413d620 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.982+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b8005080 con 0x7f41d413d620 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.983+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(111..111 src has 1..111) ==== 5803+0+0 (unknown 4115951566 0 0) 0x7f41cc0476c0 con 0x7f41d413d620 2024-09-16T04:09:45.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.983+0000 7f41d259c640 10 monclient: get_auth_request con 0x7f41b4039070 auth_method 0 2024-09-16T04:09:45.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.987+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f41cc015cc0 con 0x7f41d413d620 2024-09-16T04:09:45.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.987+0000 7f41c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:45.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:44.987+0000 7f41c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:45.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:45.159+0000 7f41d8fe8640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T04:09:45.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:45.159+0000 7f41d8fe8640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T04:09:45.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:45.159+0000 7f41d8fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:45.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:45.159+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) -- 0x7f41b8005370 con 0x7f41d413d620 2024-09-16T04:09:46.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.060+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v112) ==== 168+0+0 (unknown 435931192 0 0) 0x7f41cc0155c0 con 0x7f41d413d620 2024-09-16T04:09:46.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.060+0000 7f41c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T04:09:46.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.060+0000 7f41c2ffd640 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T04:09:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.164+0000 7f41d8fe8640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T04:09:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.164+0000 7f41d8fe8640 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T04:09:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.164+0000 7f41d8fe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:46.164+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) -- 0x7f41b8005ab0 con 0x7f41d413d620 2024-09-16T04:09:47.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.061+0000 7f41c2ffd640 1 -- 172.21.15.171:0/2290889181 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v113) ==== 168+0+0 (unknown 3273076486 0 0) 0x7f41cc019910 con 0x7f41d413d620 2024-09-16T04:09:47.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.061+0000 7f41c2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-16T04:09:47.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.061+0000 7f41c2ffd640 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T04:09:47.063 INFO:tasks.workunit.client.0.smithi171.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-16T04:09:47.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 >> v1:172.21.15.171:6812/7327273 conn(0x7f41b4039070 legacy=0x7f41b403b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 10 monclient: shutdown 2024-09-16T04:09:47.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:47.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 >> v1:172.21.15.171:6789/0 conn(0x7f41d413d620 legacy=0x7f41d413ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 shutdown_connections 2024-09-16T04:09:47.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 >> 172.21.15.171:0/2290889181 conn(0x7f41d40a98b0 msgr2=0x7f41d40a75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:47.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 shutdown_connections 2024-09-16T04:09:47.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.064+0000 7f41d8fe8640 1 -- 172.21.15.171:0/2290889181 wait complete. 2024-09-16T04:09:47.078 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-16T04:09:47.128 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:47.129 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.158+0000 7f28ca90a640 10 monclient: get_monmap_and_config 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.158+0000 7f28ca90a640 10 monclient: build_initial_monmap 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.158+0000 7f28ca90a640 10 monclient: monmap: 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:47.158944+0000 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:47.158944+0000 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:47.160 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:47.161 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 1 Processor -- start 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 1 -- start start 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient: init 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient(hunting): picked mon.noname-c con 0x7f28c4136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient(hunting): picked mon.noname-a con 0x7f28c4139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:47.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient(hunting): picked mon.noname-b con 0x7f28c413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c40a1340 con 0x7f28c4139b10 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c4137650 con 0x7f28c4136b60 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c40a0aa0 con 0x7f28c413d620 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28ca90a640 10 monclient(hunting): authenticate will time out at 962.664695s 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28c3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f28c4136b60 0x7f28c4136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33174/0 (socket says 172.21.15.171:33174) 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28c3fff640 1 -- 172.21.15.171:0/456459771 learned_addr learned my addr 172.21.15.171:0/456459771 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28bbfff640 1 -- 172.21.15.171:0/456459771 >> v1:172.21.15.148:6789/0 conn(0x7f28c4139b10 legacy=0x7f28c413bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.159+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590298215 0 0) 0x7f28c4137650 con 0x7f28c4136b60 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient(hunting): my global_id is 5428 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28b0003520 con 0x7f28c4136b60 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1103300406 0 0) 0x7f28b0003520 con 0x7f28c4136b60 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 >> v1:172.21.15.148:6790/0 conn(0x7f28c413d620 legacy=0x7f28c413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 >> v1:172.21.15.148:6789/0 conn(0x7f28c4139b10 legacy=0x7f28c413bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c40a0550 con 0x7f28c4136b60 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28b4001610 con 0x7f28c4136b60 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 5 monclient: authenticate success, global_id 5428 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: dump: 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.170 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28b4001970 con 0x7f28c4136b60 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 1 -- 172.21.15.171:0/456459771 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28b40048a0 con 0x7f28c4136b60 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 4 set_mon_vals no callback set 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28c37fe640 10 monclient: dump: 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.171 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 10 monclient: shutdown 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 1 -- 172.21.15.171:0/456459771 >> v1:172.21.15.171:6789/0 conn(0x7f28c4136b60 legacy=0x7f28c4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 1 -- 172.21.15.171:0/456459771 shutdown_connections 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 1 -- 172.21.15.171:0/456459771 >> 172.21.15.171:0/456459771 conn(0x7f28c40a98b0 msgr2=0x7f28c40a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 1 -- 172.21.15.171:0/456459771 shutdown_connections 2024-09-16T04:09:47.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.160+0000 7f28ca90a640 1 -- 172.21.15.171:0/456459771 wait complete. 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.161+0000 7f28ca90a640 10 monclient: build_initial_monmap 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.161+0000 7f28ca90a640 10 monclient: monmap: 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.161+0000 7f28ca90a640 1 Processor -- start 2024-09-16T04:09:47.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.161+0000 7f28ca90a640 1 -- start start 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.161+0000 7f28ca90a640 10 monclient: init 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient(hunting): picked mon.noname-b con 0x7f28c4136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient(hunting): picked mon.noname-c con 0x7f28c4139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient(hunting): picked mon.noname-a con 0x7f28c413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c409f4e0 con 0x7f28c413d620 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c4120ec0 con 0x7f28c4136b60 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c40a70c0 con 0x7f28c4139b10 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient(hunting): authenticate will time out at 962.667237s 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f28c4136b60 0x7f28c41cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33182/0 (socket says 172.21.15.171:33182) 2024-09-16T04:09:47.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c3fff640 1 -- 172.21.15.171:0/1149071657 learned_addr learned my addr 172.21.15.171:0/1149071657 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c8ea7640 1 -- 172.21.15.171:0/1149071657 >> v1:172.21.15.148:6789/0 conn(0x7f28c413d620 legacy=0x7f28c41d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368597189 0 0) 0x7f28c4120ec0 con 0x7f28c4136b60 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): my global_id is 5431 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28a4003550 con 0x7f28c4136b60 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431522725 0 0) 0x7f28c40a70c0 con 0x7f28c4139b10 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): my global_id is 5462 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28c4120ec0 con 0x7f28c4139b10 2024-09-16T04:09:47.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2723586952 0 0) 0x7f28a4003550 con 0x7f28c4136b60 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28c40a70c0 con 0x7f28c4136b60 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28b4001430 con 0x7f28c4136b60 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): dump: 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3486516467 0 0) 0x7f28c40a70c0 con 0x7f28c4136b60 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 >> v1:172.21.15.148:6790/0 conn(0x7f28c4139b10 legacy=0x7f28c41d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 >> v1:172.21.15.148:6789/0 conn(0x7f28c413d620 legacy=0x7f28c41d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 monclient: found mon.b 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c40a2b40 con 0x7f28c4136b60 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:47.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 5 monclient: authenticate success, global_id 5431 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient: _renew_subs 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28c4120900 con 0x7f28c4136b60 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient: _renew_subs 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.162+0000 7f28ca90a640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f28c41212e0 con 0x7f28c4136b60 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28b40016e0 con 0x7f28c4136b60 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f28b4005000 con 0x7f28c4136b60 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c2ffd640 4 set_mon_vals no callback set 2024-09-16T04:09:47.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 10 monclient: dump: 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f28b4010430 con 0x7f28c4136b60 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28ca90a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28ca90a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28ca90a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.163+0000 7f28ca90a640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a8005080 con 0x7f28c4136b60 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.164+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(113..113 src has 1..113) ==== 5803+0+0 (unknown 2006538005 0 0) 0x7f28b4047710 con 0x7f28c4136b60 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.164+0000 7f28bbfff640 10 monclient: get_auth_request con 0x7f28a4038c10 auth_method 0 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.168+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f28b4015dd0 con 0x7f28c4136b60 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.168+0000 7f28c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.168+0000 7f28c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:47.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28ca90a640 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28ca90a640 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28ca90a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28ca90a640 1 -- 172.21.15.171:0/1149071657 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) -- 0x7f28a8005370 con 0x7f28c4136b60 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28c17fa640 1 -- 172.21.15.171:0/1149071657 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v113) ==== 154+0+0 (unknown 2757362208 0 0) 0x7f28b40156d0 con 0x7f28c4136b60 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.340+0000 7f28c17fa640 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-16T04:09:47.343 INFO:tasks.workunit.client.0.smithi171.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-16T04:09:47.346 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:47.347 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:47.347 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep osd.12 2024-09-16T04:09:47.347 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 155 2024-09-16T04:09:47.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.421+0000 7f78c2b81640 10 monclient: get_monmap_and_config 2024-09-16T04:09:47.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.421+0000 7f78c2b81640 10 monclient: build_initial_monmap 2024-09-16T04:09:47.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.421+0000 7f78c2b81640 10 monclient: monmap: 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:47.422189+0000 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:47.422189+0000 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:47.424 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:47.425 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:47.425 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 1 Processor -- start 2024-09-16T04:09:47.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 1 -- start start 2024-09-16T04:09:47.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient: init 2024-09-16T04:09:47.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:47.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:47.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient(hunting): picked mon.noname-b con 0x7f78bc09b680 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient(hunting): picked mon.noname-a con 0x7f78bc09ba60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient(hunting): picked mon.noname-c con 0x7f78bc09be80 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc0998b0 con 0x7f78bc09ba60 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc099b90 con 0x7f78bc09be80 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc099e70 con 0x7f78bc09b680 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c2b81640 10 monclient(hunting): authenticate will time out at 962.927800s 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78bbfff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f78bc09ba60 legacy=0x7f78bc13bc20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c111e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78bc09be80 0x7f78bc13f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52452/0 (socket says 172.21.15.171:52452) 2024-09-16T04:09:47.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.422+0000 7f78c111e640 1 -- 172.21.15.171:0/564186438 learned_addr learned my addr 172.21.15.171:0/564186438 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773287290 0 0) 0x7f78bc099b90 con 0x7f78bc09be80 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient(hunting): my global_id is 5434 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a8003520 con 0x7f78bc09be80 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3580926758 0 0) 0x7f78a8003520 con 0x7f78bc09be80 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 >> v1:172.21.15.148:6790/0 conn(0x7f78bc09b680 legacy=0x7f78bc136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 >> v1:172.21.15.148:6789/0 conn(0x7f78bc09ba60 legacy=0x7f78bc13bc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc068840 con 0x7f78bc09be80 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a4001610 con 0x7f78bc09be80 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 5 monclient: authenticate success, global_id 5434 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: dump: 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.435 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78a4001970 con 0x7f78bc09be80 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 1 -- 172.21.15.171:0/564186438 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78a4001d40 con 0x7f78bc09be80 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 4 set_mon_vals no callback set 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:47.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78bb7fe640 10 monclient: dump: 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 10 monclient: shutdown 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.423+0000 7f78c2b81640 1 -- 172.21.15.171:0/564186438 >> v1:172.21.15.171:6789/0 conn(0x7f78bc09be80 legacy=0x7f78bc13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 1 -- 172.21.15.171:0/564186438 shutdown_connections 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 1 -- 172.21.15.171:0/564186438 >> 172.21.15.171:0/564186438 conn(0x7f78bc0a1e90 msgr2=0x7f78bc0a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 1 -- 172.21.15.171:0/564186438 shutdown_connections 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 1 -- 172.21.15.171:0/564186438 wait complete. 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 10 monclient: build_initial_monmap 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 10 monclient: monmap: 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:47.438 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 1 Processor -- start 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 1 -- start start 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.424+0000 7f78c2b81640 10 monclient: init 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient(hunting): picked mon.noname-b con 0x7f78bc09b680 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient(hunting): picked mon.noname-a con 0x7f78bc09ba60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient(hunting): picked mon.noname-c con 0x7f78bc09be80 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc08e6f0 con 0x7f78bc09ba60 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc08e9d0 con 0x7f78bc09b680 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc09c580 con 0x7f78bc09be80 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c2b81640 10 monclient(hunting): authenticate will time out at 962.930600s 2024-09-16T04:09:47.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c091d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f78bc09b680 0x7f78bc08bfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52454/0 (socket says 172.21.15.171:52454) 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78c091d640 1 -- 172.21.15.171:0/993447927 learned_addr learned my addr 172.21.15.171:0/993447927 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78bbfff640 1 -- 172.21.15.171:0/993447927 >> v1:172.21.15.148:6789/0 conn(0x7f78bc09ba60 legacy=0x7f78bc08c6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 354893401 0 0) 0x7f78bc08e9d0 con 0x7f78bc09b680 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): my global_id is 5437 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f789c003520 con 0x7f78bc09b680 2024-09-16T04:09:47.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732198366 0 0) 0x7f78bc09c580 con 0x7f78bc09be80 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 10 monclient(hunting): my global_id is 5468 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.425+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78bc08e9d0 con 0x7f78bc09be80 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1918436675 0 0) 0x7f789c003520 con 0x7f78bc09b680 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78bc09c580 con 0x7f78bc09b680 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78b40018d0 con 0x7f78bc09b680 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient(hunting): dump: 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.441 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1186307997 0 0) 0x7f78bc09c580 con 0x7f78bc09b680 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 >> v1:172.21.15.148:6790/0 conn(0x7f78bc09be80 legacy=0x7f78bc1d3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 >> v1:172.21.15.148:6789/0 conn(0x7f78bc09ba60 legacy=0x7f78bc08c6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 monclient: found mon.b 2024-09-16T04:09:47.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc08d240 con 0x7f78bc09b680 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 5 monclient: authenticate success, global_id 5437 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 10 monclient: _renew_subs 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78bc08cdd0 con 0x7f78bc09b680 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 10 monclient: _renew_subs 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78bc0970b0 con 0x7f78bc09b680 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78b4003610 con 0x7f78bc09b680 2024-09-16T04:09:47.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f78b4003960 con 0x7f78bc09b680 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78baffd640 4 set_mon_vals no callback set 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.426+0000 7f78b97fa640 10 monclient: dump: 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.444 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f78b4003b00 con 0x7f78bc09b680 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(113..113 src has 1..113) ==== 5803+0+0 (unknown 2006538005 0 0) 0x7f78b40475d0 con 0x7f78bc09b680 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78bbfff640 10 monclient: get_auth_request con 0x7f789c038f90 auth_method 0 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78c2b81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78c2b81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78c2b81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.427+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a0005080 con 0x7f78bc09b680 2024-09-16T04:09:47.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.431+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f78b4015c30 con 0x7f78bc09b680 2024-09-16T04:09:47.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.431+0000 7f78b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.431+0000 7f78b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:47.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.597+0000 7f78c2b81640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:47.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.597+0000 7f78c2b81640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:47.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.597+0000 7f78c2b81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.597+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f78a0005b80 con 0x7f78bc09b680 2024-09-16T04:09:47.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.598+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) ==== 56+0+1449 (unknown 1143467041 0 2496839416) 0x7f78b4015530 con 0x7f78bc09b680 2024-09-16T04:09:47.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.598+0000 7f78b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:47.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.598+0000 7f78b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:47.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.698+0000 7f78c2b81640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:47.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.698+0000 7f78c2b81640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:47.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.698+0000 7f78c2b81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.698+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f78a0006730 con 0x7f78bc09b680 2024-09-16T04:09:47.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.699+0000 7f78b97fa640 1 -- 172.21.15.171:0/993447927 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) ==== 56+0+1449 (unknown 1143467041 0 2496839416) 0x7f78b401e0e0 con 0x7f78bc09b680 2024-09-16T04:09:47.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.699+0000 7f78b97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:47.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.699+0000 7f78b97fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 >> v1:172.21.15.171:6812/7327273 conn(0x7f789c038f90 legacy=0x7f789c03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 10 monclient: shutdown 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 >> v1:172.21.15.171:6789/0 conn(0x7f78bc09b680 legacy=0x7f78bc08bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 shutdown_connections 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 >> 172.21.15.171:0/993447927 conn(0x7f78bc0a1e90 msgr2=0x7f78bc0987b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 shutdown_connections 2024-09-16T04:09:47.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.701+0000 7f78c2b81640 1 -- 172.21.15.171:0/993447927 wait complete. 2024-09-16T04:09:47.712 INFO:tasks.workunit.client.0.smithi171.stdout: 12 155.00000 osd.12 down 1.00000 1.00000 2024-09-16T04:09:47.712 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd tree 2024-09-16T04:09:47.713 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep osd.14 2024-09-16T04:09:47.713 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 155 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.787+0000 7f38647f6640 10 monclient: get_monmap_and_config 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.787+0000 7f38647f6640 10 monclient: build_initial_monmap 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.787+0000 7f38647f6640 10 monclient: monmap: 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:47.788114+0000 2024-09-16T04:09:47.789 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:47.788114+0000 2024-09-16T04:09:47.790 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:47.790 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.790 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:47.790 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:47.790 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:47.790 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.787+0000 7f38647f6640 1 Processor -- start 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.787+0000 7f38647f6640 1 -- start start 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.787+0000 7f38647f6640 10 monclient: init 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient(hunting): picked mon.noname-a con 0x7f385c130930 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient(hunting): picked mon.noname-b con 0x7f385c09eb60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient(hunting): picked mon.noname-c con 0x7f385c130d10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:47.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c1343a0 con 0x7f385c130930 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c09c1b0 con 0x7f385c130d10 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c09d8e0 con 0x7f385c09eb60 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f38647f6640 10 monclient(hunting): authenticate will time out at 963.293673s 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f3862592640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f385c130930 legacy=0x7f385c133580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f3862d93640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f385c130d10 0x7f385c13c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52470/0 (socket says 172.21.15.171:52470) 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f3862d93640 1 -- 172.21.15.171:0/1858702187 learned_addr learned my addr 172.21.15.171:0/1858702187 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.788+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343697379 0 0) 0x7f385c09c1b0 con 0x7f385c130d10 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient(hunting): my global_id is 5440 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3848003520 con 0x7f385c130d10 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3049061447 0 0) 0x7f3848003520 con 0x7f385c130d10 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 >> v1:172.21.15.148:6790/0 conn(0x7f385c09eb60 legacy=0x7f385c133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 >> v1:172.21.15.148:6789/0 conn(0x7f385c130930 legacy=0x7f385c133580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: _finish_hunting 0 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 monclient: found mon.noname-c 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c09dbc0 con 0x7f385c130d10 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: _finish_auth 0 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: _check_auth_tickets 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3854001990 con 0x7f385c130d10 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 5 monclient: authenticate success, global_id 5440 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: dump: 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.801 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3854001c80 con 0x7f385c130d10 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 1 -- 172.21.15.171:0/1858702187 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3854002050 con 0x7f385c130d10 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f3861590640 10 monclient: dump: 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 4 set_mon_vals no callback set 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 10 monclient: shutdown 2024-09-16T04:09:47.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.789+0000 7f38647f6640 1 -- 172.21.15.171:0/1858702187 >> v1:172.21.15.171:6789/0 conn(0x7f385c130d10 legacy=0x7f385c13c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 1 -- 172.21.15.171:0/1858702187 shutdown_connections 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 1 -- 172.21.15.171:0/1858702187 >> 172.21.15.171:0/1858702187 conn(0x7f385c0a52a0 msgr2=0x7f385c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 1 -- 172.21.15.171:0/1858702187 shutdown_connections 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 1 -- 172.21.15.171:0/1858702187 wait complete. 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 10 monclient: build_initial_monmap 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 10 monclient: monmap: 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:47.804 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.790+0000 7f38647f6640 1 Processor -- start 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 1 -- start start 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient: init 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient(hunting): picked mon.noname-a con 0x7f385c09eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient(hunting): picked mon.noname-c con 0x7f385c130930 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient(hunting): picked mon.noname-b con 0x7f385c130d10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:47.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c1cb9d0 con 0x7f385c09eb60 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c06cea0 con 0x7f385c130d10 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c130270 con 0x7f385c130930 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f38647f6640 10 monclient(hunting): authenticate will time out at 963.296499s 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3862592640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f385c09eb60 legacy=0x7f385c12d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3862d93640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f385c130d10 0x7f385c1d3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52480/0 (socket says 172.21.15.171:52480) 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3862d93640 1 -- 172.21.15.171:0/2923879007 learned_addr learned my addr 172.21.15.171:0/2923879007 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384587124 0 0) 0x7f385c06cea0 con 0x7f385c130d10 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): my global_id is 5443 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3840003520 con 0x7f385c130d10 2024-09-16T04:09:47.806 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83318451 0 0) 0x7f385c130270 con 0x7f385c130930 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): my global_id is 5474 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f385c06cea0 con 0x7f385c130930 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1124466213 0 0) 0x7f3840003520 con 0x7f385c130d10 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f385c130270 con 0x7f385c130d10 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f38540017b0 con 0x7f385c130d10 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:47.807 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.791+0000 7f3852ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3373374821 0 0) 0x7f385c06cea0 con 0x7f385c130930 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3840003520 con 0x7f385c130930 2024-09-16T04:09:47.808 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f384c001830 con 0x7f385c130930 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.809 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3175781188 0 0) 0x7f385c130270 con 0x7f385c130d10 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 >> v1:172.21.15.148:6790/0 conn(0x7f385c130930 legacy=0x7f385c1d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 >> v1:172.21.15.148:6789/0 conn(0x7f385c09eb60 legacy=0x7f385c12d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 monclient: found mon.b 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c12fd40 con 0x7f385c130d10 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 5 monclient: authenticate success, global_id 5443 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 10 monclient: _renew_subs 2024-09-16T04:09:47.810 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f385c121c30 con 0x7f385c130d10 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 10 monclient: _renew_subs 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f385c12dbf0 con 0x7f385c130d10 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3854001c80 con 0x7f385c130d10 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f38540051e0 con 0x7f385c130d10 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3860d8f640 4 set_mon_vals no callback set 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.792+0000 7f3852ffd640 10 monclient: dump: 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:47.811 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3854010610 con 0x7f385c130d10 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f38647f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f38647f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f38647f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3844005080 con 0x7f385c130d10 2024-09-16T04:09:47.812 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(113..113 src has 1..113) ==== 5803+0+0 (unknown 2006538005 0 0) 0x7f3854047b00 con 0x7f385c130d10 2024-09-16T04:09:47.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.793+0000 7f3862592640 10 monclient: get_auth_request con 0x7f3840038ec0 auth_method 0 2024-09-16T04:09:47.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.797+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f38540161c0 con 0x7f385c130d10 2024-09-16T04:09:47.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.797+0000 7f3852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:47.813 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.797+0000 7f3852ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:47.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f38647f6640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:47.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f38647f6640 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:47.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f38647f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:47.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f3844005b80 con 0x7f385c130d10 2024-09-16T04:09:47.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) ==== 56+0+1449 (unknown 1143467041 0 2496839416) 0x7f3854015ac0 con 0x7f385c130d10 2024-09-16T04:09:47.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f3852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-16T04:09:47.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:47.965+0000 7f3852ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:48.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f38647f6640 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-16T04:09:48.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f38647f6640 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:48.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f38647f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd tree"} v 0) -- 0x7f3844005d50 con 0x7f385c130d10 2024-09-16T04:09:48.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f3852ffd640 1 -- 172.21.15.171:0/2923879007 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v113) ==== 56+0+1449 (unknown 1143467041 0 2496839416) 0x7f385401e610 con 0x7f385c130d10 2024-09-16T04:09:48.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f3852ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-16T04:09:48.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.065+0000 7f3852ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 >> v1:172.21.15.171:6812/7327273 conn(0x7f3840038ec0 legacy=0x7f384003b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 10 monclient: shutdown 2024-09-16T04:09:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 >> v1:172.21.15.171:6789/0 conn(0x7f385c130d10 legacy=0x7f385c1d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 shutdown_connections 2024-09-16T04:09:48.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 >> 172.21.15.171:0/2923879007 conn(0x7f385c0a52a0 msgr2=0x7f385c1339e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:48.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 shutdown_connections 2024-09-16T04:09:48.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.067+0000 7f38647f6640 1 -- 172.21.15.171:0/2923879007 wait complete. 2024-09-16T04:09:48.078 INFO:tasks.workunit.client.0.smithi171.stdout: 14 155.00000 osd.14 down 1.00000 1.00000 2024-09-16T04:09:48.079 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.12 2024-09-16T04:09:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.155+0000 7f39b2f42640 10 monclient: get_monmap_and_config 2024-09-16T04:09:48.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.155+0000 7f39b2f42640 10 monclient: build_initial_monmap 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.155+0000 7f39b2f42640 10 monclient: monmap: 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:48.156624+0000 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:48.156624+0000 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:48.158 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:48.159 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:48.159 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:48.159 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:48.159 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.156+0000 7f39b2f42640 1 Processor -- start 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.156+0000 7f39b2f42640 1 -- start start 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.156+0000 7f39b2f42640 10 monclient: init 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient(hunting): picked mon.noname-a con 0x7f39ac132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient(hunting): picked mon.noname-c con 0x7f39ac09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient(hunting): picked mon.noname-b con 0x7f39ac132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:48.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac1332b0 con 0x7f39ac132a80 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac134310 con 0x7f39ac09ec20 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac09d980 con 0x7f39ac132e60 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient(hunting): authenticate will time out at 963.662113s 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39abfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f39ac09ec20 0x7f39ac133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52494/0 (socket says 172.21.15.171:52494) 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39abfff640 1 -- 172.21.15.171:0/3832779926 learned_addr learned my addr 172.21.15.171:0/3832779926 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b0cde640 1 -- 172.21.15.171:0/3832779926 >> v1:172.21.15.148:6789/0 conn(0x7f39ac132a80 legacy=0x7f39ac1334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992191000 0 0) 0x7f39ac134310 con 0x7f39ac09ec20 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:48.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient(hunting): my global_id is 5446 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f399c003520 con 0x7f39ac09ec20 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4064050208 0 0) 0x7f399c003520 con 0x7f39ac09ec20 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 >> v1:172.21.15.148:6790/0 conn(0x7f39ac132e60 legacy=0x7f39ac13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 >> v1:172.21.15.148:6789/0 conn(0x7f39ac132a80 legacy=0x7f39ac1334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 monclient: found mon.noname-c 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac09dc60 con 0x7f39ac09ec20 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:48.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3998001610 con 0x7f39ac09ec20 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 5 monclient: authenticate success, global_id 5446 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: dump: 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:48.171 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3998001900 con 0x7f39ac09ec20 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 1 -- 172.21.15.171:0/3832779926 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3998001d10 con 0x7f39ac09ec20 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39b2f42640 4 set_mon_vals no callback set 2024-09-16T04:09:48.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.157+0000 7f39ab7fe640 10 monclient: dump: 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:48.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 10 monclient: shutdown 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 1 -- 172.21.15.171:0/3832779926 >> v1:172.21.15.171:6789/0 conn(0x7f39ac09ec20 legacy=0x7f39ac133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 1 -- 172.21.15.171:0/3832779926 shutdown_connections 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 1 -- 172.21.15.171:0/3832779926 >> 172.21.15.171:0/3832779926 conn(0x7f39ac0a52a0 msgr2=0x7f39ac0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 1 -- 172.21.15.171:0/3832779926 shutdown_connections 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 1 -- 172.21.15.171:0/3832779926 wait complete. 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 10 monclient: build_initial_monmap 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.158+0000 7f39b2f42640 10 monclient: monmap: 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:48.174 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 1 Processor -- start 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 1 -- start start 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient: init 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient(hunting): picked mon.noname-b con 0x7f39ac09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient(hunting): picked mon.noname-a con 0x7f39ac132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient(hunting): picked mon.noname-c con 0x7f39ac132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac1209e0 con 0x7f39ac132a80 2024-09-16T04:09:48.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac11f150 con 0x7f39ac09ec20 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac120cc0 con 0x7f39ac132e60 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b2f42640 10 monclient(hunting): authenticate will time out at 963.664974s 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b0cde640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f39ac09ec20 0x7f39ac1c7320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52502/0 (socket says 172.21.15.171:52502) 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.159+0000 7f39b0cde640 1 -- 172.21.15.171:0/1189288851 learned_addr learned my addr 172.21.15.171:0/1189288851 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39abfff640 1 -- 172.21.15.171:0/1189288851 >> v1:172.21.15.148:6789/0 conn(0x7f39ac132a80 legacy=0x7f39ac1cc2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737694719 0 0) 0x7f39ac11f150 con 0x7f39ac09ec20 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): my global_id is 5449 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3990003520 con 0x7f39ac09ec20 2024-09-16T04:09:48.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 942359956 0 0) 0x7f3990003520 con 0x7f39ac09ec20 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39ac11f150 con 0x7f39ac09ec20 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f39a0004180 con 0x7f39ac09ec20 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): dump: 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:48.177 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406209461 0 0) 0x7f39ac120cc0 con 0x7f39ac132e60 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient(hunting): my global_id is 5480 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3990003520 con 0x7f39ac132e60 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2768826044 0 0) 0x7f39ac11f150 con 0x7f39ac09ec20 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 >> v1:172.21.15.148:6790/0 conn(0x7f39ac132e60 legacy=0x7f39ac1cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 >> v1:172.21.15.148:6789/0 conn(0x7f39ac132a80 legacy=0x7f39ac1cc2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 monclient: found mon.b 2024-09-16T04:09:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac131a30 con 0x7f39ac09ec20 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: _finish_auth 0 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 5 monclient: authenticate success, global_id 5449 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 10 monclient: _renew_subs 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac0a3430 con 0x7f39ac09ec20 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 10 monclient: _renew_subs 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f39ac11e020 con 0x7f39ac09ec20 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f39a00044e0 con 0x7f39ac09ec20 2024-09-16T04:09:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f39a00048b0 con 0x7f39ac09ec20 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39aaffd640 4 set_mon_vals no callback set 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.160+0000 7f39a97fa640 10 monclient: dump: 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:48.180 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f39a0004ad0 con 0x7f39ac09ec20 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39b2f42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39b2f42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39b2f42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398c005080 con 0x7f39ac09ec20 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(113..113 src has 1..113) ==== 5803+0+0 (unknown 2006538005 0 0) 0x7f39a00476a0 con 0x7f39ac09ec20 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.161+0000 7f39abfff640 10 monclient: get_auth_request con 0x7f3990038f90 auth_method 0 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.166+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f39a0015e40 con 0x7f39ac09ec20 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.166+0000 7f39a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:48.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.166+0000 7f39a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:48.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.338+0000 7f39b2f42640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T04:09:48.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.338+0000 7f39b2f42640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T04:09:48.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.338+0000 7f39b2f42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:48.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:48.338+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) -- 0x7f398c005370 con 0x7f39ac09ec20 2024-09-16T04:09:49.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.073+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v114) ==== 125+0+0 (unknown 2612350878 0 0) 0x7f39a0015740 con 0x7f39ac09ec20 2024-09-16T04:09:49.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.073+0000 7f39a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T04:09:49.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.073+0000 7f39a97fa640 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-16T04:09:49.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39b2f42640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T04:09:49.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39b2f42640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T04:09:49.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39b2f42640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) -- 0x7f398c001090 con 0x7f39ac09ec20 2024-09-16T04:09:49.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39a97fa640 1 -- 172.21.15.171:0/1189288851 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v114) ==== 126+0+0 (unknown 204447076 0 0) 0x7f39a001e2f0 con 0x7f39ac09ec20 2024-09-16T04:09:49.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39a97fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-16T04:09:49.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.181+0000 7f39a97fa640 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-16T04:09:49.184 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.12' does not appear in the crush map 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.183+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 >> v1:172.21.15.171:6812/7327273 conn(0x7f3990038f90 legacy=0x7f399003b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 10 monclient: shutdown 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 >> v1:172.21.15.171:6789/0 conn(0x7f39ac09ec20 legacy=0x7f39ac1c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 shutdown_connections 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 >> 172.21.15.171:0/1189288851 conn(0x7f39ac0a52a0 msgr2=0x7f39ac131f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 shutdown_connections 2024-09-16T04:09:49.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.184+0000 7f39b2f42640 1 -- 172.21.15.171:0/1189288851 wait complete. 2024-09-16T04:09:49.195 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm osd.14 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.273+0000 7fdbcf8dd640 10 monclient: get_monmap_and_config 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.273+0000 7fdbcf8dd640 10 monclient: build_initial_monmap 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.273+0000 7fdbcf8dd640 10 monclient: monmap: 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:49.274004+0000 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:49.274004+0000 2024-09-16T04:09:49.275 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:49.276 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.276 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:49.276 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:49.276 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:49.276 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.273+0000 7fdbcf8dd640 1 Processor -- start 2024-09-16T04:09:49.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 1 -- start start 2024-09-16T04:09:49.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient: init 2024-09-16T04:09:49.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:49.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:49.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc8136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:49.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc8139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:49.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:49.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc80a1340 con 0x7fdbc8139b10 2024-09-16T04:09:49.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc8137650 con 0x7fdbc813d620 2024-09-16T04:09:49.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc80a0aa0 con 0x7fdbc8136b60 2024-09-16T04:09:49.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcf8dd640 10 monclient(hunting): authenticate will time out at 964.779642s 2024-09-16T04:09:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcce78640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdbc8139b10 legacy=0x7fdbc813bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcde7a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdbc813d620 0x7fdbc813fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52516/0 (socket says 172.21.15.171:52516) 2024-09-16T04:09:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbcde7a640 1 -- 172.21.15.171:0/2184713319 learned_addr learned my addr 172.21.15.171:0/2184713319 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454366443 0 0) 0x7fdbc8137650 con 0x7fdbc813d620 2024-09-16T04:09:49.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbbffff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:49.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:49.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:49.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.274+0000 7fdbbffff640 10 monclient(hunting): my global_id is 5452 2024-09-16T04:09:49.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb4003520 con 0x7fdbc813d620 2024-09-16T04:09:49.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1195798994 0 0) 0x7fdbb4003520 con 0x7fdbc813d620 2024-09-16T04:09:49.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 >> v1:172.21.15.148:6790/0 conn(0x7fdbc8136b60 legacy=0x7fdbc8136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:49.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 >> v1:172.21.15.148:6789/0 conn(0x7fdbc8139b10 legacy=0x7fdbc813bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:49.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:49.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 monclient: found mon.noname-c 2024-09-16T04:09:49.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:49.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc80a0550 con 0x7fdbc813d620 2024-09-16T04:09:49.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: _finish_auth 0 2024-09-16T04:09:49.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:49.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:49.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbc0001990 con 0x7fdbc813d620 2024-09-16T04:09:49.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 5 monclient: authenticate success, global_id 5452 2024-09-16T04:09:49.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:49.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:49.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:49.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: dump: 2024-09-16T04:09:49.300 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:49.300 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:49.300 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.300 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.300 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:49.301 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.301 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:49.301 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:49.301 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:49.301 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:49.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbc0001c70 con 0x7fdbc813d620 2024-09-16T04:09:49.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:49.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 1 -- 172.21.15.171:0/2184713319 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbc0002040 con 0x7fdbc813d620 2024-09-16T04:09:49.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:49.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 4 set_mon_vals no callback set 2024-09-16T04:09:49.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:49.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:49.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbbffff640 10 monclient: dump: 2024-09-16T04:09:49.303 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:49.303 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:49.303 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.304 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.304 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:49.304 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.304 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:49.304 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:49.304 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:49.305 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 10 monclient: shutdown 2024-09-16T04:09:49.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:49.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/2184713319 >> v1:172.21.15.171:6789/0 conn(0x7fdbc813d620 legacy=0x7fdbc813fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:49.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/2184713319 shutdown_connections 2024-09-16T04:09:49.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/2184713319 >> 172.21.15.171:0/2184713319 conn(0x7fdbc80a98b0 msgr2=0x7fdbc80a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/2184713319 shutdown_connections 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.275+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/2184713319 wait complete. 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.276+0000 7fdbcf8dd640 10 monclient: build_initial_monmap 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.276+0000 7fdbcf8dd640 10 monclient: monmap: 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:49.306 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.276+0000 7fdbcf8dd640 1 Processor -- start 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.276+0000 7fdbcf8dd640 1 -- start start 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.276+0000 7fdbcf8dd640 10 monclient: init 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc8136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc8139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc813d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc813ad10 con 0x7fdbc8136b60 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc80a0240 con 0x7fdbc8139b10 2024-09-16T04:09:49.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc813baa0 con 0x7fdbc813d620 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcd679640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdbc8136b60 legacy=0x7fdbc8139ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient(hunting): authenticate will time out at 964.782264s 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcce78640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdbc8139b10 0x7fdbc813a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52518/0 (socket says 172.21.15.171:52518) 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcce78640 1 -- 172.21.15.171:0/3759647279 learned_addr learned my addr 172.21.15.171:0/3759647279 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761672949 0 0) 0x7fdbc80a0240 con 0x7fdbc8139b10 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): my global_id is 5455 2024-09-16T04:09:49.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbac003520 con 0x7fdbc8139b10 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969792638 0 0) 0x7fdbc813baa0 con 0x7fdbc813d620 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): my global_id is 5486 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc80a0240 con 0x7fdbc813d620 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 688249762 0 0) 0x7fdbac003520 con 0x7fdbc8139b10 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbc813baa0 con 0x7fdbc8139b10 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbb8004180 con 0x7fdbc8139b10 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:49.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): dump: 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:49.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 782347223 0 0) 0x7fdbc80a0240 con 0x7fdbc813d620 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbac003520 con 0x7fdbc813d620 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbc0001890 con 0x7fdbc813d620 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient(hunting): dump: 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.312 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3046096604 0 0) 0x7fdbc813baa0 con 0x7fdbc8139b10 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 >> v1:172.21.15.148:6790/0 conn(0x7fdbc813d620 legacy=0x7fdbc81d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 >> v1:172.21.15.148:6789/0 conn(0x7fdbc8136b60 legacy=0x7fdbc8139ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 monclient: found mon.b 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc809aac0 con 0x7fdbc8139b10 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:49.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 5 monclient: authenticate success, global_id 5455 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient: _renew_subs 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.277+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc80a28f0 con 0x7fdbc8139b10 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbb8004430 con 0x7fdbc8139b10 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbb8004780 con 0x7fdbc8139b10 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 10 monclient: dump: 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:49.314 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbcf8dd640 10 monclient: _renew_subs 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbcf8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbc8136340 con 0x7fdbc8139b10 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbf7fe640 4 set_mon_vals no callback set 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdbb8004920 con 0x7fdbc8139b10 2024-09-16T04:09:49.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(114..114 src has 1..114) ==== 5781+0+0 (unknown 1472969554 0 0) 0x7fdbb80475c0 con 0x7fdbc8139b10 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.278+0000 7fdbcf8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.279+0000 7fdbcf8dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.279+0000 7fdbcf8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.279+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc80a3290 con 0x7fdbc8139b10 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.279+0000 7fdbcd679640 10 monclient: get_auth_request con 0x7fdbac038f10 auth_method 0 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.283+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdbc80a3290 con 0x7fdbc8139b10 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.283+0000 7fdbbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:49.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.283+0000 7fdbbdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:49.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.454+0000 7fdbcf8dd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T04:09:49.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.454+0000 7fdbcf8dd640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T04:09:49.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.454+0000 7fdbcf8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:49.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:49.454+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) -- 0x7fdbc806bcf0 con 0x7fdbc8139b10 2024-09-16T04:09:50.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.077+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 removed item id 14 name 'osd.14' from crush map v115) ==== 125+0+0 (unknown 2272131927 0 0) 0x7fdbb8019820 con 0x7fdbc8139b10 2024-09-16T04:09:50.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.077+0000 7fdbbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T04:09:50.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.077+0000 7fdbbdffb640 10 monclient: _finish_command 2 = system:0 removed item id 14 name 'osd.14' from crush map 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.183+0000 7fdbcf8dd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.183+0000 7fdbcf8dd640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.183+0000 7fdbcf8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.183+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "osd.14"} v 0) -- 0x7fdbc80a4ca0 con 0x7fdbc8139b10 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.184+0000 7fdbbdffb640 1 -- 172.21.15.171:0/3759647279 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.14"}]=0 device 'osd.14' does not appear in the crush map v115) ==== 126+0+0 (unknown 1992080539 0 0) 0x7fdbb801e0e0 con 0x7fdbc8139b10 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.184+0000 7fdbbdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.14"}] 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.184+0000 7fdbbdffb640 10 monclient: _finish_command 3 = system:0 device 'osd.14' does not appear in the crush map 2024-09-16T04:09:50.186 INFO:tasks.workunit.client.0.smithi171.stderr:device 'osd.14' does not appear in the crush map 2024-09-16T04:09:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 >> v1:172.21.15.171:6812/7327273 conn(0x7fdbac038f10 legacy=0x7fdbac03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 10 monclient: shutdown 2024-09-16T04:09:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 >> v1:172.21.15.171:6789/0 conn(0x7fdbc8139b10 legacy=0x7fdbc813a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:50.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 shutdown_connections 2024-09-16T04:09:50.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 >> 172.21.15.171:0/3759647279 conn(0x7fdbc80a98b0 msgr2=0x7fdbc80a7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:50.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 shutdown_connections 2024-09-16T04:09:50.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.186+0000 7fdbcf8dd640 1 -- 172.21.15.171:0/3759647279 wait complete. 2024-09-16T04:09:50.197 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm osd.12 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.273+0000 7f69bbdd6640 10 monclient: get_monmap_and_config 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.273+0000 7f69bbdd6640 10 monclient: build_initial_monmap 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.273+0000 7f69bbdd6640 10 monclient: monmap: 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:50.274050+0000 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:50.274050+0000 2024-09-16T04:09:50.275 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:50.276 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:50.276 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:50.276 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:50.276 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:50.276 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:50.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.273+0000 7f69bbdd6640 1 Processor -- start 2024-09-16T04:09:50.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 1 -- start start 2024-09-16T04:09:50.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient: init 2024-09-16T04:09:50.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:50.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient(hunting): picked mon.noname-b con 0x7f69b4136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient(hunting): picked mon.noname-a con 0x7f69b4139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient(hunting): picked mon.noname-c con 0x7f69b413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b40a1340 con 0x7f69b4139b10 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b4137650 con 0x7f69b413d620 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b40a0aa0 con 0x7f69b4136b60 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69bbdd6640 10 monclient(hunting): authenticate will time out at 965.779586s 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b9371640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f69b4139b10 legacy=0x7f69b413bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69ba373640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f69b413d620 0x7f69b413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52522/0 (socket says 172.21.15.171:52522) 2024-09-16T04:09:50.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69ba373640 1 -- 172.21.15.171:0/2722953096 learned_addr learned my addr 172.21.15.171:0/2722953096 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957283499 0 0) 0x7f69b4137650 con 0x7f69b413d620 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b8b70640 10 monclient(hunting): _negotiate 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b8b70640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b8b70640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b8b70640 10 monclient(hunting): my global_id is 5458 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.274+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69a0003520 con 0x7f69b413d620 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2411806928 0 0) 0x7f69a0003520 con 0x7f69b413d620 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 >> v1:172.21.15.148:6790/0 conn(0x7f69b4136b60 legacy=0x7f69b4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 >> v1:172.21.15.148:6789/0 conn(0x7f69b4139b10 legacy=0x7f69b413bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: _finish_hunting 0 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 monclient: found mon.noname-c 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:50.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b40a0550 con 0x7f69b413d620 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: _finish_auth 0 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: _check_auth_tickets 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 5 monclient: authenticate success, global_id 5458 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69ac001990 con 0x7f69b413d620 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: dump: 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.291 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69ac001cf0 con 0x7f69b413d620 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 1 -- 172.21.15.171:0/2722953096 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69ac004c20 con 0x7f69b413d620 2024-09-16T04:09:50.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 4 set_mon_vals no callback set 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69b8b70640 10 monclient: dump: 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:50.293 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 10 monclient: shutdown 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 1 -- 172.21.15.171:0/2722953096 >> v1:172.21.15.171:6789/0 conn(0x7f69b413d620 legacy=0x7f69b413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 1 -- 172.21.15.171:0/2722953096 shutdown_connections 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 1 -- 172.21.15.171:0/2722953096 >> 172.21.15.171:0/2722953096 conn(0x7f69b40a98b0 msgr2=0x7f69b40a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 1 -- 172.21.15.171:0/2722953096 shutdown_connections 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.275+0000 7f69bbdd6640 1 -- 172.21.15.171:0/2722953096 wait complete. 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.276+0000 7f69bbdd6640 10 monclient: build_initial_monmap 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.276+0000 7f69bbdd6640 10 monclient: monmap: 2024-09-16T04:09:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:50.295 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:50.295 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:50.295 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:50.295 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:50.295 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:50.295 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.276+0000 7f69bbdd6640 1 Processor -- start 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.276+0000 7f69bbdd6640 1 -- start start 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.276+0000 7f69bbdd6640 10 monclient: init 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient(hunting): picked mon.noname-b con 0x7f69b4136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient(hunting): picked mon.noname-c con 0x7f69b4139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:50.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient(hunting): picked mon.noname-a con 0x7f69b413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b4120b70 con 0x7f69b413d620 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b4120e50 con 0x7f69b4136b60 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b40a0240 con 0x7f69b4139b10 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient(hunting): authenticate will time out at 965.782373s 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69b9b72640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f69b4136b60 0x7f69b41cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52526/0 (socket says 172.21.15.171:52526) 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69b9b72640 1 -- 172.21.15.171:0/4210209341 learned_addr learned my addr 172.21.15.171:0/4210209341 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69ba373640 1 -- 172.21.15.171:0/4210209341 >> v1:172.21.15.148:6789/0 conn(0x7f69b413d620 legacy=0x7f69b41d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 244133992 0 0) 0x7f69b4120e50 con 0x7f69b4136b60 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): my global_id is 5461 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6998003520 con 0x7f69b4136b60 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406224603 0 0) 0x7f69b40a0240 con 0x7f69b4139b10 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): my global_id is 5492 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4120e50 con 0x7f69b4139b10 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1749593641 0 0) 0x7f6998003520 con 0x7f69b4136b60 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69b40a0240 con 0x7f69b4136b60 2024-09-16T04:09:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69a40019c0 con 0x7f69b4136b60 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): dump: 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:50.299 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2684819343 0 0) 0x7f69b40a0240 con 0x7f69b4136b60 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 >> v1:172.21.15.148:6790/0 conn(0x7f69b4139b10 legacy=0x7f69b41d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 >> v1:172.21.15.148:6789/0 conn(0x7f69b413d620 legacy=0x7f69b41d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 monclient: found mon.b 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b40a5650 con 0x7f69b4136b60 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:50.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 5 monclient: authenticate success, global_id 5461 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient: _renew_subs 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.277+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f69b40a2b50 con 0x7f69b4136b60 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69bbdd6640 10 monclient: _renew_subs 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69bbdd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f69b4121270 con 0x7f69b4136b60 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69a4001ca0 con 0x7f69b4136b60 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:50.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69a4003a60 con 0x7f69b4136b60 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69abfff640 4 set_mon_vals no callback set 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69aa7fc640 10 monclient: dump: 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:50.302 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.278+0000 7f69bbdd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.279+0000 7f69bbdd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.279+0000 7f69bbdd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.279+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f69a4003c00 con 0x7f69b4136b60 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.279+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699c005080 con 0x7f69b4136b60 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.279+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(115..115 src has 1..115) ==== 5759+0+0 (unknown 1861698942 0 0) 0x7f69a40475f0 con 0x7f69b4136b60 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.279+0000 7f69b9371640 10 monclient: get_auth_request con 0x7f6998039000 auth_method 0 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.283+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f69a4015ce0 con 0x7f69b4136b60 2024-09-16T04:09:50.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.283+0000 7f69aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:50.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.283+0000 7f69aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:50.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.457+0000 7f69bbdd6640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T04:09:50.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.457+0000 7f69bbdd6640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T04:09:50.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.457+0000 7f69bbdd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:50.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:50.457+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) -- 0x7f699c005370 con 0x7f69b4136b60 2024-09-16T04:09:50.460 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-09-16T04:09:50.458+0000 7f1fb544d640 -1 mon.b@1(leader).osd e115 definitely_dead 0 2024-09-16T04:09:51.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.078+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v116) ==== 87+0+0 (unknown 4094448600 0 0) 0x7f69a40155e0 con 0x7f69b4136b60 2024-09-16T04:09:51.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.078+0000 7f69aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T04:09:51.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.078+0000 7f69aa7fc640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T04:09:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.190+0000 7f69bbdd6640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T04:09:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.190+0000 7f69bbdd6640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T04:09:51.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.190+0000 7f69bbdd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.190+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) -- 0x7f699c000bc0 con 0x7f69b4136b60 2024-09-16T04:09:51.194 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-09-16T04:09:51.191+0000 7f1fb544d640 -1 mon.b@1(leader).osd e116 definitely_dead 0 2024-09-16T04:09:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.191+0000 7f69aa7fc640 1 -- 172.21.15.171:0/4210209341 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v116) ==== 96+0+0 (unknown 2057870606 0 0) 0x7f69a401e190 con 0x7f69b4136b60 2024-09-16T04:09:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.191+0000 7f69aa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-16T04:09:51.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.191+0000 7f69aa7fc640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T04:09:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:osd.12 does not exist. 2024-09-16T04:09:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 >> v1:172.21.15.171:6812/7327273 conn(0x7f6998039000 legacy=0x7f699803b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 10 monclient: shutdown 2024-09-16T04:09:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 >> v1:172.21.15.171:6789/0 conn(0x7f69b4136b60 legacy=0x7f69b41cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:51.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 shutdown_connections 2024-09-16T04:09:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 >> 172.21.15.171:0/4210209341 conn(0x7f69b40a98b0 msgr2=0x7f69b4072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 shutdown_connections 2024-09-16T04:09:51.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.193+0000 7f69bbdd6640 1 -- 172.21.15.171:0/4210209341 wait complete. 2024-09-16T04:09:51.204 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm osd.14 2024-09-16T04:09:51.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.280+0000 7f93643f6640 10 monclient: get_monmap_and_config 2024-09-16T04:09:51.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.280+0000 7f93643f6640 10 monclient: build_initial_monmap 2024-09-16T04:09:51.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.280+0000 7f93643f6640 10 monclient: monmap: 2024-09-16T04:09:51.282 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:51.283 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:51.283 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:51.281101+0000 2024-09-16T04:09:51.283 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:51.281101+0000 2024-09-16T04:09:51.284 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:51.284 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:51.284 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:51.284 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:51.284 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:51.285 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:51.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.280+0000 7f93643f6640 1 Processor -- start 2024-09-16T04:09:51.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.280+0000 7f93643f6640 1 -- start start 2024-09-16T04:09:51.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.280+0000 7f93643f6640 10 monclient: init 2024-09-16T04:09:51.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:51.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient(hunting): picked mon.noname-a con 0x7f935c09ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient(hunting): picked mon.noname-c con 0x7f935c0a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient(hunting): picked mon.noname-b con 0x7f935c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c09fbf0 con 0x7f935c09ba30 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c0a0e10 con 0x7f935c0a0750 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c09f640 con 0x7f935c09e570 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f93643f6640 10 monclient(hunting): authenticate will time out at 966.786713s 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f9362192640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f935c09ba30 legacy=0x7f935c0a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f9361991640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f935c0a0750 0x7f935c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52530/0 (socket says 172.21.15.171:52530) 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.281+0000 7f9361991640 1 -- 172.21.15.171:0/1388246346 learned_addr learned my addr 172.21.15.171:0/1388246346 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012790191 0 0) 0x7f935c0a0e10 con 0x7f935c0a0750 2024-09-16T04:09:51.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): _negotiate 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): my global_id is 5464 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9348003520 con 0x7f935c0a0750 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814322746 0 0) 0x7f935c09f640 con 0x7f935c09e570 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): _negotiate 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient(hunting): my global_id is 5495 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f935c0a0e10 con 0x7f935c09e570 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1360243893 0 0) 0x7f9348003520 con 0x7f935c0a0750 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 >> v1:172.21.15.148:6790/0 conn(0x7f935c09e570 legacy=0x7f935c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 >> v1:172.21.15.148:6789/0 conn(0x7f935c09ba30 legacy=0x7f935c0a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: _finish_hunting 0 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 monclient: found mon.noname-c 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935c09f810 con 0x7f935c0a0750 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: _finish_auth 0 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: _check_auth_tickets 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9350001610 con 0x7f935c0a0750 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: dump: 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 5 monclient: authenticate success, global_id 5464 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9350001870 con 0x7f935c0a0750 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 1 -- 172.21.15.171:0/1388246346 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9350001c40 con 0x7f935c0a0750 2024-09-16T04:09:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f9361190640 10 monclient: dump: 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 4 set_mon_vals no callback set 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 10 monclient: shutdown 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.282+0000 7f93643f6640 1 -- 172.21.15.171:0/1388246346 >> v1:172.21.15.171:6789/0 conn(0x7f935c0a0750 legacy=0x7f935c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 1 -- 172.21.15.171:0/1388246346 shutdown_connections 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 1 -- 172.21.15.171:0/1388246346 >> 172.21.15.171:0/1388246346 conn(0x7f935c0a52a0 msgr2=0x7f935c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 1 -- 172.21.15.171:0/1388246346 shutdown_connections 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 1 -- 172.21.15.171:0/1388246346 wait complete. 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 10 monclient: build_initial_monmap 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 10 monclient: monmap: 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 1 Processor -- start 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.283+0000 7f93643f6640 1 -- start start 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient: init 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient(hunting): picked mon.noname-b con 0x7f935c09ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient(hunting): picked mon.noname-c con 0x7f935c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient(hunting): picked mon.noname-a con 0x7f935c0a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c12c690 con 0x7f935c0a0750 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c132930 con 0x7f935c09ba30 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f935c06ceb0 con 0x7f935c09e570 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f93643f6640 10 monclient(hunting): authenticate will time out at 966.789501s 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f9362192640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f935c09ba30 0x7f935c12b870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52532/0 (socket says 172.21.15.171:52532) 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f9362192640 1 -- 172.21.15.171:0/16442652 learned_addr learned my addr 172.21.15.171:0/16442652 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f9362993640 1 -- 172.21.15.171:0/16442652 >> v1:172.21.15.148:6789/0 conn(0x7f935c0a0750 legacy=0x7f935c1d1b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3937162769 0 0) 0x7f935c132930 con 0x7f935c09ba30 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:51.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): my global_id is 5467 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9340003520 con 0x7f935c09ba30 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817354778 0 0) 0x7f935c06ceb0 con 0x7f935c09e570 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): my global_id is 5498 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f935c132930 con 0x7f935c09e570 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1958893261 0 0) 0x7f9340003520 con 0x7f935c09ba30 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f935c06ceb0 con 0x7f935c09ba30 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f93580041d0 con 0x7f935c09ba30 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:51.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): dump: 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.284+0000 7f934effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3412392483 0 0) 0x7f935c06ceb0 con 0x7f935c09ba30 2024-09-16T04:09:51.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 >> v1:172.21.15.148:6790/0 conn(0x7f935c09e570 legacy=0x7f935c12bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 >> v1:172.21.15.148:6789/0 conn(0x7f935c0a0750 legacy=0x7f935c1d1b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 monclient: found mon.b 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935c12e7b0 con 0x7f935c09ba30 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 5 monclient: authenticate success, global_id 5467 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 10 monclient: _renew_subs 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f935c12ea90 con 0x7f935c09ba30 2024-09-16T04:09:51.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 10 monclient: _renew_subs 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f935c12d8b0 con 0x7f935c09ba30 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9358004430 con 0x7f935c09ba30 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9358004780 con 0x7f935c09ba30 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f934effd640 10 monclient: dump: 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:51.303 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.285+0000 7f936098f640 4 set_mon_vals no callback set 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f93580049a0 con 0x7f935c09ba30 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f93643f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f93643f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f93643f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f935c0008d0 con 0x7f935c09ba30 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(116..116 src has 1..116) ==== 5759+0+0 (unknown 3775119368 0 0) 0x7f93580474e0 con 0x7f935c09ba30 2024-09-16T04:09:51.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.286+0000 7f9361991640 10 monclient: get_auth_request con 0x7f9340038be0 auth_method 0 2024-09-16T04:09:51.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.290+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f93580198a0 con 0x7f935c09ba30 2024-09-16T04:09:51.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.290+0000 7f934effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:51.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.290+0000 7f934effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:51.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.466+0000 7f93643f6640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T04:09:51.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.466+0000 7f93643f6640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T04:09:51.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.466+0000 7f93643f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:51.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:51.466+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) -- 0x7f935c12d570 con 0x7f935c09ba30 2024-09-16T04:09:51.469 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-09-16T04:09:51.467+0000 7f1fb544d640 -1 mon.b@1(leader).osd e116 definitely_dead 0 2024-09-16T04:09:52.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.093+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 removed osd.14 v117) ==== 87+0+0 (unknown 1962155109 0 0) 0x7f935801e160 con 0x7f935c09ba30 2024-09-16T04:09:52.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.093+0000 7f934effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T04:09:52.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.093+0000 7f934effd640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T04:09:52.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f93643f6640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T04:09:52.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f93643f6640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T04:09:52.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f93643f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["osd.14"]} v 0) -- 0x7f935c13e810 con 0x7f935c09ba30 2024-09-16T04:09:52.208 INFO:tasks.ceph.mon.b.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f1fb544d640 -1 mon.b@1(leader).osd e117 definitely_dead 0 2024-09-16T04:09:52.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f934effd640 1 -- 172.21.15.171:0/16442652 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.14"]}]=0 osd.14 does not exist. v117) ==== 96+0+0 (unknown 906570153 0 0) 0x7f9358044b90 con 0x7f935c09ba30 2024-09-16T04:09:52.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f934effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.14"]}] 2024-09-16T04:09:52.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.205+0000 7f934effd640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T04:09:52.208 INFO:tasks.workunit.client.0.smithi171.stderr:osd.14 does not exist. 2024-09-16T04:09:52.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 >> v1:172.21.15.171:6812/7327273 conn(0x7f9340038be0 legacy=0x7f934003b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:52.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 10 monclient: shutdown 2024-09-16T04:09:52.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:52.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 >> v1:172.21.15.171:6789/0 conn(0x7f935c09ba30 legacy=0x7f935c12b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:52.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 shutdown_connections 2024-09-16T04:09:52.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 >> 172.21.15.171:0/16442652 conn(0x7f935c0a52a0 msgr2=0x7f935c131cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:52.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 shutdown_connections 2024-09-16T04:09:52.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.207+0000 7f93643f6640 1 -- 172.21.15.171:0/16442652 wait complete. 2024-09-16T04:09:52.219 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.295+0000 7fbda9c77640 10 monclient: get_monmap_and_config 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.295+0000 7fbda9c77640 10 monclient: build_initial_monmap 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.295+0000 7fbda9c77640 10 monclient: monmap: 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:52.296028+0000 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:52.296028+0000 2024-09-16T04:09:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.295+0000 7fbda9c77640 1 Processor -- start 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 1 -- start start 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient: init 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient(hunting): picked mon.noname-a con 0x7fbda4136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient(hunting): picked mon.noname-c con 0x7fbda4139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient(hunting): picked mon.noname-b con 0x7fbda413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda40a1340 con 0x7fbda4136b60 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda4137650 con 0x7fbda4139b10 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda40a0aa0 con 0x7fbda413d620 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda9c77640 10 monclient(hunting): authenticate will time out at 967.801593s 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbda4139b10 0x7fbda413bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52538/0 (socket says 172.21.15.171:52538) 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbda4136b60 legacy=0x7fbda4136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda2ffd640 1 -- 172.21.15.171:0/3523059822 learned_addr learned my addr 172.21.15.171:0/3523059822 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324349578 0 0) 0x7fbda4137650 con 0x7fbda4139b10 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda27fc640 10 monclient(hunting): my global_id is 5470 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.296+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd8c003520 con 0x7fbda4139b10 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2184430328 0 0) 0x7fbd8c003520 con 0x7fbda4139b10 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 >> v1:172.21.15.148:6790/0 conn(0x7fbda413d620 legacy=0x7fbda413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 >> v1:172.21.15.148:6789/0 conn(0x7fbda4136b60 legacy=0x7fbda4136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 monclient: found mon.noname-c 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda40a0550 con 0x7fbda4139b10 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: _finish_auth 0 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 5 monclient: authenticate success, global_id 5470 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd98001610 con 0x7fbda4139b10 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:52.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: dump: 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:52.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd98001970 con 0x7fbda4139b10 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 1 -- 172.21.15.171:0/3523059822 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd98001d40 con 0x7fbda4139b10 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 4 set_mon_vals no callback set 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda27fc640 10 monclient: dump: 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:52.310 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 10 monclient: shutdown 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 1 -- 172.21.15.171:0/3523059822 >> v1:172.21.15.171:6789/0 conn(0x7fbda4139b10 legacy=0x7fbda413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 1 -- 172.21.15.171:0/3523059822 shutdown_connections 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 1 -- 172.21.15.171:0/3523059822 >> 172.21.15.171:0/3523059822 conn(0x7fbda40a98b0 msgr2=0x7fbda40a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:52.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 1 -- 172.21.15.171:0/3523059822 shutdown_connections 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.297+0000 7fbda9c77640 1 -- 172.21.15.171:0/3523059822 wait complete. 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.298+0000 7fbda9c77640 10 monclient: build_initial_monmap 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.298+0000 7fbda9c77640 10 monclient: monmap: 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:52.312 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.298+0000 7fbda9c77640 1 Processor -- start 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.298+0000 7fbda9c77640 1 -- start start 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.298+0000 7fbda9c77640 10 monclient: init 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:52.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient(hunting): picked mon.noname-a con 0x7fbda4136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient(hunting): picked mon.noname-c con 0x7fbda4139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient(hunting): picked mon.noname-b con 0x7fbda413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda409f750 con 0x7fbda4136b60 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda409e350 con 0x7fbda413d620 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda40a6f50 con 0x7fbda4139b10 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient(hunting): authenticate will time out at 967.804270s 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbda4136b60 legacy=0x7fbda409d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbda413d620 0x7fbda41d3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52554/0 (socket says 172.21.15.171:52554) 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda3fff640 1 -- 172.21.15.171:0/1246843141 learned_addr learned my addr 172.21.15.171:0/1246843141 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840770879 0 0) 0x7fbda409e350 con 0x7fbda413d620 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:52.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): my global_id is 5473 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd90003550 con 0x7fbda413d620 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724296381 0 0) 0x7fbda40a6f50 con 0x7fbda4139b10 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): my global_id is 5504 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbda409e350 con 0x7fbda4139b10 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 595011098 0 0) 0x7fbd90003550 con 0x7fbda413d620 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbda40a6f50 con 0x7fbda413d620 2024-09-16T04:09:52.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd9c001c70 con 0x7fbda413d620 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): dump: 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4166677973 0 0) 0x7fbda40a6f50 con 0x7fbda413d620 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 >> v1:172.21.15.148:6790/0 conn(0x7fbda4139b10 legacy=0x7fbda409dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 >> v1:172.21.15.148:6789/0 conn(0x7fbda4136b60 legacy=0x7fbda409d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 monclient: found mon.b 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda40a29e0 con 0x7fbda413d620 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient: _finish_auth 0 2024-09-16T04:09:52.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbd8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 5 monclient: authenticate success, global_id 5473 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient: _renew_subs 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbda4120a00 con 0x7fbda413d620 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient: _renew_subs 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.299+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbda41213e0 con 0x7fbda413d620 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd9c002040 con 0x7fbda413d620 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd9c003de0 con 0x7fbda413d620 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:52.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbda1ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 10 monclient: dump: 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbd9c003f80 con 0x7fbda413d620 2024-09-16T04:09:52.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbda9c77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbda9c77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbda9c77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.300+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd84005080 con 0x7fbda413d620 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.301+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(117..117 src has 1..117) ==== 5759+0+0 (unknown 3583132098 0 0) 0x7fbd9c047900 con 0x7fbda413d620 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.301+0000 7fbda37fe640 10 monclient: get_auth_request con 0x7fbd90038fa0 auth_method 0 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.305+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbd9c016010 con 0x7fbda413d620 2024-09-16T04:09:52.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.305+0000 7fbd8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:52.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.305+0000 7fbd8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:52.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.478+0000 7fbda9c77640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T04:09:52.480 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.478+0000 7fbda9c77640 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T04:09:52.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.478+0000 7fbda9c77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:52.481 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:52.478+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) -- 0x7fbd84005b80 con 0x7fbda413d620 2024-09-16T04:09:53.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.094+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v118) ==== 145+0+0 (unknown 3673119606 0 0) 0x7fbd9c015910 con 0x7fbda413d620 2024-09-16T04:09:53.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.094+0000 7fbd8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T04:09:53.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.094+0000 7fbd8bfff640 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-16T04:09:53.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.207+0000 7fbda9c77640 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T04:09:53.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.207+0000 7fbda9c77640 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T04:09:53.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.207+0000 7fbda9c77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:53.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:53.207+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) -- 0x7fbd84001090 con 0x7fbda413d620 2024-09-16T04:09:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.105+0000 7fbd8bfff640 1 -- 172.21.15.171:0/1246843141 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v119) ==== 145+0+0 (unknown 288591493 0 0) 0x7fbd9c019c60 con 0x7fbda413d620 2024-09-16T04:09:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.105+0000 7fbd8bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-16T04:09:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.105+0000 7fbd8bfff640 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-16T04:09:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:set require_min_compat_client to luminous 2024-09-16T04:09:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.108+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 >> v1:172.21.15.171:6812/7327273 conn(0x7fbd90038fa0 legacy=0x7fbd9003b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.108+0000 7fbda9c77640 10 monclient: shutdown 2024-09-16T04:09:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.108+0000 7fbda9c77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.108+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 >> v1:172.21.15.171:6789/0 conn(0x7fbda413d620 legacy=0x7fbda41d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.108+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 shutdown_connections 2024-09-16T04:09:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.109+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 >> 172.21.15.171:0/1246843141 conn(0x7fbda40a98b0 msgr2=0x7fbda41392e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.109+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 shutdown_connections 2024-09-16T04:09:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.109+0000 7fbda9c77640 1 -- 172.21.15.171:0/1246843141 wait complete. 2024-09-16T04:09:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set dump 2024-09-16T04:09:54.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.204+0000 7f62802ba640 10 monclient: get_monmap_and_config 2024-09-16T04:09:54.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.204+0000 7f62802ba640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.204+0000 7f62802ba640 10 monclient: monmap: 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:54.205551+0000 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:54.205551+0000 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:54.207 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.205+0000 7f62802ba640 1 Processor -- start 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.205+0000 7f62802ba640 1 -- start start 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.205+0000 7f62802ba640 10 monclient: init 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.205+0000 7f62802ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.205+0000 7f62802ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 10 monclient(hunting): picked mon.noname-c con 0x7f62781327c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 10 monclient(hunting): picked mon.noname-a con 0x7f627809eb20 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6278139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627809ef30 con 0x7f627809eb20 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627809d880 con 0x7f62781327c0 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627809db60 con 0x7f6278139ea0 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 10 monclient(hunting): authenticate will time out at 969.711194s 2024-09-16T04:09:54.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627e056640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f62781327c0 0x7f6278132ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52556/0 (socket says 172.21.15.171:52556) 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627e056640 1 -- 172.21.15.171:0/1054076282 learned_addr learned my addr 172.21.15.171:0/1054076282 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d855640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f627809eb20 legacy=0x7f6278138780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1283713604 0 0) 0x7f627809d880 con 0x7f62781327c0 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): my global_id is 5476 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6264003520 con 0x7f62781327c0 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156798918 0 0) 0x7f627809db60 con 0x7f6278139ea0 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient(hunting): my global_id is 5507 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f627809d880 con 0x7f6278139ea0 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3500088073 0 0) 0x7f6264003520 con 0x7f62781327c0 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 >> v1:172.21.15.148:6790/0 conn(0x7f6278139ea0 legacy=0x7f627813c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 >> v1:172.21.15.148:6789/0 conn(0x7f627809eb20 legacy=0x7f6278138780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: _finish_hunting 0 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 monclient: found mon.noname-c 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627809de40 con 0x7f62781327c0 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: _finish_auth 0 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: _check_auth_tickets 2024-09-16T04:09:54.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6274001540 con 0x7f62781327c0 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 5 monclient: authenticate success, global_id 5476 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f627d054640 10 monclient: dump: 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.218 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.206+0000 7f62802ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62740017f0 con 0x7f62781327c0 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f627d054640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f627d054640 1 -- 172.21.15.171:0/1054076282 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6274001bc0 con 0x7f62781327c0 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 4 set_mon_vals no callback set 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f627d054640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f627d054640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f627d054640 10 monclient: dump: 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.219 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 10 monclient: shutdown 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 1 -- 172.21.15.171:0/1054076282 >> v1:172.21.15.171:6789/0 conn(0x7f62781327c0 legacy=0x7f6278132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 1 -- 172.21.15.171:0/1054076282 shutdown_connections 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 1 -- 172.21.15.171:0/1054076282 >> 172.21.15.171:0/1054076282 conn(0x7f62780a5370 msgr2=0x7f62780a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:54.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 1 -- 172.21.15.171:0/1054076282 shutdown_connections 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.207+0000 7f62802ba640 1 -- 172.21.15.171:0/1054076282 wait complete. 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.208+0000 7f62802ba640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.208+0000 7f62802ba640 10 monclient: monmap: 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:54.221 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.208+0000 7f62802ba640 1 Processor -- start 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.208+0000 7f62802ba640 1 -- start start 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.208+0000 7f62802ba640 10 monclient: init 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient(hunting): picked mon.noname-b con 0x7f627809eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient(hunting): picked mon.noname-a con 0x7f62781327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6278139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627813c290 con 0x7f62781327c0 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627813c570 con 0x7f627809eb20 2024-09-16T04:09:54.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627813b2e0 con 0x7f6278139ea0 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient(hunting): authenticate will time out at 969.714314s 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f627e056640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f627809eb20 0x7f62780719b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52570/0 (socket says 172.21.15.171:52570) 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f627e056640 1 -- 172.21.15.171:0/977444929 learned_addr learned my addr 172.21.15.171:0/977444929 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f627d855640 1 -- 172.21.15.171:0/977444929 >> v1:172.21.15.148:6789/0 conn(0x7f62781327c0 legacy=0x7f62780720c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841147294 0 0) 0x7f627813c570 con 0x7f627809eb20 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): my global_id is 5479 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f625c003520 con 0x7f627809eb20 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3642565882 0 0) 0x7f625c003520 con 0x7f627809eb20 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f627813c570 con 0x7f627809eb20 2024-09-16T04:09:54.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183679230 0 0) 0x7f627813b2e0 con 0x7f6278139ea0 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): my global_id is 5510 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f625c003520 con 0x7f6278139ea0 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f62740015c0 con 0x7f627809eb20 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): dump: 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.224 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 476021578 0 0) 0x7f627813c570 con 0x7f627809eb20 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 >> v1:172.21.15.148:6790/0 conn(0x7f6278139ea0 legacy=0x7f62781d3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 >> v1:172.21.15.148:6789/0 conn(0x7f62781327c0 legacy=0x7f62780720c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:54.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 monclient: found mon.b 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6278139750 con 0x7f627809eb20 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f626effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 5 monclient: authenticate success, global_id 5479 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient: _renew_subs 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.209+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6278140600 con 0x7f627809eb20 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f62802ba640 10 monclient: _renew_subs 2024-09-16T04:09:54.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f62802ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f62780727d0 con 0x7f627809eb20 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62740017f0 con 0x7f627809eb20 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6274004cc0 con 0x7f627809eb20 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f627c853640 4 set_mon_vals no callback set 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 10 monclient: dump: 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.227 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.210+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6274004e60 con 0x7f627809eb20 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.211+0000 7f62802ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.211+0000 7f62802ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.211+0000 7f62802ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.211+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6260005080 con 0x7f627809eb20 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.211+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(119..119 src has 1..119) ==== 5759+0+0 (unknown 3165733462 0 0) 0x7f6274047690 con 0x7f627809eb20 2024-09-16T04:09:54.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.211+0000 7f627d855640 10 monclient: get_auth_request con 0x7f625c038f30 auth_method 0 2024-09-16T04:09:54.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.215+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6274015d80 con 0x7f627809eb20 2024-09-16T04:09:54.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.215+0000 7f626effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.215+0000 7f626effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.389+0000 7f62802ba640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.389+0000 7f62802ba640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.389+0000 7f62802ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.389+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7f6260005370 con 0x7f627809eb20 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.390+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v119) ==== 73+0+3 (unknown 3293156399 0 2423995997) 0x7f6274015680 con 0x7f627809eb20 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.390+0000 7f626effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:09:54.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.390+0000 7f626effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:54.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f62802ba640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:09:54.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f62802ba640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:09:54.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f62802ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7f6260000bb0 con 0x7f627809eb20 2024-09-16T04:09:54.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f626effd640 1 -- 172.21.15.171:0/977444929 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v119) ==== 73+0+3 (unknown 3293156399 0 2423995997) 0x7f627401e230 con 0x7f627809eb20 2024-09-16T04:09:54.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f626effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:09:54.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.495+0000 7f626effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:54.498 INFO:tasks.workunit.client.0.smithi171.stdout:{} 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.497+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 >> v1:172.21.15.171:6812/7327273 conn(0x7f625c038f30 legacy=0x7f625c03b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.497+0000 7f62802ba640 10 monclient: shutdown 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.497+0000 7f62802ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.497+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 >> v1:172.21.15.171:6789/0 conn(0x7f627809eb20 legacy=0x7f62780719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.497+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 shutdown_connections 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.497+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 >> 172.21.15.171:0/977444929 conn(0x7f62780a5370 msgr2=0x7f62780a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.498+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 shutdown_connections 2024-09-16T04:09:54.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.498+0000 7f62802ba640 1 -- 172.21.15.171:0/977444929 wait complete. 2024-09-16T04:09:54.509 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:09:54.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.584+0000 7f02789ed640 10 monclient: get_monmap_and_config 2024-09-16T04:09:54.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.584+0000 7f02789ed640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.584+0000 7f02789ed640 10 monclient: monmap: 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:54.585523+0000 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:54.585523+0000 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:54.587 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 1 Processor -- start 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 1 -- start start 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient: init 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient(hunting): picked mon.noname-a con 0x7f02741327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient(hunting): picked mon.noname-b con 0x7f027409eb20 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient(hunting): picked mon.noname-c con 0x7f0274139ea0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027409ef30 con 0x7f02741327c0 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027409d880 con 0x7f0274139ea0 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027409db60 con 0x7f027409eb20 2024-09-16T04:09:54.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.585+0000 7f02789ed640 10 monclient(hunting): authenticate will time out at 970.090954s 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027259c640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f02741327c0 legacy=0x7f0274132ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f0272d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0274139ea0 0x7f027413c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52574/0 (socket says 172.21.15.171:52574) 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f0272d9d640 1 -- 172.21.15.171:0/1161278770 learned_addr learned my addr 172.21.15.171:0/1161278770 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385905582 0 0) 0x7f027409d880 con 0x7f0274139ea0 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): my global_id is 5482 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f025c003520 con 0x7f0274139ea0 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974490684 0 0) 0x7f027409db60 con 0x7f027409eb20 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient(hunting): my global_id is 5513 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f027409d880 con 0x7f027409eb20 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 235286581 0 0) 0x7f025c003520 con 0x7f0274139ea0 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 >> v1:172.21.15.148:6790/0 conn(0x7f027409eb20 legacy=0x7f0274138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 >> v1:172.21.15.148:6789/0 conn(0x7f02741327c0 legacy=0x7f0274132ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: _finish_hunting 0 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 monclient: found mon.noname-c 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027409de40 con 0x7f0274139ea0 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: _finish_auth 0 2024-09-16T04:09:54.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f02789ed640 5 monclient: authenticate success, global_id 5482 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f026c001990 con 0x7f0274139ea0 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: dump: 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.597 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f02789ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f026c001c40 con 0x7f0274139ea0 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f02789ed640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 1 -- 172.21.15.171:0/1161278770 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f026c002010 con 0x7f0274139ea0 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f02789ed640 4 set_mon_vals no callback set 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.586+0000 7f027159a640 10 monclient: dump: 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.598 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 10 monclient: shutdown 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 1 -- 172.21.15.171:0/1161278770 >> v1:172.21.15.171:6789/0 conn(0x7f0274139ea0 legacy=0x7f027413c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 1 -- 172.21.15.171:0/1161278770 shutdown_connections 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 1 -- 172.21.15.171:0/1161278770 >> 172.21.15.171:0/1161278770 conn(0x7f02740a5370 msgr2=0x7f02740a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 1 -- 172.21.15.171:0/1161278770 shutdown_connections 2024-09-16T04:09:54.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 1 -- 172.21.15.171:0/1161278770 wait complete. 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.587+0000 7f02789ed640 10 monclient: monmap: 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 1 Processor -- start 2024-09-16T04:09:54.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 1 -- start start 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient: init 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient(hunting): picked mon.noname-b con 0x7f027409eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient(hunting): picked mon.noname-a con 0x7f02741327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient(hunting): picked mon.noname-c con 0x7f0274139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027406ed60 con 0x7f02741327c0 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0274139b30 con 0x7f027409eb20 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0274139040 con 0x7f0274139ea0 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f02789ed640 10 monclient(hunting): authenticate will time out at 970.093779s 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f0271d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f02741327c0 legacy=0x7f027406e650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:54.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f027259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f027409eb20 0x7f027406df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52586/0 (socket says 172.21.15.171:52586) 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.588+0000 7f027259c640 1 -- 172.21.15.171:0/1257625223 learned_addr learned my addr 172.21.15.171:0/1257625223 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680204168 0 0) 0x7f0274139b30 con 0x7f027409eb20 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): my global_id is 5485 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0254003520 con 0x7f027409eb20 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744968816 0 0) 0x7f0274139040 con 0x7f0274139ea0 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): my global_id is 5516 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0274139b30 con 0x7f0274139ea0 2024-09-16T04:09:54.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2091891604 0 0) 0x7f0254003520 con 0x7f027409eb20 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0274139040 con 0x7f027409eb20 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0268004180 con 0x7f027409eb20 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.603 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 15994495 0 0) 0x7f0274139040 con 0x7f027409eb20 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 >> v1:172.21.15.148:6790/0 conn(0x7f0274139ea0 legacy=0x7f02741d80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 >> v1:172.21.15.148:6789/0 conn(0x7f02741327c0 legacy=0x7f027406e650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 monclient: found mon.b 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027411bb90 con 0x7f027409eb20 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:54.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 5 monclient: authenticate success, global_id 5485 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 10 monclient: _renew_subs 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0274131f50 con 0x7f027409eb20 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 10 monclient: _renew_subs 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0274138620 con 0x7f027409eb20 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0268004430 con 0x7f027409eb20 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0268004800 con 0x7f027409eb20 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0270d99640 4 set_mon_vals no callback set 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.589+0000 7f0262ffd640 10 monclient: dump: 2024-09-16T04:09:54.605 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f02680049a0 con 0x7f027409eb20 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f02789ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f02789ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f02789ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0258005080 con 0x7f027409eb20 2024-09-16T04:09:54.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(119..119 src has 1..119) ==== 5759+0+0 (unknown 3165733462 0 0) 0x7f0268047620 con 0x7f027409eb20 2024-09-16T04:09:54.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.590+0000 7f0271d9b640 10 monclient: get_auth_request con 0x7f0254038c10 auth_method 0 2024-09-16T04:09:54.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.595+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0268015bf0 con 0x7f027409eb20 2024-09-16T04:09:54.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.595+0000 7f0262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.595+0000 7f0262ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:54.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f02789ed640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:09:54.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f02789ed640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:09:54.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f02789ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f0258005370 con 0x7f027409eb20 2024-09-16T04:09:54.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v119) ==== 71+0+0 (unknown 567077014 0 0) 0x7f02680198a0 con 0x7f027409eb20 2024-09-16T04:09:54.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f0262ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:09:54.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.768+0000 7f0262ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:54.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.872+0000 7f02789ed640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:09:54.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.872+0000 7f02789ed640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:09:54.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.872+0000 7f02789ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.872+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f0258000be0 con 0x7f027409eb20 2024-09-16T04:09:54.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.873+0000 7f0262ffd640 1 -- 172.21.15.171:0/1257625223 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v119) ==== 71+0+0 (unknown 567077014 0 0) 0x7f0268005380 con 0x7f027409eb20 2024-09-16T04:09:54.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.873+0000 7f0262ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:09:54.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.873+0000 7f0262ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.874+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 >> v1:172.21.15.171:6812/7327273 conn(0x7f0254038c10 legacy=0x7f025403b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.874+0000 7f02789ed640 10 monclient: shutdown 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.874+0000 7f02789ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.874+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 >> v1:172.21.15.171:6789/0 conn(0x7f027409eb20 legacy=0x7f027406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.874+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 shutdown_connections 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.875+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 >> 172.21.15.171:0/1257625223 conn(0x7f02740a5370 msgr2=0x7f027409b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.875+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 shutdown_connections 2024-09-16T04:09:54.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.875+0000 7f02789ed640 1 -- 172.21.15.171:0/1257625223 wait complete. 2024-09-16T04:09:54.886 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T04:09:54.886 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:54.886 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T04:09:54.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.967+0000 7f2028cdd640 10 monclient: get_monmap_and_config 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.967+0000 7f2028cdd640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.967+0000 7f2028cdd640 10 monclient: monmap: 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:54.968632+0000 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:54.968632+0000 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:54.970 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:54.971 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.968+0000 7f2028cdd640 1 Processor -- start 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.968+0000 7f2028cdd640 1 -- start start 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.968+0000 7f2028cdd640 10 monclient: init 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f2024133700 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient(hunting): picked mon.noname-c con 0x7f20241341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient(hunting): picked mon.noname-a con 0x7f202413a350 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202409da90 con 0x7f202413a350 2024-09-16T04:09:54.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202409dd70 con 0x7f20241341f0 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202409e050 con 0x7f2024133700 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient(hunting): authenticate will time out at 970.474242s 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2021d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f20241341f0 0x7f2024138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52598/0 (socket says 172.21.15.171:52598) 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2021d9b640 1 -- 172.21.15.171:0/4244646302 learned_addr learned my addr 172.21.15.171:0/4244646302 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2022d9d640 1 -- 172.21.15.171:0/4244646302 >> v1:172.21.15.148:6789/0 conn(0x7f202413a350 legacy=0x7f202413c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142504824 0 0) 0x7f202409dd70 con 0x7f20241341f0 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient(hunting): my global_id is 5488 2024-09-16T04:09:54.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f200c003520 con 0x7f20241341f0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3415572912 0 0) 0x7f200c003520 con 0x7f20241341f0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 >> v1:172.21.15.148:6790/0 conn(0x7f2024133700 legacy=0x7f2024133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 >> v1:172.21.15.148:6789/0 conn(0x7f202413a350 legacy=0x7f202413c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: _finish_hunting 0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 monclient: found mon.noname-c 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202409d4f0 con 0x7f20241341f0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: _finish_auth 0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2018001610 con 0x7f20241341f0 2024-09-16T04:09:54.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 5 monclient: authenticate success, global_id 5488 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: dump: 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.981 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2018001970 con 0x7f20241341f0 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 1 -- 172.21.15.171:0/4244646302 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f20180048a0 con 0x7f20241341f0 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f2028cdd640 4 set_mon_vals no callback set 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.969+0000 7f202159a640 10 monclient: dump: 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.982 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 10 monclient: shutdown 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 1 -- 172.21.15.171:0/4244646302 >> v1:172.21.15.171:6789/0 conn(0x7f20241341f0 legacy=0x7f2024138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 1 -- 172.21.15.171:0/4244646302 shutdown_connections 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 1 -- 172.21.15.171:0/4244646302 >> 172.21.15.171:0/4244646302 conn(0x7f20240a5450 msgr2=0x7f20240a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 1 -- 172.21.15.171:0/4244646302 shutdown_connections 2024-09-16T04:09:54.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.970+0000 7f2028cdd640 1 -- 172.21.15.171:0/4244646302 wait complete. 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.971+0000 7f2028cdd640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.971+0000 7f2028cdd640 10 monclient: monmap: 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.971+0000 7f2028cdd640 1 Processor -- start 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.971+0000 7f2028cdd640 1 -- start start 2024-09-16T04:09:54.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.971+0000 7f2028cdd640 10 monclient: init 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.971+0000 7f2028cdd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient(hunting): picked mon.noname-b con 0x7f2024133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient(hunting): picked mon.noname-a con 0x7f20241341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient(hunting): picked mon.noname-c con 0x7f202413a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202411e9f0 con 0x7f20241341f0 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202411ecd0 con 0x7f2024133700 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20240a46b0 con 0x7f202413a350 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient(hunting): authenticate will time out at 970.477110s 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f202259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2024133700 0x7f2024132200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52612/0 (socket says 172.21.15.171:52612) 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2021d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f20241341f0 legacy=0x7f20241ce930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f202259c640 1 -- 172.21.15.171:0/2058100801 learned_addr learned my addr 172.21.15.171:0/2058100801 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2033179140 0 0) 0x7f202411ecd0 con 0x7f2024133700 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): my global_id is 5491 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2004003520 con 0x7f2024133700 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404368159 0 0) 0x7f20240a46b0 con 0x7f202413a350 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): my global_id is 5522 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f202411ecd0 con 0x7f202413a350 2024-09-16T04:09:54.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 385839359 0 0) 0x7f2004003520 con 0x7f2024133700 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20240a46b0 con 0x7f2024133700 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f20140019c0 con 0x7f2024133700 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.987 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 457730171 0 0) 0x7f202411ecd0 con 0x7f202413a350 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2004003520 con 0x7f202413a350 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f201c0043b0 con 0x7f202413a350 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.988 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3496172594 0 0) 0x7f20240a46b0 con 0x7f2024133700 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 >> v1:172.21.15.148:6790/0 conn(0x7f202413a350 legacy=0x7f20241d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 >> v1:172.21.15.148:6789/0 conn(0x7f20241341f0 legacy=0x7f20241ce930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 monclient: found mon.b 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.989 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202411e710 con 0x7f2024133700 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 5 monclient: authenticate success, global_id 5491 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient: _renew_subs 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f202411d5a0 con 0x7f2024133700 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient: _renew_subs 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2028cdd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f202411c600 con 0x7f2024133700 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2014001ca0 con 0x7f2024133700 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2014003a60 con 0x7f2024133700 2024-09-16T04:09:54.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2020d99640 4 set_mon_vals no callback set 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.972+0000 7f2012ffd640 10 monclient: dump: 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:54.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.973+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2014003c80 con 0x7f2024133700 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.973+0000 7f2028cdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.973+0000 7f2028cdd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.973+0000 7f2028cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.973+0000 7f2028cdd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2008005080 con 0x7f2024133700 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.974+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(119..119 src has 1..119) ==== 5759+0+0 (unknown 3165733462 0 0) 0x7f2014047570 con 0x7f2024133700 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.974+0000 7f2021d9b640 10 monclient: get_auth_request con 0x7f2004038ea0 auth_method 0 2024-09-16T04:09:54.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.978+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2014019950 con 0x7f2024133700 2024-09-16T04:09:54.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.978+0000 7f2012ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:54.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:54.978+0000 7f2012ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.152+0000 7f2028cdd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.152+0000 7f2028cdd640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.152+0000 7f2028cdd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.152+0000 7f2028cdd640 1 -- 172.21.15.171:0/2058100801 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f2008005370 con 0x7f2024133700 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.153+0000 7f2012ffd640 1 -- 172.21.15.171:0/2058100801 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v119) ==== 152+0+0 (unknown 120825130 0 0) 0x7f201401e210 con 0x7f2024133700 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.153+0000 7f2012ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.153+0000 7f2012ffd640 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-16T04:09:55.155 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOENT: pool 'fooset' not found 2024-09-16T04:09:55.159 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:55.159 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create fooset 8 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.237+0000 7fee60edd640 10 monclient: get_monmap_and_config 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.237+0000 7fee60edd640 10 monclient: build_initial_monmap 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.237+0000 7fee60edd640 10 monclient: monmap: 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:55.238167+0000 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:55.238167+0000 2024-09-16T04:09:55.239 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:55.240 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:55.240 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:55.240 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:55.240 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:55.240 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:55.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 1 Processor -- start 2024-09-16T04:09:55.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 1 -- start start 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient: init 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient(hunting): picked mon.noname-a con 0x7fee5c09ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient(hunting): picked mon.noname-c con 0x7fee5c0a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient(hunting): picked mon.noname-b con 0x7fee5c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c09fbf0 con 0x7fee5c09ba30 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c0a0e10 con 0x7fee5c0a0750 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c09f640 con 0x7fee5c09e570 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee60edd640 10 monclient(hunting): authenticate will time out at 970.743687s 2024-09-16T04:09:55.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee53fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fee5c0a0750 0x7fee5c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52628/0 (socket says 172.21.15.171:52628) 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee53fff640 1 -- 172.21.15.171:0/171362448 learned_addr learned my addr 172.21.15.171:0/171362448 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.238+0000 7fee5a59c640 1 -- 172.21.15.171:0/171362448 >> v1:172.21.15.148:6789/0 conn(0x7fee5c09ba30 legacy=0x7fee5c0a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1797367936 0 0) 0x7fee5c0a0e10 con 0x7fee5c0a0750 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient(hunting): my global_id is 5494 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee48003520 con 0x7fee5c0a0750 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3976658152 0 0) 0x7fee48003520 con 0x7fee5c0a0750 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 >> v1:172.21.15.148:6790/0 conn(0x7fee5c09e570 legacy=0x7fee5c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 >> v1:172.21.15.148:6789/0 conn(0x7fee5c09ba30 legacy=0x7fee5c0a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:09:55.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 monclient: found mon.noname-c 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c09f810 con 0x7fee5c0a0750 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: _finish_auth 0 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 5 monclient: authenticate success, global_id 5494 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fee44001610 con 0x7fee5c0a0750 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: dump: 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:55.250 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee44001970 con 0x7fee5c0a0750 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 1 -- 172.21.15.171:0/171362448 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fee440048a0 con 0x7fee5c0a0750 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:55.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 4 set_mon_vals no callback set 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee59d9b640 10 monclient: dump: 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:55.252 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 10 monclient: shutdown 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 1 -- 172.21.15.171:0/171362448 >> v1:172.21.15.171:6789/0 conn(0x7fee5c0a0750 legacy=0x7fee5c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 1 -- 172.21.15.171:0/171362448 shutdown_connections 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 1 -- 172.21.15.171:0/171362448 >> 172.21.15.171:0/171362448 conn(0x7fee5c0a52a0 msgr2=0x7fee5c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 1 -- 172.21.15.171:0/171362448 shutdown_connections 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.239+0000 7fee60edd640 1 -- 172.21.15.171:0/171362448 wait complete. 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.240+0000 7fee60edd640 10 monclient: build_initial_monmap 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.240+0000 7fee60edd640 10 monclient: monmap: 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:55.253 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.240+0000 7fee60edd640 1 Processor -- start 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.240+0000 7fee60edd640 1 -- start start 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.240+0000 7fee60edd640 10 monclient: init 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient(hunting): picked mon.noname-c con 0x7fee5c09ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient(hunting): picked mon.noname-b con 0x7fee5c09e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:55.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient(hunting): picked mon.noname-a con 0x7fee5c0a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c135890 con 0x7fee5c0a0750 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c135b70 con 0x7fee5c09e570 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5c12ea70 con 0x7fee5c09ba30 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee60edd640 10 monclient(hunting): authenticate will time out at 970.746319s 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee5ad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fee5c0a0750 legacy=0x7fee5c1d1b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee53fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fee5c09e570 0x7fee5c12e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52634/0 (socket says 172.21.15.171:52634) 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee53fff640 1 -- 172.21.15.171:0/2527838233 learned_addr learned my addr 172.21.15.171:0/2527838233 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:55.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479194283 0 0) 0x7fee5c135b70 con 0x7fee5c09e570 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): my global_id is 5497 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee3c003520 con 0x7fee5c09e570 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231785079 0 0) 0x7fee5c12ea70 con 0x7fee5c09ba30 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): my global_id is 5528 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee5c135b70 con 0x7fee5c09ba30 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1663943243 0 0) 0x7fee3c003520 con 0x7fee5c09e570 2024-09-16T04:09:55.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee5c12ea70 con 0x7fee5c09e570 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fee44001430 con 0x7fee5c09e570 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:55.257 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.241+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 618536042 0 0) 0x7fee5c135b70 con 0x7fee5c09ba30 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee3c003520 con 0x7fee5c09ba30 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 104643572 0 0) 0x7fee5c12ea70 con 0x7fee5c09e570 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 >> v1:172.21.15.148:6790/0 conn(0x7fee5c09ba30 legacy=0x7fee5c12dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 >> v1:172.21.15.148:6789/0 conn(0x7fee5c0a0750 legacy=0x7fee5c1d1b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 monclient: found mon.b 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:55.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c13d850 con 0x7fee5c09e570 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 5 monclient: authenticate success, global_id 5497 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 10 monclient: _renew_subs 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee5c130450 con 0x7fee5c09e570 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 10 monclient: _renew_subs 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee5c131340 con 0x7fee5c09e570 2024-09-16T04:09:55.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee44001970 con 0x7fee5c09e570 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fee44004e60 con 0x7fee5c09e570 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee5959a640 4 set_mon_vals no callback set 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 10 monclient: dump: 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:55.260 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.242+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fee44010290 con 0x7fee5c09e570 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.243+0000 7fee60edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.243+0000 7fee60edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.243+0000 7fee60edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.243+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee40005080 con 0x7fee5c09e570 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.243+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(119..119 src has 1..119) ==== 5759+0+0 (unknown 3165733462 0 0) 0x7fee44047630 con 0x7fee5c09e570 2024-09-16T04:09:55.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.243+0000 7fee5a59c640 10 monclient: get_auth_request con 0x7fee3c038e50 auth_method 0 2024-09-16T04:09:55.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.247+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fee44015ec0 con 0x7fee5c09e570 2024-09-16T04:09:55.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.247+0000 7fee52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:55.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.247+0000 7fee52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.416+0000 7fee60edd640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T04:09:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.416+0000 7fee60edd640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T04:09:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.416+0000 7fee60edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:55.416+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) -- 0x7fee40005b80 con 0x7fee5c09e570 2024-09-16T04:09:56.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.115+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v120) ==== 115+0+0 (unknown 467655282 0 0) 0x7fee440157c0 con 0x7fee5c09e570 2024-09-16T04:09:56.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.115+0000 7fee52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T04:09:56.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.115+0000 7fee52ffd640 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-16T04:09:56.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.218+0000 7fee60edd640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T04:09:56.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.218+0000 7fee60edd640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T04:09:56.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.218+0000 7fee60edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.218+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) -- 0x7fee40001090 con 0x7fee5c09e570 2024-09-16T04:09:56.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.219+0000 7fee52ffd640 1 -- 172.21.15.171:0/2527838233 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v120) ==== 122+0+0 (unknown 697129571 0 0) 0x7fee44044ce0 con 0x7fee5c09e570 2024-09-16T04:09:56.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.219+0000 7fee52ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-16T04:09:56.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.219+0000 7fee52ffd640 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-16T04:09:56.221 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'fooset' already exists 2024-09-16T04:09:56.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 >> v1:172.21.15.171:6812/7327273 conn(0x7fee3c038e50 legacy=0x7fee3c03b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:56.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 10 monclient: shutdown 2024-09-16T04:09:56.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:56.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 >> v1:172.21.15.171:6789/0 conn(0x7fee5c09e570 legacy=0x7fee5c12e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:56.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 shutdown_connections 2024-09-16T04:09:56.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 >> 172.21.15.171:0/2527838233 conn(0x7fee5c0a52a0 msgr2=0x7fee5c1348c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:56.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 shutdown_connections 2024-09-16T04:09:56.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.221+0000 7fee60edd640 1 -- 172.21.15.171:0/2527838233 wait complete. 2024-09-16T04:09:56.233 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create barset 8 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.312+0000 7f0278766640 10 monclient: get_monmap_and_config 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.312+0000 7f0278766640 10 monclient: build_initial_monmap 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.312+0000 7f0278766640 10 monclient: monmap: 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:56.312996+0000 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:56.312996+0000 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:56.314 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:56.315 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:56.315 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.312+0000 7f0278766640 1 Processor -- start 2024-09-16T04:09:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.312+0000 7f0278766640 1 -- start start 2024-09-16T04:09:56.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.312+0000 7f0278766640 10 monclient: init 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient(hunting): picked mon.noname-b con 0x7f027009e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient(hunting): picked mon.noname-a con 0x7f0270136530 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient(hunting): picked mon.noname-c con 0x7f027013a040 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02700a1260 con 0x7f0270136530 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027009d0c0 con 0x7f027013a040 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027009d3a0 con 0x7f027009e1d0 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0278766640 10 monclient(hunting): authenticate will time out at 971.818507s 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0276d03640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f027013a040 0x7f027013c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52642/0 (socket says 172.21.15.171:52642) 2024-09-16T04:09:56.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0276d03640 1 -- 172.21.15.171:0/3048840192 learned_addr learned my addr 172.21.15.171:0/3048840192 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275d01640 1 -- 172.21.15.171:0/3048840192 >> v1:172.21.15.148:6789/0 conn(0x7f0270136530 legacy=0x7f0270138920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 808245576 0 0) 0x7f027009d0c0 con 0x7f027013a040 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 10 monclient(hunting): _negotiate 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 10 monclient(hunting): my global_id is 5500 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f025c003520 con 0x7f027013a040 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.313+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087128100 0 0) 0x7f027009d3a0 con 0x7f027009e1d0 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient(hunting): _negotiate 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:56.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient(hunting): my global_id is 5531 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f027009d0c0 con 0x7f027009e1d0 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2146847229 0 0) 0x7f025c003520 con 0x7f027013a040 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 >> v1:172.21.15.148:6790/0 conn(0x7f027009e1d0 legacy=0x7f02700a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 >> v1:172.21.15.148:6789/0 conn(0x7f0270136530 legacy=0x7f0270138920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: _finish_hunting 0 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 monclient: found mon.noname-c 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027009d680 con 0x7f027013a040 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: _finish_auth 0 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: _check_auth_tickets 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:56.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0268001990 con 0x7f027013a040 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 5 monclient: authenticate success, global_id 5500 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: dump: 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:56.326 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0268001c40 con 0x7f027013a040 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 4 set_mon_vals no callback set 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 1 -- 172.21.15.171:0/3048840192 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0268002010 con 0x7f027013a040 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0275500640 10 monclient: dump: 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:56.327 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 10 monclient: shutdown 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 1 -- 172.21.15.171:0/3048840192 >> v1:172.21.15.171:6789/0 conn(0x7f027013a040 legacy=0x7f027013c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 1 -- 172.21.15.171:0/3048840192 shutdown_connections 2024-09-16T04:09:56.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 1 -- 172.21.15.171:0/3048840192 >> 172.21.15.171:0/3048840192 conn(0x7f02700a52a0 msgr2=0x7f02700a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 1 -- 172.21.15.171:0/3048840192 shutdown_connections 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.314+0000 7f0278766640 1 -- 172.21.15.171:0/3048840192 wait complete. 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.315+0000 7f0278766640 10 monclient: build_initial_monmap 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.315+0000 7f0278766640 10 monclient: monmap: 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:56.329 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.315+0000 7f0278766640 1 Processor -- start 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.315+0000 7f0278766640 1 -- start start 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.315+0000 7f0278766640 10 monclient: init 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient(hunting): picked mon.noname-a con 0x7f027009e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient(hunting): picked mon.noname-b con 0x7f0270136530 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient(hunting): picked mon.noname-c con 0x7f027013a040 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02701235a0 con 0x7f027009e1d0 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0270123880 con 0x7f0270136530 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02700a3fe0 con 0x7f027013a040 2024-09-16T04:09:56.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient(hunting): authenticate will time out at 971.821219s 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0276502640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f027009e1d0 legacy=0x7f0270135310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0275d01640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0270136530 0x7f02701d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52658/0 (socket says 172.21.15.171:52658) 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0275d01640 1 -- 172.21.15.171:0/3031214037 learned_addr learned my addr 172.21.15.171:0/3031214037 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181553533 0 0) 0x7f0270123880 con 0x7f0270136530 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): my global_id is 5503 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0254003520 con 0x7f0270136530 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 216380376 0 0) 0x7f02700a3fe0 con 0x7f027013a040 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:56.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): my global_id is 5534 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0270123880 con 0x7f027013a040 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1728385342 0 0) 0x7f0254003520 con 0x7f0270136530 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02700a3fe0 con 0x7f0270136530 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0260004150 con 0x7f0270136530 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.332 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1873091292 0 0) 0x7f0270123880 con 0x7f027013a040 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0254003520 con 0x7f027013a040 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0268001df0 con 0x7f027013a040 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient(hunting): dump: 2024-09-16T04:09:56.333 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3669670808 0 0) 0x7f02700a3fe0 con 0x7f0270136530 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 >> v1:172.21.15.148:6790/0 conn(0x7f027013a040 legacy=0x7f02701d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 >> v1:172.21.15.148:6789/0 conn(0x7f027009e1d0 legacy=0x7f0270135310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:56.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 monclient: found mon.b 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0270139a70 con 0x7f0270136530 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 5 monclient: authenticate success, global_id 5503 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient: _renew_subs 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.316+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0270123b60 con 0x7f0270136530 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 10 monclient: _renew_subs 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0270135a20 con 0x7f0270136530 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0260004400 con 0x7f0270136530 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02600047d0 con 0x7f0270136530 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0274cff640 4 set_mon_vals no callback set 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 10 monclient: dump: 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:56.336 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0260004970 con 0x7f0270136530 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.317+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0258005080 con 0x7f0270136530 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.318+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(120..120 src has 1..120) ==== 6104+0+0 (unknown 2916767788 0 0) 0x7f02600476e0 con 0x7f0270136530 2024-09-16T04:09:56.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.318+0000 7f0276502640 10 monclient: get_auth_request con 0x7f0254038ef0 auth_method 0 2024-09-16T04:09:56.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.322+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0260015b50 con 0x7f0270136530 2024-09-16T04:09:56.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.322+0000 7f0266ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:56.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.322+0000 7f0266ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:56.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.490+0000 7f0278766640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T04:09:56.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.490+0000 7f0278766640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T04:09:56.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.490+0000 7f0278766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:56.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:56.490+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) -- 0x7f0258005b80 con 0x7f0270136530 2024-09-16T04:09:57.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.117+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v121) ==== 115+0+0 (unknown 2564846363 0 0) 0x7f0260019800 con 0x7f0270136530 2024-09-16T04:09:57.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.117+0000 7f0266ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T04:09:57.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.117+0000 7f0266ffd640 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-16T04:09:57.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0278766640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T04:09:57.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0278766640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T04:09:57.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0278766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) -- 0x7f0258001090 con 0x7f0270136530 2024-09-16T04:09:57.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0266ffd640 1 -- 172.21.15.171:0/3031214037 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v121) ==== 122+0+0 (unknown 181482528 0 0) 0x7f0260015290 con 0x7f0270136530 2024-09-16T04:09:57.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0266ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-16T04:09:57.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.227+0000 7f0266ffd640 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-16T04:09:57.230 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'barset' already exists 2024-09-16T04:09:57.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 >> v1:172.21.15.171:6812/7327273 conn(0x7f0254038ef0 legacy=0x7f025403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:57.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 10 monclient: shutdown 2024-09-16T04:09:57.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:57.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 >> v1:172.21.15.171:6789/0 conn(0x7f0270136530 legacy=0x7f02701d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:57.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 shutdown_connections 2024-09-16T04:09:57.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 >> 172.21.15.171:0/3031214037 conn(0x7f02700a52a0 msgr2=0x7f0270138c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:57.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 shutdown_connections 2024-09-16T04:09:57.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.229+0000 7f0278766640 1 -- 172.21.15.171:0/3031214037 wait complete. 2024-09-16T04:09:57.241 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set barset size 3 2024-09-16T04:09:57.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.322+0000 7f653546e640 10 monclient: get_monmap_and_config 2024-09-16T04:09:57.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.322+0000 7f653546e640 10 monclient: build_initial_monmap 2024-09-16T04:09:57.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.322+0000 7f653546e640 10 monclient: monmap: 2024-09-16T04:09:57.324 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:57.324 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:57.323074+0000 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:57.323074+0000 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.322+0000 7f653546e640 1 Processor -- start 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.322+0000 7f653546e640 1 -- start start 2024-09-16T04:09:57.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.322+0000 7f653546e640 10 monclient: init 2024-09-16T04:09:57.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient(hunting): picked mon.noname-a con 0x7f653009b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient(hunting): picked mon.noname-c con 0x7f6530136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient(hunting): picked mon.noname-b con 0x7f653013d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:57.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6530073a20 con 0x7f653009b920 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6530099920 con 0x7f6530136da0 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6530099c00 con 0x7f653013d320 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f653546e640 10 monclient(hunting): authenticate will time out at 972.828459s 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652e7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6530136da0 0x7f653013bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56830/0 (socket says 172.21.15.171:56830) 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652e7fc640 1 -- 172.21.15.171:0/721525718 learned_addr learned my addr 172.21.15.171:0/721525718 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652effd640 1 -- 172.21.15.171:0/721525718 >> v1:172.21.15.148:6789/0 conn(0x7f653009b920 legacy=0x7f653009bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339449804 0 0) 0x7f6530099920 con 0x7f6530136da0 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:57.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient(hunting): my global_id is 5506 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6518003520 con 0x7f6530136da0 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 53075778 0 0) 0x7f6518003520 con 0x7f6530136da0 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 >> v1:172.21.15.148:6790/0 conn(0x7f653013d320 legacy=0x7f653013f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 >> v1:172.21.15.148:6789/0 conn(0x7f653009b920 legacy=0x7f653009bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 monclient: found mon.noname-c 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6530099ee0 con 0x7f6530136da0 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:57.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:57.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:57.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.323+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6524001610 con 0x7f6530136da0 2024-09-16T04:09:57.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:57.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:57.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: dump: 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 5 monclient: authenticate success, global_id 5506 2024-09-16T04:09:57.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6524001900 con 0x7f6530136da0 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 1 -- 172.21.15.171:0/721525718 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6524001cd0 con 0x7f6530136da0 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 4 set_mon_vals no callback set 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f652dffb640 10 monclient: dump: 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.338 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 10 monclient: shutdown 2024-09-16T04:09:57.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 1 -- 172.21.15.171:0/721525718 >> v1:172.21.15.171:6789/0 conn(0x7f6530136da0 legacy=0x7f653013bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 1 -- 172.21.15.171:0/721525718 shutdown_connections 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 1 -- 172.21.15.171:0/721525718 >> 172.21.15.171:0/721525718 conn(0x7f65300a2150 msgr2=0x7f65300a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 1 -- 172.21.15.171:0/721525718 shutdown_connections 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.324+0000 7f653546e640 1 -- 172.21.15.171:0/721525718 wait complete. 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.325+0000 7f653546e640 10 monclient: build_initial_monmap 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.325+0000 7f653546e640 10 monclient: monmap: 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.325+0000 7f653546e640 1 Processor -- start 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.325+0000 7f653546e640 1 -- start start 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.325+0000 7f653546e640 10 monclient: init 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient(hunting): picked mon.noname-b con 0x7f653009b920 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:57.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient(hunting): picked mon.noname-c con 0x7f6530136da0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:57.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient(hunting): picked mon.noname-a con 0x7f653013d320 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:57.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653012be20 con 0x7f653013d320 2024-09-16T04:09:57.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653012a900 con 0x7f653009b920 2024-09-16T04:09:57.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653012c100 con 0x7f6530136da0 2024-09-16T04:09:57.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:57.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient(hunting): authenticate will time out at 972.831176s 2024-09-16T04:09:57.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f652effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f653009b920 0x7f6530098630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56832/0 (socket says 172.21.15.171:56832) 2024-09-16T04:09:57.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f652effd640 1 -- 172.21.15.171:0/2287799082 learned_addr learned my addr 172.21.15.171:0/2287799082 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:57.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f652f7fe640 1 -- 172.21.15.171:0/2287799082 >> v1:172.21.15.148:6789/0 conn(0x7f653013d320 legacy=0x7f65301d3e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:57.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744767062 0 0) 0x7f653012a900 con 0x7f653009b920 2024-09-16T04:09:57.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:57.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:57.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:57.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): my global_id is 5509 2024-09-16T04:09:57.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f651c003520 con 0x7f653009b920 2024-09-16T04:09:57.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3086136172 0 0) 0x7f653012c100 con 0x7f6530136da0 2024-09-16T04:09:57.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): my global_id is 5540 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f653012a900 con 0x7f6530136da0 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2439609989 0 0) 0x7f651c003520 con 0x7f653009b920 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f653012c100 con 0x7f653009b920 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6520004500 con 0x7f653009b920 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): dump: 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:57.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3961757659 0 0) 0x7f653012c100 con 0x7f653009b920 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 >> v1:172.21.15.148:6790/0 conn(0x7f6530136da0 legacy=0x7f65301d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:57.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 >> v1:172.21.15.148:6789/0 conn(0x7f653013d320 legacy=0x7f65301d3e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 monclient: found mon.b 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653013c2d0 con 0x7f653009b920 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 5 monclient: authenticate success, global_id 5509 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient: _renew_subs 2024-09-16T04:09:57.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6530098d40 con 0x7f653009b920 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient: _renew_subs 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f653013bba0 con 0x7f653009b920 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6520004760 con 0x7f653009b920 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6520004b30 con 0x7f653009b920 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f65177fe640 10 monclient: dump: 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:57.348 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:57.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:57.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:57.351 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.326+0000 7f652d7fa640 4 set_mon_vals no callback set 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6520004d50 con 0x7f653009b920 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f653546e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f653546e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f653546e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6530098f10 con 0x7f653009b920 2024-09-16T04:09:57.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(121..121 src has 1..121) ==== 6449+0+0 (unknown 3343353862 0 0) 0x7f6520047c00 con 0x7f653009b920 2024-09-16T04:09:57.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.327+0000 7f652e7fc640 10 monclient: get_auth_request con 0x7f651c038be0 auth_method 0 2024-09-16T04:09:57.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.332+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6530098f10 con 0x7f653009b920 2024-09-16T04:09:57.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.332+0000 7f65177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:57.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.332+0000 7f65177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:57.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.498+0000 7f653546e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T04:09:57.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.498+0000 7f653546e640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T04:09:57.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.498+0000 7f653546e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:57.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:57.498+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) -- 0x7f653009c660 con 0x7f653009b920 2024-09-16T04:09:58.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.118+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v122) ==== 126+0+0 (unknown 1015358924 0 0) 0x7f6520016010 con 0x7f653009b920 2024-09-16T04:09:58.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.118+0000 7f65177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T04:09:58.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.118+0000 7f65177fe640 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-16T04:09:58.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.226+0000 7f653546e640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T04:09:58.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.226+0000 7f653546e640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T04:09:58.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.226+0000 7f653546e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:58.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:58.226+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) -- 0x7f653013ccd0 con 0x7f653009b920 2024-09-16T04:09:59.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.120+0000 7f65177fe640 1 -- 172.21.15.171:0/2287799082 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v123) ==== 126+0+0 (unknown 3189611222 0 0) 0x7f6520015910 con 0x7f653009b920 2024-09-16T04:09:59.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.120+0000 7f65177fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-16T04:09:59.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.120+0000 7f65177fe640 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-16T04:09:59.123 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 10 size to 3 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 >> v1:172.21.15.171:6812/7327273 conn(0x7f651c038be0 legacy=0x7f651c03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 10 monclient: shutdown 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 >> v1:172.21.15.171:6789/0 conn(0x7f653009b920 legacy=0x7f6530098630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 shutdown_connections 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 >> 172.21.15.171:0/2287799082 conn(0x7f65300a2150 msgr2=0x7f65300a0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 shutdown_connections 2024-09-16T04:09:59.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.124+0000 7f653546e640 1 -- 172.21.15.171:0/2287799082 wait complete. 2024-09-16T04:09:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T04:09:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:09:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.227+0000 7fc495a77640 10 monclient: get_monmap_and_config 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.227+0000 7fc495a77640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.227+0000 7fc495a77640 10 monclient: monmap: 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:59.227907+0000 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:59.227907+0000 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:59.229 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:59.230 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:59.230 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 1 Processor -- start 2024-09-16T04:09:59.236 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 1 -- start start 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient: init 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient(hunting): picked mon.noname-c con 0x7fc490136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient(hunting): picked mon.noname-b con 0x7fc490137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient(hunting): picked mon.noname-a con 0x7fc4901375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc49009b2c0 con 0x7fc4901375a0 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc49009b5a0 con 0x7fc490136d60 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4900a6640 con 0x7fc490137140 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc495a77640 10 monclient(hunting): authenticate will time out at 974.733701s 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc48f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc490136d60 0x7fc49009bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56838/0 (socket says 172.21.15.171:56838) 2024-09-16T04:09:59.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc48f7fe640 1 -- 172.21.15.171:0/2390774767 learned_addr learned my addr 172.21.15.171:0/2390774767 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.228+0000 7fc48ffff640 1 -- 172.21.15.171:0/2390774767 >> v1:172.21.15.148:6789/0 conn(0x7fc4901375a0 legacy=0x7fc49013f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175689347 0 0) 0x7fc49009b5a0 con 0x7fc490136d60 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient(hunting): my global_id is 5512 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc47c003520 con 0x7fc490136d60 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1617117176 0 0) 0x7fc47c003520 con 0x7fc490136d60 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 >> v1:172.21.15.148:6790/0 conn(0x7fc490137140 legacy=0x7fc49013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 >> v1:172.21.15.148:6789/0 conn(0x7fc4901375a0 legacy=0x7fc49013f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 monclient: found mon.noname-c 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc490143b20 con 0x7fc490136d60 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc480001610 con 0x7fc490136d60 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 5 monclient: authenticate success, global_id 5512 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: dump: 2024-09-16T04:09:59.239 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc480001900 con 0x7fc490136d60 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 1 -- 172.21.15.171:0/2390774767 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc480001cd0 con 0x7fc490136d60 2024-09-16T04:09:59.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 4 set_mon_vals no callback set 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc48effd640 10 monclient: dump: 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.241 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 10 monclient: shutdown 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 1 -- 172.21.15.171:0/2390774767 >> v1:172.21.15.171:6789/0 conn(0x7fc490136d60 legacy=0x7fc49009bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.229+0000 7fc495a77640 1 -- 172.21.15.171:0/2390774767 shutdown_connections 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.230+0000 7fc495a77640 1 -- 172.21.15.171:0/2390774767 >> 172.21.15.171:0/2390774767 conn(0x7fc4900a9ae0 msgr2=0x7fc4900a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.230+0000 7fc495a77640 1 -- 172.21.15.171:0/2390774767 shutdown_connections 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.230+0000 7fc495a77640 1 -- 172.21.15.171:0/2390774767 wait complete. 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.230+0000 7fc495a77640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.230+0000 7fc495a77640 10 monclient: monmap: 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:59.242 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.230+0000 7fc495a77640 1 Processor -- start 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 1 -- start start 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient: init 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:59.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient(hunting): picked mon.noname-b con 0x7fc490136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient(hunting): picked mon.noname-a con 0x7fc490137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient(hunting): picked mon.noname-c con 0x7fc4901375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc490136810 con 0x7fc490137140 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4900a8410 con 0x7fc490136d60 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc49012ec20 con 0x7fc4901375a0 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc495a77640 10 monclient(hunting): authenticate will time out at 974.736498s 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc490136d60 0x7fc49009e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56852/0 (socket says 172.21.15.171:56852) 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48f7fe640 1 -- 172.21.15.171:0/1821011442 learned_addr learned my addr 172.21.15.171:0/1821011442 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc487fff640 1 -- 172.21.15.171:0/1821011442 >> v1:172.21.15.148:6789/0 conn(0x7fc490137140 legacy=0x7fc49009e820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 224550469 0 0) 0x7fc490143b20 con 0x7fc490136d60 2024-09-16T04:09:59.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 10 monclient(hunting): my global_id is 5515 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc470003520 con 0x7fc490136d60 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.231+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3757267124 0 0) 0x7fc470003520 con 0x7fc490136d60 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc490143b20 con 0x7fc490136d60 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc4800015d0 con 0x7fc490136d60 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): dump: 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.245 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514316323 0 0) 0x7fc49012ec20 con 0x7fc4901375a0 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient(hunting): my global_id is 5546 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc470003520 con 0x7fc4901375a0 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1793484092 0 0) 0x7fc490143b20 con 0x7fc490136d60 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 >> v1:172.21.15.148:6790/0 conn(0x7fc4901375a0 legacy=0x7fc4901d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 >> v1:172.21.15.148:6789/0 conn(0x7fc490137140 legacy=0x7fc49009e820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 monclient: found mon.b 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49009ef30 con 0x7fc490136d60 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 5 monclient: authenticate success, global_id 5515 2024-09-16T04:09:59.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 10 monclient: _renew_subs 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4900a03b0 con 0x7fc490136d60 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 10 monclient: _renew_subs 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc495a77640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc49012ef00 con 0x7fc490136d60 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc480004d40 con 0x7fc490136d60 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc480005110 con 0x7fc490136d60 2024-09-16T04:09:59.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48cff9640 10 monclient: dump: 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.249 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.232+0000 7fc48e7fc640 4 set_mon_vals no callback set 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc4800052b0 con 0x7fc490136d60 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc495a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc495a77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc495a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc495a77640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc474005080 con 0x7fc490136d60 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(123..123 src has 1..123) ==== 6449+0+0 (unknown 4167813095 0 0) 0x7fc480047a10 con 0x7fc490136d60 2024-09-16T04:09:59.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.233+0000 7fc487fff640 10 monclient: get_auth_request con 0x7fc470039070 auth_method 0 2024-09-16T04:09:59.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.238+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc4900a8410 con 0x7fc490136d60 2024-09-16T04:09:59.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.238+0000 7fc48cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.238+0000 7fc48cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:59.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc495a77640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:09:59.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc495a77640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:09:59.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc495a77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc495a77640 1 -- 172.21.15.171:0/1821011442 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7fc474005370 con 0x7fc490136d60 2024-09-16T04:09:59.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc48cff9640 1 -- 172.21.15.171:0/1821011442 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v123) ==== 160+0+0 (unknown 3825085323 0 0) 0x7fc480015ea0 con 0x7fc490136d60 2024-09-16T04:09:59.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc48cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:09:59.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.410+0000 7fc48cff9640 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-16T04:09:59.413 INFO:tasks.workunit.client.0.smithi171.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-16T04:09:59.416 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:09:59.416 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.495+0000 7f7b15118640 10 monclient: get_monmap_and_config 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.495+0000 7f7b15118640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.495+0000 7f7b15118640 10 monclient: monmap: 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:09:59.496037+0000 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:09:59.496037+0000 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.497 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:59.498 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:09:59.498 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:09:59.498 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.495+0000 7f7b15118640 1 Processor -- start 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.495+0000 7f7b15118640 1 -- start start 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.495+0000 7f7b15118640 10 monclient: init 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient(hunting): picked mon.noname-a con 0x7f7b10136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient(hunting): picked mon.noname-c con 0x7f7b10137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:59.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient(hunting): picked mon.noname-b con 0x7f7b101375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b1009b2c0 con 0x7f7b10136d60 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b1009b5a0 con 0x7f7b10137140 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b100a6640 con 0x7f7b101375a0 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b15118640 10 monclient(hunting): authenticate will time out at 975.001457s 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0ed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7b10136d60 legacy=0x7f7b1009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0e59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7b10137140 0x7f7b1013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56862/0 (socket says 172.21.15.171:56862) 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0e59c640 1 -- 172.21.15.171:0/2775364093 learned_addr learned my addr 172.21.15.171:0/2775364093 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1938607737 0 0) 0x7f7b1009b5a0 con 0x7f7b10137140 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient(hunting): my global_id is 5518 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7af8003520 con 0x7f7b10137140 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3740954117 0 0) 0x7f7af8003520 con 0x7f7b10137140 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 >> v1:172.21.15.148:6790/0 conn(0x7f7b101375a0 legacy=0x7f7b1013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 >> v1:172.21.15.148:6789/0 conn(0x7f7b10136d60 legacy=0x7f7b1009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 monclient: found mon.noname-c 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b10143b20 con 0x7f7b10137140 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b04001610 con 0x7f7b10137140 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.496+0000 7f7b0dd9b640 10 monclient: dump: 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.508 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 5 monclient: authenticate success, global_id 5518 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b04001900 con 0x7f7b10137140 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b0dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b0dd9b640 1 -- 172.21.15.171:0/2775364093 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b04001cd0 con 0x7f7b10137140 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 4 set_mon_vals no callback set 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b0dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b0dd9b640 10 monclient: dump: 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.509 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 10 monclient: shutdown 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 1 -- 172.21.15.171:0/2775364093 >> v1:172.21.15.171:6789/0 conn(0x7f7b10137140 legacy=0x7f7b1013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 1 -- 172.21.15.171:0/2775364093 shutdown_connections 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 1 -- 172.21.15.171:0/2775364093 >> 172.21.15.171:0/2775364093 conn(0x7f7b100a9ae0 msgr2=0x7f7b100a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 1 -- 172.21.15.171:0/2775364093 shutdown_connections 2024-09-16T04:09:59.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.497+0000 7f7b15118640 1 -- 172.21.15.171:0/2775364093 wait complete. 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.498+0000 7f7b15118640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.498+0000 7f7b15118640 10 monclient: monmap: 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.498+0000 7f7b15118640 1 Processor -- start 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.498+0000 7f7b15118640 1 -- start start 2024-09-16T04:09:59.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.498+0000 7f7b15118640 10 monclient: init 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient(hunting): picked mon.noname-a con 0x7f7b10136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient(hunting): picked mon.noname-c con 0x7f7b10137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient(hunting): picked mon.noname-b con 0x7f7b101375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b1009f350 con 0x7f7b10136d60 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b101391a0 con 0x7f7b101375a0 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b0ed9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7b10136d60 legacy=0x7f7b1009e530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b10139760 con 0x7f7b10137140 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b15118640 10 monclient(hunting): authenticate will time out at 975.004396s 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b0f59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7b101375a0 0x7f7b101d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56876/0 (socket says 172.21.15.171:56876) 2024-09-16T04:09:59.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7b0f59e640 1 -- 172.21.15.171:0/4104206406 learned_addr learned my addr 172.21.15.171:0/4104206406 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872726420 0 0) 0x7f7b101391a0 con 0x7f7b101375a0 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): my global_id is 5521 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7afc003520 con 0x7f7b101375a0 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526852037 0 0) 0x7f7b10139760 con 0x7f7b10137140 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): my global_id is 5552 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b101391a0 con 0x7f7b10137140 2024-09-16T04:09:59.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3606355858 0 0) 0x7f7afc003520 con 0x7f7b101375a0 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b10139760 con 0x7f7b101375a0 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b08001cc0 con 0x7f7b101375a0 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): dump: 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.514 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2557656501 0 0) 0x7f7b101391a0 con 0x7f7b10137140 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7afc003520 con 0x7f7b10137140 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b04001430 con 0x7f7b10137140 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.499+0000 7f7af77fe640 10 monclient(hunting): dump: 2024-09-16T04:09:59.515 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3629468098 0 0) 0x7f7b10139760 con 0x7f7b101375a0 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 >> v1:172.21.15.148:6790/0 conn(0x7f7b10137140 legacy=0x7f7b1009ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 >> v1:172.21.15.148:6789/0 conn(0x7f7b10136d60 legacy=0x7f7b1009e530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:59.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 monclient: found mon.b 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b10139480 con 0x7f7b101375a0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 5 monclient: authenticate success, global_id 5521 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 10 monclient: _renew_subs 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b101382b0 con 0x7f7b101375a0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 10 monclient: _renew_subs 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b10136390 con 0x7f7b101375a0 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b08002070 con 0x7f7b101375a0 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b08003de0 con 0x7f7b101375a0 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 10 monclient: dump: 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:09:59.518 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7b0d59a640 4 set_mon_vals no callback set 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.500+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7b08004000 con 0x7f7b101375a0 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.501+0000 7f7b15118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.501+0000 7f7b15118640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.501+0000 7f7b15118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.501+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af0005080 con 0x7f7b101375a0 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.501+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(123..123 src has 1..123) ==== 6449+0+0 (unknown 4167813095 0 0) 0x7f7b08047b80 con 0x7f7b101375a0 2024-09-16T04:09:59.519 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.501+0000 7f7b0ed9d640 10 monclient: get_auth_request con 0x7f7afc038ea0 auth_method 0 2024-09-16T04:09:59.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.505+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7b08016010 con 0x7f7b101375a0 2024-09-16T04:09:59.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.505+0000 7f7af77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.520 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.505+0000 7f7af77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:59.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.677+0000 7f7b15118640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T04:09:59.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.677+0000 7f7b15118640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T04:09:59.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.677+0000 7f7b15118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:09:59.679 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:09:59.677+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) -- 0x7f7af0005370 con 0x7f7b101375a0 2024-09-16T04:10:00.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.122+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v124) ==== 109+0+0 (unknown 1662038444 0 0) 0x7f7b08019cc0 con 0x7f7b101375a0 2024-09-16T04:10:00.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.122+0000 7f7af77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T04:10:00.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.122+0000 7f7af77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:00.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7b15118640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T04:10:00.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7b15118640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T04:10:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7b15118640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) -- 0x7f7af0005bd0 con 0x7f7b101375a0 2024-09-16T04:10:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7af77fe640 1 -- 172.21.15.171:0/4104206406 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v124) ==== 153+0+0 (unknown 2625356952 0 0) 0x7f7b0801e580 con 0x7f7b101375a0 2024-09-16T04:10:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7af77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-16T04:10:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.230+0000 7f7af77fe640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-16T04:10:00.233 INFO:tasks.workunit.client.0.smithi171.stderr:weight-set for pool 'fooset' already created 2024-09-16T04:10:00.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 >> v1:172.21.15.171:6812/7327273 conn(0x7f7afc038ea0 legacy=0x7f7afc03b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 10 monclient: shutdown 2024-09-16T04:10:00.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:00.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 >> v1:172.21.15.171:6789/0 conn(0x7f7b101375a0 legacy=0x7f7b101d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 shutdown_connections 2024-09-16T04:10:00.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 >> 172.21.15.171:0/4104206406 conn(0x7f7b100a9ae0 msgr2=0x7f7b10072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:00.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 shutdown_connections 2024-09-16T04:10:00.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.232+0000 7f7b15118640 1 -- 172.21.15.171:0/4104206406 wait complete. 2024-09-16T04:10:00.243 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create barset positional 2024-09-16T04:10:00.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.323+0000 7ff85ea53640 10 monclient: get_monmap_and_config 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.323+0000 7ff85ea53640 10 monclient: build_initial_monmap 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.323+0000 7ff85ea53640 10 monclient: monmap: 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:00.324719+0000 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:00.324719+0000 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:00.326 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.324+0000 7ff85ea53640 1 Processor -- start 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.324+0000 7ff85ea53640 1 -- start start 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.324+0000 7ff85ea53640 10 monclient: init 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient(hunting): picked mon.noname-b con 0x7ff85809f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient(hunting): picked mon.noname-a con 0x7ff85809fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:00.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient(hunting): picked mon.noname-c con 0x7ff8581395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8580a1490 con 0x7ff85809fba0 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85809d1b0 con 0x7ff8581395d0 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85809d490 con 0x7ff85809f7c0 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient(hunting): authenticate will time out at 975.830271s 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff8577fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff85809fba0 legacy=0x7ff8580a0030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85cff0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff8581395d0 0x7ff85813b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56884/0 (socket says 172.21.15.171:56884) 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85cff0640 1 -- 172.21.15.171:0/1303742282 learned_addr learned my addr 172.21.15.171:0/1303742282 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 396079682 0 0) 0x7ff85809d1b0 con 0x7ff8581395d0 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): my global_id is 5524 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff840003520 con 0x7ff8581395d0 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945494701 0 0) 0x7ff85809d490 con 0x7ff85809f7c0 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient(hunting): my global_id is 5555 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff85809d1b0 con 0x7ff85809f7c0 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1457964368 0 0) 0x7ff840003520 con 0x7ff8581395d0 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 >> v1:172.21.15.148:6790/0 conn(0x7ff85809f7c0 legacy=0x7ff85809de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 >> v1:172.21.15.148:6789/0 conn(0x7ff85809fba0 legacy=0x7ff8580a0030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:00.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 monclient: found mon.noname-c 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85809bca0 con 0x7ff8581395d0 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff850001990 con 0x7ff8581395d0 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: dump: 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.336 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 5 monclient: authenticate success, global_id 5524 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff850001cb0 con 0x7ff8581395d0 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 1 -- 172.21.15.171:0/1303742282 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff850002080 con 0x7ff8581395d0 2024-09-16T04:10:00.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff856ffd640 10 monclient: dump: 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:00.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.325+0000 7ff85ea53640 4 set_mon_vals no callback set 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 10 monclient: shutdown 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 1 -- 172.21.15.171:0/1303742282 >> v1:172.21.15.171:6789/0 conn(0x7ff8581395d0 legacy=0x7ff85813b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 1 -- 172.21.15.171:0/1303742282 shutdown_connections 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 1 -- 172.21.15.171:0/1303742282 >> 172.21.15.171:0/1303742282 conn(0x7ff8580a5450 msgr2=0x7ff8580a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 1 -- 172.21.15.171:0/1303742282 shutdown_connections 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.326+0000 7ff85ea53640 1 -- 172.21.15.171:0/1303742282 wait complete. 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient: build_initial_monmap 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient: monmap: 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:00.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 1 Processor -- start 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 1 -- start start 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient: init 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient(hunting): picked mon.noname-c con 0x7ff85809f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:00.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient(hunting): picked mon.noname-b con 0x7ff85809fba0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.327+0000 7ff85ea53640 10 monclient(hunting): picked mon.noname-a con 0x7ff8581395d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff858120a90 con 0x7ff8581395d0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff858120d70 con 0x7ff85809fba0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8581369a0 con 0x7ff85809f7c0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 10 monclient(hunting): authenticate will time out at 975.833027s 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85cff0640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff8581395d0 legacy=0x7ff8581d3c00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff8577fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff85809fba0 0x7ff858136290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56892/0 (socket says 172.21.15.171:56892) 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff8577fe640 1 -- 172.21.15.171:0/38950339 learned_addr learned my addr 172.21.15.171:0/38950339 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2583842535 0 0) 0x7ff858120d70 con 0x7ff85809fba0 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): my global_id is 5527 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff844003520 con 0x7ff85809fba0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538164912 0 0) 0x7ff8581369a0 con 0x7ff85809f7c0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): my global_id is 5558 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff858120d70 con 0x7ff85809f7c0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3567692764 0 0) 0x7ff844003520 con 0x7ff85809fba0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8581369a0 con 0x7ff85809fba0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff84c004180 con 0x7ff85809fba0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:00.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): dump: 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2639229824 0 0) 0x7ff8581369a0 con 0x7ff85809fba0 2024-09-16T04:10:00.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 >> v1:172.21.15.148:6790/0 conn(0x7ff85809f7c0 legacy=0x7ff858135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 >> v1:172.21.15.148:6789/0 conn(0x7ff8581395d0 legacy=0x7ff8581d3c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 monclient: found mon.b 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8580a3e60 con 0x7ff85809fba0 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: _finish_auth 0 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 5 monclient: authenticate success, global_id 5527 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 10 monclient: _renew_subs 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff85809c0b0 con 0x7ff85809fba0 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 10 monclient: _renew_subs 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff8581378a0 con 0x7ff85809fba0 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff84c0044a0 con 0x7ff85809fba0 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff84c004870 con 0x7ff85809fba0 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff8567fc640 4 set_mon_vals no callback set 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.328+0000 7ff854ff9640 10 monclient: dump: 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.345 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff84c004a10 con 0x7ff85809fba0 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff85ea53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff85ea53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff85ea53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(124..124 src has 1..124) ==== 6781+0+0 (unknown 2298488555 0 0) 0x7ff84c047a30 con 0x7ff85809fba0 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff838005080 con 0x7ff85809fba0 2024-09-16T04:10:00.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.329+0000 7ff857fff640 10 monclient: get_auth_request con 0x7ff844038fb0 auth_method 0 2024-09-16T04:10:00.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.334+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff84c015cc0 con 0x7ff85809fba0 2024-09-16T04:10:00.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.334+0000 7ff854ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:00.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.334+0000 7ff854ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:00.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.506+0000 7ff85ea53640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T04:10:00.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.506+0000 7ff85ea53640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T04:10:00.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.506+0000 7ff85ea53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:00.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.506+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) -- 0x7ff838005370 con 0x7ff85809fba0 2024-09-16T04:10:00.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.994+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v125) ==== 115+0+0 (unknown 3580137992 0 0) 0x7ff84c0155c0 con 0x7ff85809fba0 2024-09-16T04:10:00.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.994+0000 7ff854ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T04:10:00.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:00.994+0000 7ff854ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff85ea53640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff85ea53640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff85ea53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) -- 0x7ff838005ab0 con 0x7ff85809fba0 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff854ff9640 1 -- 172.21.15.171:0/38950339 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v125) ==== 159+0+0 (unknown 1214491048 0 0) 0x7ff84c01e170 con 0x7ff85809fba0 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff854ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.102+0000 7ff854ff9640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-16T04:10:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:weight-set for pool 'barset' already created 2024-09-16T04:10:01.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 >> v1:172.21.15.171:6812/7327273 conn(0x7ff844038fb0 legacy=0x7ff84403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 10 monclient: shutdown 2024-09-16T04:10:01.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 >> v1:172.21.15.171:6789/0 conn(0x7ff85809fba0 legacy=0x7ff858136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 shutdown_connections 2024-09-16T04:10:01.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 >> 172.21.15.171:0/38950339 conn(0x7ff8580a5450 msgr2=0x7ff858135210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 shutdown_connections 2024-09-16T04:10:01.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.104+0000 7ff85ea53640 1 -- 172.21.15.171:0/38950339 wait complete. 2024-09-16T04:10:01.116 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:01.116 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep fooset 2024-09-16T04:10:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.193+0000 7f16d99c4640 10 monclient: get_monmap_and_config 2024-09-16T04:10:01.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.193+0000 7f16d99c4640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.193+0000 7f16d99c4640 10 monclient: monmap: 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:01.194685+0000 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:01.194685+0000 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:01.196 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.194+0000 7f16d99c4640 1 Processor -- start 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.194+0000 7f16d99c4640 1 -- start start 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.194+0000 7f16d99c4640 10 monclient: init 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient(hunting): picked mon.noname-b con 0x7f16d4136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient(hunting): picked mon.noname-c con 0x7f16d4139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient(hunting): picked mon.noname-a con 0x7f16d413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d40a1340 con 0x7f16d413d620 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d4137650 con 0x7f16d4139b10 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d40a0aa0 con 0x7f16d4136b60 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient(hunting): authenticate will time out at 976.700144s 2024-09-16T04:10:01.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d27fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f16d4139b10 0x7f16d413bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56904/0 (socket says 172.21.15.171:56904) 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d27fc640 1 -- 172.21.15.171:0/3647794212 learned_addr learned my addr 172.21.15.171:0/3647794212 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d37fe640 1 -- 172.21.15.171:0/3647794212 >> v1:172.21.15.148:6789/0 conn(0x7f16d413d620 legacy=0x7f16d413fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3151121240 0 0) 0x7f16d4137650 con 0x7f16d4139b10 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient(hunting): my global_id is 5530 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16bc003520 con 0x7f16d4139b10 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3281334852 0 0) 0x7f16bc003520 con 0x7f16d4139b10 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 >> v1:172.21.15.148:6790/0 conn(0x7f16d4136b60 legacy=0x7f16d4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 >> v1:172.21.15.148:6789/0 conn(0x7f16d413d620 legacy=0x7f16d413fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 monclient: found mon.noname-c 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d40a0550 con 0x7f16d4139b10 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16c8001610 con 0x7f16d4139b10 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 5 monclient: authenticate success, global_id 5530 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: dump: 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16c8001970 con 0x7f16d4139b10 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 1 -- 172.21.15.171:0/3647794212 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16c80048a0 con 0x7f16d4139b10 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d1ffb640 10 monclient: dump: 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.210 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.195+0000 7f16d99c4640 4 set_mon_vals no callback set 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 10 monclient: shutdown 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 1 -- 172.21.15.171:0/3647794212 >> v1:172.21.15.171:6789/0 conn(0x7f16d4139b10 legacy=0x7f16d413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 1 -- 172.21.15.171:0/3647794212 shutdown_connections 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 1 -- 172.21.15.171:0/3647794212 >> 172.21.15.171:0/3647794212 conn(0x7f16d40a98b0 msgr2=0x7f16d40a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 1 -- 172.21.15.171:0/3647794212 shutdown_connections 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 1 -- 172.21.15.171:0/3647794212 wait complete. 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.196+0000 7f16d99c4640 10 monclient: monmap: 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:01.212 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 1 Processor -- start 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 1 -- start start 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient: init 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient(hunting): picked mon.noname-c con 0x7f16d4136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient(hunting): picked mon.noname-b con 0x7f16d4139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient(hunting): picked mon.noname-a con 0x7f16d413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d40a8b80 con 0x7f16d413d620 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d40a0240 con 0x7f16d4139b10 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d4120d80 con 0x7f16d4136b60 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d99c4640 10 monclient(hunting): authenticate will time out at 976.702848s 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f16d413d620 legacy=0x7f16d41d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d27fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f16d4139b10 0x7f16d41d09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56916/0 (socket says 172.21.15.171:56916) 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.197+0000 7f16d27fc640 1 -- 172.21.15.171:0/159642260 learned_addr learned my addr 172.21.15.171:0/159642260 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 264918936 0 0) 0x7f16d40a0240 con 0x7f16d4139b10 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): my global_id is 5533 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16c0003550 con 0x7f16d4139b10 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990460714 0 0) 0x7f16d4120d80 con 0x7f16d4136b60 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): my global_id is 5564 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16d40a0240 con 0x7f16d4136b60 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2706441447 0 0) 0x7f16c0003550 con 0x7f16d4139b10 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4120d80 con 0x7f16d4139b10 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16c8001430 con 0x7f16d4139b10 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): dump: 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.215 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 298928381 0 0) 0x7f16d40a0240 con 0x7f16d4136b60 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16c0003550 con 0x7f16d4136b60 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16c4001830 con 0x7f16d4136b60 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient(hunting): dump: 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.216 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3598017410 0 0) 0x7f16d4120d80 con 0x7f16d4139b10 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 >> v1:172.21.15.148:6790/0 conn(0x7f16d4136b60 legacy=0x7f16d41cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 >> v1:172.21.15.148:6789/0 conn(0x7f16d413d620 legacy=0x7f16d41d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 monclient: found mon.b 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d4120aa0 con 0x7f16d4139b10 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 5 monclient: authenticate success, global_id 5533 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 10 monclient: _renew_subs 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f16d4136640 con 0x7f16d4139b10 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 10 monclient: _renew_subs 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16d409e3f0 con 0x7f16d4139b10 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f16c80016e0 con 0x7f16d4139b10 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16c8004e60 con 0x7f16d4139b10 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16d17fa640 4 set_mon_vals no callback set 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.198+0000 7f16bbfff640 10 monclient: dump: 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.219 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f16c8010290 con 0x7f16d4139b10 2024-09-16T04:10:01.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(125..125 src has 1..125) ==== 7393+0+0 (unknown 1445267536 0 0) 0x7f16c8047cd0 con 0x7f16d4139b10 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16d2ffd640 10 monclient: get_auth_request con 0x7f16c00390c0 auth_method 0 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16d99c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16d99c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16d99c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.199+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b4005080 con 0x7f16d4139b10 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.204+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f16c8015cf0 con 0x7f16d4139b10 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.204+0000 7f16bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.204+0000 7f16bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:01.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16d99c4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16d99c4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16d99c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f16b4005370 con 0x7f16d4139b10 2024-09-16T04:10:01.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v125) ==== 71+0+14 (unknown 3330414156 0 3003798303) 0x7f16c801e1a0 con 0x7f16d4139b10 2024-09-16T04:10:01.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.381+0000 7f16bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:01.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16d99c4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16d99c4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16d99c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f16b4000be0 con 0x7f16d4139b10 2024-09-16T04:10:01.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16bbfff640 1 -- 172.21.15.171:0/159642260 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v125) ==== 71+0+14 (unknown 3330414156 0 3003798303) 0x7f16c80139c0 con 0x7f16d4139b10 2024-09-16T04:10:01.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16bbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.488+0000 7f16bbfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:01.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 >> v1:172.21.15.171:6812/7327273 conn(0x7f16c00390c0 legacy=0x7f16c003b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 10 monclient: shutdown 2024-09-16T04:10:01.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 >> v1:172.21.15.171:6789/0 conn(0x7f16d4139b10 legacy=0x7f16d41d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 shutdown_connections 2024-09-16T04:10:01.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 >> 172.21.15.171:0/159642260 conn(0x7f16d40a98b0 msgr2=0x7f16d40a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 shutdown_connections 2024-09-16T04:10:01.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.490+0000 7f16d99c4640 1 -- 172.21.15.171:0/159642260 wait complete. 2024-09-16T04:10:01.502 INFO:tasks.workunit.client.0.smithi171.stdout:fooset 2024-09-16T04:10:01.503 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:01.503 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep barset 2024-09-16T04:10:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.578+0000 7f0c06d2c640 10 monclient: get_monmap_and_config 2024-09-16T04:10:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.578+0000 7f0c06d2c640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.578+0000 7f0c06d2c640 10 monclient: monmap: 2024-09-16T04:10:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:01.579414+0000 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:01.579414+0000 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 1 Processor -- start 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 1 -- start start 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient: init 2024-09-16T04:10:01.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient(hunting): picked mon.noname-c con 0x7f0c001327c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient(hunting): picked mon.noname-b con 0x7f0c0009eb20 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient(hunting): picked mon.noname-a con 0x7f0c00139ea0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0009ef30 con 0x7f0c00139ea0 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0009d880 con 0x7f0c001327c0 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0009db60 con 0x7f0c0009eb20 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c06d2c640 10 monclient(hunting): authenticate will time out at 977.084839s 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c052c9640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f0c00139ea0 legacy=0x7f0c0013c290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:01.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c04ac8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0c001327c0 0x7f0c00132ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56924/0 (socket says 172.21.15.171:56924) 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.579+0000 7f0c04ac8640 1 -- 172.21.15.171:0/533451938 learned_addr learned my addr 172.21.15.171:0/533451938 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84572205 0 0) 0x7f0c0009d880 con 0x7f0c001327c0 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient(hunting): my global_id is 5536 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bf0003520 con 0x7f0c001327c0 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3918395760 0 0) 0x7f0bf0003520 con 0x7f0c001327c0 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 >> v1:172.21.15.148:6790/0 conn(0x7f0c0009eb20 legacy=0x7f0c00138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 >> v1:172.21.15.148:6789/0 conn(0x7f0c00139ea0 legacy=0x7f0c0013c290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 monclient: found mon.noname-c 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0009de40 con 0x7f0c001327c0 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0bf4001610 con 0x7f0c001327c0 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 5 monclient: authenticate success, global_id 5536 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:01.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: dump: 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bf4001900 con 0x7f0c001327c0 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 1 -- 172.21.15.171:0/533451938 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0bf4001d10 con 0x7f0c001327c0 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 4 set_mon_vals no callback set 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0bff7fe640 10 monclient: dump: 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.595 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.596 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.596 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.596 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 10 monclient: shutdown 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 1 -- 172.21.15.171:0/533451938 >> v1:172.21.15.171:6789/0 conn(0x7f0c001327c0 legacy=0x7f0c00132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 1 -- 172.21.15.171:0/533451938 shutdown_connections 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.580+0000 7f0c06d2c640 1 -- 172.21.15.171:0/533451938 >> 172.21.15.171:0/533451938 conn(0x7f0c000a5370 msgr2=0x7f0c000a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 1 -- 172.21.15.171:0/533451938 shutdown_connections 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 1 -- 172.21.15.171:0/533451938 wait complete. 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 10 monclient: monmap: 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:01.597 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 1 Processor -- start 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 1 -- start start 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.581+0000 7f0c06d2c640 10 monclient: init 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient(hunting): picked mon.noname-a con 0x7f0c0009eb20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient(hunting): picked mon.noname-c con 0x7f0c001327c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient(hunting): picked mon.noname-b con 0x7f0c00139ea0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c00139130 con 0x7f0c0009eb20 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0011b9a0 con 0x7f0c00139ea0 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0006ce80 con 0x7f0c001327c0 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c06d2c640 10 monclient(hunting): authenticate will time out at 977.087502s 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c052c9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0c00139ea0 0x7f0c001cfc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56940/0 (socket says 172.21.15.171:56940) 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c052c9640 1 -- 172.21.15.171:0/330658916 learned_addr learned my addr 172.21.15.171:0/330658916 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0c04ac8640 1 -- 172.21.15.171:0/330658916 >> v1:172.21.15.148:6789/0 conn(0x7f0c0009eb20 legacy=0x7f0c00130df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:01.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78578221 0 0) 0x7f0c0011b9a0 con 0x7f0c00139ea0 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 5539 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be4003520 con 0x7f0c00139ea0 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221868888 0 0) 0x7f0c0006ce80 con 0x7f0c001327c0 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 10 monclient(hunting): my global_id is 5570 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c0011b9a0 con 0x7f0c001327c0 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.582+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4195908568 0 0) 0x7f0be4003520 con 0x7f0c00139ea0 2024-09-16T04:10:01.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0c0006ce80 con 0x7f0c00139ea0 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0bf8004440 con 0x7f0c00139ea0 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient(hunting): dump: 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1644803117 0 0) 0x7f0c0006ce80 con 0x7f0c00139ea0 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 >> v1:172.21.15.148:6790/0 conn(0x7f0c001327c0 legacy=0x7f0c001cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 >> v1:172.21.15.148:6789/0 conn(0x7f0c0009eb20 legacy=0x7f0c00130df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 monclient: found mon.b 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c00139930 con 0x7f0c00139ea0 2024-09-16T04:10:01.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 5 monclient: authenticate success, global_id 5539 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 10 monclient: _renew_subs 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0011b6c0 con 0x7f0c00139ea0 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 10 monclient: _renew_subs 2024-09-16T04:10:01.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0c00137240 con 0x7f0c00139ea0 2024-09-16T04:10:01.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0bf80046a0 con 0x7f0c00139ea0 2024-09-16T04:10:01.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0bf80049f0 con 0x7f0c00139ea0 2024-09-16T04:10:01.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfeffd640 4 set_mon_vals no callback set 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.583+0000 7f0bfd7fa640 10 monclient: dump: 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.605 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0bf8004b90 con 0x7f0c00139ea0 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0c06d2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0c06d2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0c06d2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be0005080 con 0x7f0c00139ea0 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(125..125 src has 1..125) ==== 7393+0+0 (unknown 1445267536 0 0) 0x7f0bf8047e40 con 0x7f0c00139ea0 2024-09-16T04:10:01.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.584+0000 7f0c04ac8640 10 monclient: get_auth_request con 0x7f0be4038c60 auth_method 0 2024-09-16T04:10:01.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.588+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0bf8015e00 con 0x7f0c00139ea0 2024-09-16T04:10:01.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.588+0000 7f0bfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.588+0000 7f0bfd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:01.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0c06d2c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0c06d2c640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0c06d2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f0be0005370 con 0x7f0c00139ea0 2024-09-16T04:10:01.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v125) ==== 71+0+14 (unknown 3330414156 0 3003798303) 0x7f0bf8019ab0 con 0x7f0c00139ea0 2024-09-16T04:10:01.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0bfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.764+0000 7f0bfd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:01.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0c06d2c640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0c06d2c640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0c06d2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f0be0000bb0 con 0x7f0c00139ea0 2024-09-16T04:10:01.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0bfd7fa640 1 -- 172.21.15.171:0/330658916 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v125) ==== 71+0+14 (unknown 3330414156 0 3003798303) 0x7f0bf801e370 con 0x7f0c00139ea0 2024-09-16T04:10:01.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0bfd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:01.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.871+0000 7f0bfd7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:01.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 >> v1:172.21.15.171:6812/7327273 conn(0x7f0be4038c60 legacy=0x7f0be403b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 10 monclient: shutdown 2024-09-16T04:10:01.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 >> v1:172.21.15.171:6789/0 conn(0x7f0c00139ea0 legacy=0x7f0c001cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 shutdown_connections 2024-09-16T04:10:01.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 >> 172.21.15.171:0/330658916 conn(0x7f0c000a5370 msgr2=0x7f0c000a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 shutdown_connections 2024-09-16T04:10:01.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.873+0000 7f0c06d2c640 1 -- 172.21.15.171:0/330658916 wait complete. 2024-09-16T04:10:01.884 INFO:tasks.workunit.client.0.smithi171.stdout:barset 2024-09-16T04:10:01.884 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set dump 2024-09-16T04:10:01.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.960+0000 7f01cb59f640 10 monclient: get_monmap_and_config 2024-09-16T04:10:01.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.960+0000 7f01cb59f640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.960+0000 7f01cb59f640 10 monclient: monmap: 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:01.961485+0000 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:01.961485+0000 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:01.963 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 1 Processor -- start 2024-09-16T04:10:01.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 1 -- start start 2024-09-16T04:10:01.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient: init 2024-09-16T04:10:01.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient(hunting): picked mon.noname-b con 0x7f01c40a4040 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient(hunting): picked mon.noname-a con 0x7f01c409fc20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient(hunting): picked mon.noname-c con 0x7f01c40a0050 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c40737d0 con 0x7f01c409fc20 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c406c630 con 0x7f01c40a0050 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c406d8e0 con 0x7f01c40a4040 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.961+0000 7f01cb59f640 10 monclient(hunting): authenticate will time out at 977.466993s 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01c8b3a640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f01c409fc20 legacy=0x7f01c413b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01c9b3c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f01c40a0050 0x7f01c413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56942/0 (socket says 172.21.15.171:56942) 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01c9b3c640 1 -- 172.21.15.171:0/3526558951 learned_addr learned my addr 172.21.15.171:0/3526558951 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:01.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426030392 0 0) 0x7f01c406c630 con 0x7f01c40a0050 2024-09-16T04:10:01.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient(hunting): my global_id is 5542 2024-09-16T04:10:01.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b0003520 con 0x7f01c40a0050 2024-09-16T04:10:01.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3217534477 0 0) 0x7f01b0003520 con 0x7f01c40a0050 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 >> v1:172.21.15.148:6790/0 conn(0x7f01c40a4040 legacy=0x7f01c40a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 >> v1:172.21.15.148:6789/0 conn(0x7f01c409fc20 legacy=0x7f01c413b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 monclient: found mon.noname-c 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c4143840 con 0x7f01c40a0050 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01bc001990 con 0x7f01c40a0050 2024-09-16T04:10:01.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: dump: 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01cb59f640 5 monclient: authenticate success, global_id 5542 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01cb59f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01bc001c80 con 0x7f01c40a0050 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 1 -- 172.21.15.171:0/3526558951 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01bc002090 con 0x7f01c40a0050 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01cb59f640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01cb59f640 4 set_mon_vals no callback set 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.962+0000 7f01bbfff640 10 monclient: dump: 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.976 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.977 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 10 monclient: shutdown 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 1 -- 172.21.15.171:0/3526558951 >> v1:172.21.15.171:6789/0 conn(0x7f01c40a0050 legacy=0x7f01c413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 1 -- 172.21.15.171:0/3526558951 shutdown_connections 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 1 -- 172.21.15.171:0/3526558951 >> 172.21.15.171:0/3526558951 conn(0x7f01c40a7520 msgr2=0x7f01c40a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 1 -- 172.21.15.171:0/3526558951 shutdown_connections 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 1 -- 172.21.15.171:0/3526558951 wait complete. 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.963+0000 7f01cb59f640 10 monclient: monmap: 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:01.978 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 1 Processor -- start 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 1 -- start start 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient: init 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient(hunting): picked mon.noname-c con 0x7f01c409fc20 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient(hunting): picked mon.noname-b con 0x7f01c40a0050 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:01.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient(hunting): picked mon.noname-a con 0x7f01c40a4040 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4072c30 con 0x7f01c40a4040 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c4070f70 con 0x7f01c40a0050 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01c40a1dc0 con 0x7f01c409fc20 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01cb59f640 10 monclient(hunting): authenticate will time out at 977.469602s 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01c8b3a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f01c40a0050 0x7f01c4070860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56948/0 (socket says 172.21.15.171:56948) 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01c8b3a640 1 -- 172.21.15.171:0/2407106140 learned_addr learned my addr 172.21.15.171:0/2407106140 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01c9b3c640 1 -- 172.21.15.171:0/2407106140 >> v1:172.21.15.148:6789/0 conn(0x7f01c40a4040 legacy=0x7f01c41d39b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676823063 0 0) 0x7f01c4070f70 con 0x7f01c40a0050 2024-09-16T04:10:01.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01b9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01b9ffb640 10 monclient(hunting): my global_id is 5545 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.964+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a8003520 con 0x7f01c40a0050 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3669570131 0 0) 0x7f01a8003520 con 0x7f01c40a0050 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01c4070f70 con 0x7f01c40a0050 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01b40040b0 con 0x7f01c40a0050 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:01.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.982 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906171846 0 0) 0x7f01c40a1dc0 con 0x7f01c409fc20 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient(hunting): my global_id is 5576 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01a8003520 con 0x7f01c409fc20 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2870545488 0 0) 0x7f01c4070f70 con 0x7f01c40a0050 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 >> v1:172.21.15.148:6790/0 conn(0x7f01c409fc20 legacy=0x7f01c4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 >> v1:172.21.15.148:6789/0 conn(0x7f01c40a4040 legacy=0x7f01c41d39b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 monclient: found mon.b 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c40a1870 con 0x7f01c40a0050 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 5 monclient: authenticate success, global_id 5545 2024-09-16T04:10:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 10 monclient: _renew_subs 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01c409d7c0 con 0x7f01c40a0050 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f01b4004410 con 0x7f01c40a0050 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01b40047e0 con 0x7f01c40a0050 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01b9ffb640 10 monclient: dump: 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 10 monclient: _renew_subs 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f01c4072950 con 0x7f01c40a0050 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.965+0000 7f01bb7fe640 4 set_mon_vals no callback set 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f01b4004a00 con 0x7f01c40a0050 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(125..125 src has 1..125) ==== 7393+0+0 (unknown 1445267536 0 0) 0x7f01b4047c90 con 0x7f01c40a0050 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01c933b640 10 monclient: get_auth_request con 0x7f01a8038fe0 auth_method 0 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01cb59f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01cb59f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01cb59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.966+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c4073280 con 0x7f01c40a0050 2024-09-16T04:10:01.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.970+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f01b4015d10 con 0x7f01c40a0050 2024-09-16T04:10:01.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.970+0000 7f01b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:01.970+0000 7f01b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:02.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.143+0000 7f01cb59f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:10:02.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.143+0000 7f01cb59f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:10:02.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.143+0000 7f01cb59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.143+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7f01c40717a0 con 0x7f01c40a0050 2024-09-16T04:10:02.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.145+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v125) ==== 73+0+6748 (unknown 438279449 0 3407865348) 0x7f01b4015610 con 0x7f01c40a0050 2024-09-16T04:10:02.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.145+0000 7f01b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:10:02.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.145+0000 7f01b9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:02.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.250+0000 7f01cb59f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:10:02.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.251+0000 7f01cb59f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:10:02.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.251+0000 7f01cb59f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.251+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set dump"} v 0) -- 0x7f01c40008d0 con 0x7f01c40a0050 2024-09-16T04:10:02.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.251+0000 7f01b9ffb640 1 -- 172.21.15.171:0/2407106140 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v125) ==== 73+0+6748 (unknown 438279449 0 3407865348) 0x7f01b401e1c0 con 0x7f01c40a0050 2024-09-16T04:10:02.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.251+0000 7f01b9ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-16T04:10:02.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.251+0000 7f01b9ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:02.254 INFO:tasks.workunit.client.0.smithi171.stdout:{ 2024-09-16T04:10:02.254 INFO:tasks.workunit.client.0.smithi171.stdout: "9": [ 2024-09-16T04:10:02.256 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.256 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -1, 2024-09-16T04:10:02.256 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.256 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.256 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.257 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.257 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.257 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.257 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.257 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.257 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.258 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.258 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -2, 2024-09-16T04:10:02.258 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.258 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.258 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.259 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.259 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.259 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.259 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.259 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -3, 2024-09-16T04:10:02.259 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.260 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.260 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.260 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.260 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.260 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.260 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.261 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.261 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.261 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -4, 2024-09-16T04:10:02.261 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.261 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.262 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.262 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.262 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.262 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -5, 2024-09-16T04:10:02.262 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.262 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.263 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.263 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.263 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.263 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.263 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.263 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.264 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.264 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -6, 2024-09-16T04:10:02.264 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.264 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.264 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.264 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.265 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.265 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.265 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.265 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.265 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.266 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.266 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -7, 2024-09-16T04:10:02.266 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.266 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.266 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.266 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.267 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.267 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.267 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.267 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -8, 2024-09-16T04:10:02.267 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.267 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.268 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.268 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.268 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.268 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -9, 2024-09-16T04:10:02.268 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.268 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.269 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.269 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.269 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.269 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.269 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.269 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.270 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.270 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.270 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -10, 2024-09-16T04:10:02.270 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.270 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.271 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.271 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.271 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.271 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -11, 2024-09-16T04:10:02.271 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.271 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.272 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.272 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.272 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.272 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -12, 2024-09-16T04:10:02.272 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.272 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.273 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.273 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.273 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.273 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -13, 2024-09-16T04:10:02.273 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.273 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -14, 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.274 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -15, 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: "10": [ 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -1, 2024-09-16T04:10:02.275 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.276 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: 0.263671875, 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.277 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -2, 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.278 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.279 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -3, 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.280 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.281 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -4, 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.282 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -5, 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.283 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.284 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -6, 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.285 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.286 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.286 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.286 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.286 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.286 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.286 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.287 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.287 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.287 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.287 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.287 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.287 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.288 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.288 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.288 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.288 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.288 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.289 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.289 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.289 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.289 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -7, 2024-09-16T04:10:02.289 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.289 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.290 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.290 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.290 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.290 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.290 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.290 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.291 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625 2024-09-16T04:10:02.291 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.291 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.291 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.291 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.292 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -8, 2024-09-16T04:10:02.292 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.292 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.292 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.292 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.292 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.293 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.293 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.293 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -9, 2024-09-16T04:10:02.293 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.293 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.293 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.294 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.294 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.294 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.294 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.294 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.295 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.295 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.295 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.295 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.295 INFO:tasks.workunit.client.0.smithi171.stdout: ], 2024-09-16T04:10:02.295 INFO:tasks.workunit.client.0.smithi171.stdout: [ 2024-09-16T04:10:02.296 INFO:tasks.workunit.client.0.smithi171.stdout: 0.087890625, 2024-09-16T04:10:02.296 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.296 INFO:tasks.workunit.client.0.smithi171.stdout: 0, 2024-09-16T04:10:02.296 INFO:tasks.workunit.client.0.smithi171.stdout: 0 2024-09-16T04:10:02.296 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.296 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.297 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.297 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.297 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -10, 2024-09-16T04:10:02.297 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.297 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.297 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.298 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.298 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.298 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.298 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.298 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -11, 2024-09-16T04:10:02.298 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.299 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.299 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.299 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.299 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.299 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.299 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.300 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -12, 2024-09-16T04:10:02.300 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.300 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.300 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.300 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.301 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.301 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.301 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.301 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -13, 2024-09-16T04:10:02.302 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.302 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.302 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.302 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.302 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.302 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.303 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.303 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -14, 2024-09-16T04:10:02.303 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.303 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.304 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.304 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.304 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.304 INFO:tasks.workunit.client.0.smithi171.stdout: }, 2024-09-16T04:10:02.304 INFO:tasks.workunit.client.0.smithi171.stdout: { 2024-09-16T04:10:02.304 INFO:tasks.workunit.client.0.smithi171.stdout: "bucket_id": -15, 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: "weight_set": [ 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: [], 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: [] 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: } 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout: ] 2024-09-16T04:10:02.305 INFO:tasks.workunit.client.0.smithi171.stdout:} 2024-09-16T04:10:02.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 >> v1:172.21.15.171:6812/7327273 conn(0x7f01a8038fe0 legacy=0x7f01a803b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 10 monclient: shutdown 2024-09-16T04:10:02.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 >> v1:172.21.15.171:6789/0 conn(0x7f01c40a0050 legacy=0x7f01c4070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 shutdown_connections 2024-09-16T04:10:02.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 >> 172.21.15.171:0/2407106140 conn(0x7f01c40a7520 msgr2=0x7f01c4136c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 shutdown_connections 2024-09-16T04:10:02.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.253+0000 7f01cb59f640 1 -- 172.21.15.171:0/2407106140 wait complete. 2024-09-16T04:10:02.307 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-16T04:10:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.343+0000 7f0541aa5640 10 monclient: get_monmap_and_config 2024-09-16T04:10:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.343+0000 7f0541aa5640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.343+0000 7f0541aa5640 10 monclient: monmap: 2024-09-16T04:10:02.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:02.344444+0000 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:02.344444+0000 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:02.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 1 Processor -- start 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 1 -- start start 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient: init 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f053c09ff90 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f053c0a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f053c0a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c0a0f20 con 0x7f053c09ff90 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c0a1200 con 0x7f053c0a0800 2024-09-16T04:10:02.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c09bca0 con 0x7f053c0a0370 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.344+0000 7f0541aa5640 10 monclient(hunting): authenticate will time out at 977.849968s 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053b7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f053c09ff90 legacy=0x7f053c09f7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f053c0a0800 0x7f053c13d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56958/0 (socket says 172.21.15.171:56958) 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053bfff640 1 -- 172.21.15.171:0/2891738684 learned_addr learned my addr 172.21.15.171:0/2891738684 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3741575851 0 0) 0x7f053c0a1200 con 0x7f053c0a0800 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): my global_id is 5548 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0524003520 con 0x7f053c0a0800 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4166199241 0 0) 0x7f053c09bca0 con 0x7f053c0a0370 2024-09-16T04:10:02.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient(hunting): my global_id is 5579 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f053c0a1200 con 0x7f053c0a0370 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 666718184 0 0) 0x7f0524003520 con 0x7f053c0a0800 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 >> v1:172.21.15.148:6790/0 conn(0x7f053c0a0370 legacy=0x7f053c139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 >> v1:172.21.15.148:6789/0 conn(0x7f053c09ff90 legacy=0x7f053c09f7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c09e830 con 0x7f053c0a0800 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0534001610 con 0x7f053c0a0800 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f0541aa5640 5 monclient: authenticate success, global_id 5548 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: dump: 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:02.356 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f0541aa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0534001930 con 0x7f053c0a0800 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 1 -- 172.21.15.171:0/2891738684 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0534001d00 con 0x7f053c0a0800 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f0541aa5640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f0541aa5640 4 set_mon_vals no callback set 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.345+0000 7f053a7fc640 10 monclient: dump: 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:02.357 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 10 monclient: shutdown 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 1 -- 172.21.15.171:0/2891738684 >> v1:172.21.15.171:6789/0 conn(0x7f053c0a0800 legacy=0x7f053c13d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 1 -- 172.21.15.171:0/2891738684 shutdown_connections 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 1 -- 172.21.15.171:0/2891738684 >> 172.21.15.171:0/2891738684 conn(0x7f053c0a5450 msgr2=0x7f053c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 1 -- 172.21.15.171:0/2891738684 shutdown_connections 2024-09-16T04:10:02.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 1 -- 172.21.15.171:0/2891738684 wait complete. 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.346+0000 7f0541aa5640 10 monclient: monmap: 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 1 Processor -- start 2024-09-16T04:10:02.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 1 -- start start 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient: init 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f053c09ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f053c0a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f053c0a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c11fb50 con 0x7f053c0a0370 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c071670 con 0x7f053c09ff90 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c1cc730 con 0x7f053c0a0800 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f0541aa5640 10 monclient(hunting): authenticate will time out at 977.852847s 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f053b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f053c09ff90 0x7f053c1cb910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56970/0 (socket says 172.21.15.171:56970) 2024-09-16T04:10:02.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f053b7fe640 1 -- 172.21.15.171:0/4209921600 learned_addr learned my addr 172.21.15.171:0/4209921600 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.347+0000 7f053affd640 1 -- 172.21.15.171:0/4209921600 >> v1:172.21.15.148:6789/0 conn(0x7f053c0a0370 legacy=0x7f053c1cc020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538024777 0 0) 0x7f052c003e00 con 0x7f053c09ff90 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): my global_id is 5551 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0528003520 con 0x7f053c09ff90 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 531258051 0 0) 0x7f053c1cc730 con 0x7f053c0a0800 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): my global_id is 5582 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f052c003e00 con 0x7f053c0a0800 2024-09-16T04:10:02.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 180990658 0 0) 0x7f0528003520 con 0x7f053c09ff90 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f053c1cc730 con 0x7f053c09ff90 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f052c0046d0 con 0x7f053c09ff90 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): dump: 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:02.362 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1426745295 0 0) 0x7f053c1cc730 con 0x7f053c09ff90 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 >> v1:172.21.15.148:6790/0 conn(0x7f053c0a0800 legacy=0x7f053c1d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 >> v1:172.21.15.148:6789/0 conn(0x7f053c0a0370 legacy=0x7f053c1cc020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 monclient: found mon.b 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c06ea80 con 0x7f053c09ff90 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 5 monclient: authenticate success, global_id 5551 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 10 monclient: _renew_subs 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f053c09d0c0 con 0x7f053c09ff90 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 10 monclient: _renew_subs 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f053c120660 con 0x7f053c09ff90 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f052c0049f0 con 0x7f053c09ff90 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0539ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f052c004dc0 con 0x7f053c09ff90 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:02.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.348+0000 7f0523fff640 10 monclient: dump: 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f052c005020 con 0x7f053c09ff90 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f0541aa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f0541aa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f0541aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f051c005080 con 0x7f053c09ff90 2024-09-16T04:10:02.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(125..125 src has 1..125) ==== 7393+0+0 (unknown 1445267536 0 0) 0x7f052c048210 con 0x7f053c09ff90 2024-09-16T04:10:02.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.349+0000 7f053affd640 10 monclient: get_auth_request con 0x7f0528038b90 auth_method 0 2024-09-16T04:10:02.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.354+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f053c071670 con 0x7f053c09ff90 2024-09-16T04:10:02.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.354+0000 7f0523fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.354+0000 7f0523fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.524+0000 7f0541aa5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.524+0000 7f0541aa5640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.524+0000 7f0541aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:02.524+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f051c005370 con 0x7f053c09ff90 2024-09-16T04:10:03.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.150+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v126) ==== 129+0+0 (unknown 4042788291 0 0) 0x7f052c016230 con 0x7f053c09ff90 2024-09-16T04:10:03.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.150+0000 7f0523fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:03.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.150+0000 7f0523fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:03.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.261+0000 7f0541aa5640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:03.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.261+0000 7f0541aa5640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:03.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.261+0000 7f0541aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:03.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:03.261+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7f051c0008d0 con 0x7f053c09ff90 2024-09-16T04:10:04.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.154+0000 7f0523fff640 1 -- 172.21.15.171:0/4209921600 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v127) ==== 129+0+0 (unknown 3017890396 0 0) 0x7f052c015b30 con 0x7f053c09ff90 2024-09-16T04:10:04.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.154+0000 7f0523fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:04.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.154+0000 7f0523fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:04.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.157+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 >> v1:172.21.15.171:6812/7327273 conn(0x7f0528038b90 legacy=0x7f052803b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.157+0000 7f0541aa5640 10 monclient: shutdown 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.157+0000 7f0541aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.157+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 >> v1:172.21.15.171:6789/0 conn(0x7f053c09ff90 legacy=0x7f053c1cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.158+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 shutdown_connections 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.158+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 >> 172.21.15.171:0/4209921600 conn(0x7f053c0a5450 msgr2=0x7f053c0a2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.158+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 shutdown_connections 2024-09-16T04:10:04.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.158+0000 7f0541aa5640 1 -- 172.21.15.171:0/4209921600 wait complete. 2024-09-16T04:10:04.172 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T04:10:04.173 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:10:04.173 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.258+0000 7fdce498d640 10 monclient: get_monmap_and_config 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.258+0000 7fdce498d640 10 monclient: build_initial_monmap 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.258+0000 7fdce498d640 10 monclient: monmap: 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:04.258886+0000 2024-09-16T04:10:04.260 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:04.258886+0000 2024-09-16T04:10:04.261 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:04.261 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.261 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:04.261 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:04.261 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:04.261 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.258+0000 7fdce498d640 1 Processor -- start 2024-09-16T04:10:04.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.258+0000 7fdce498d640 1 -- start start 2024-09-16T04:10:04.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.258+0000 7fdce498d640 10 monclient: init 2024-09-16T04:10:04.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:04.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:04.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient(hunting): picked mon.noname-b con 0x7fdce0136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient(hunting): picked mon.noname-a con 0x7fdce0139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient(hunting): picked mon.noname-c con 0x7fdce013d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce00a1340 con 0x7fdce0139b10 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce0137650 con 0x7fdce013d620 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce00a0aa0 con 0x7fdce0136b60 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:04.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient(hunting): authenticate will time out at 979.764219s 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcded9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdce013d620 0x7fdce013fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56972/0 (socket says 172.21.15.171:56972) 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcded9d640 1 -- 172.21.15.171:0/2285516134 learned_addr learned my addr 172.21.15.171:0/2285516134 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcd5d9b640 1 -- 172.21.15.171:0/2285516134 >> v1:172.21.15.148:6789/0 conn(0x7fdce0139b10 legacy=0x7fdce013bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831821361 0 0) 0x7fdce0137650 con 0x7fdce013d620 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient(hunting): my global_id is 5554 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdccc003520 con 0x7fdce013d620 2024-09-16T04:10:04.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129554439 0 0) 0x7fdccc003520 con 0x7fdce013d620 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 >> v1:172.21.15.148:6790/0 conn(0x7fdce0136b60 legacy=0x7fdce0136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 >> v1:172.21.15.148:6789/0 conn(0x7fdce0139b10 legacy=0x7fdce013bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce00a0550 con 0x7fdce013d620 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: _finish_auth 0 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:04.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcd8001990 con 0x7fdce013d620 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: dump: 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.275 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 5 monclient: authenticate success, global_id 5554 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdcd8001cf0 con 0x7fdce013d620 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdce498d640 4 set_mon_vals no callback set 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 1 -- 172.21.15.171:0/2285516134 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcd8004c20 con 0x7fdce013d620 2024-09-16T04:10:04.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.259+0000 7fdcddd9b640 10 monclient: dump: 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.277 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 10 monclient: shutdown 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 1 -- 172.21.15.171:0/2285516134 >> v1:172.21.15.171:6789/0 conn(0x7fdce013d620 legacy=0x7fdce013fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 1 -- 172.21.15.171:0/2285516134 shutdown_connections 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 1 -- 172.21.15.171:0/2285516134 >> 172.21.15.171:0/2285516134 conn(0x7fdce00a98b0 msgr2=0x7fdce00a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 1 -- 172.21.15.171:0/2285516134 shutdown_connections 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.260+0000 7fdce498d640 1 -- 172.21.15.171:0/2285516134 wait complete. 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.261+0000 7fdce498d640 10 monclient: build_initial_monmap 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.261+0000 7fdce498d640 10 monclient: monmap: 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:04.278 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.261+0000 7fdce498d640 1 Processor -- start 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.261+0000 7fdce498d640 1 -- start start 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.261+0000 7fdce498d640 10 monclient: init 2024-09-16T04:10:04.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient(hunting): picked mon.noname-c con 0x7fdce0136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient(hunting): picked mon.noname-a con 0x7fdce0139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient(hunting): picked mon.noname-b con 0x7fdce013d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce009eeb0 con 0x7fdce0139b10 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce0120d80 con 0x7fdce013d620 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdce00a8c10 con 0x7fdce0136b60 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient(hunting): authenticate will time out at 979.767269s 2024-09-16T04:10:04.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd5d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdce0139b10 legacy=0x7fdce009dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcded9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdce013d620 0x7fdce01d37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56980/0 (socket says 172.21.15.171:56980) 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcded9d640 1 -- 172.21.15.171:0/2651159249 learned_addr learned my addr 172.21.15.171:0/2651159249 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1972359034 0 0) 0x7fdce0120d80 con 0x7fdce013d620 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): my global_id is 5557 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcc0003550 con 0x7fdce013d620 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158198116 0 0) 0x7fdce00a8c10 con 0x7fdce0136b60 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): my global_id is 5588 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdce0120d80 con 0x7fdce0136b60 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 791899640 0 0) 0x7fdcc0003550 con 0x7fdce013d620 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdce00a8c10 con 0x7fdce013d620 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcd80017b0 con 0x7fdce013d620 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): dump: 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.282 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3664720079 0 0) 0x7fdce0120d80 con 0x7fdce0136b60 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcc0003550 con 0x7fdce0136b60 2024-09-16T04:10:04.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1190671791 0 0) 0x7fdce00a8c10 con 0x7fdce013d620 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 >> v1:172.21.15.148:6790/0 conn(0x7fdce0136b60 legacy=0x7fdce009d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 >> v1:172.21.15.148:6789/0 conn(0x7fdce0139b10 legacy=0x7fdce009dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 monclient: found mon.b 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce00a0240 con 0x7fdce013d620 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdcd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:04.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 5 monclient: authenticate success, global_id 5557 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient: _renew_subs 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.262+0000 7fdce498d640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdce0120aa0 con 0x7fdce013d620 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdce498d640 10 monclient: _renew_subs 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdce498d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdce498d640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdce009e340 con 0x7fdce013d620 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdcd80019e0 con 0x7fdce013d620 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcd8005380 con 0x7fdce013d620 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:04.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 10 monclient: dump: 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcdd59a640 4 set_mon_vals no callback set 2024-09-16T04:10:04.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.263+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdcd80107b0 con 0x7fdce013d620 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.264+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(127..127 src has 1..127) ==== 7393+0+0 (unknown 1062140266 0 0) 0x7fdcd8048100 con 0x7fdce013d620 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.264+0000 7fdcde59c640 10 monclient: get_auth_request con 0x7fdcc0038eb0 auth_method 0 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.264+0000 7fdce498d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.264+0000 7fdce498d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.264+0000 7fdce498d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.264+0000 7fdce498d640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc4005080 con 0x7fdce013d620 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.268+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdcd8016180 con 0x7fdce013d620 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.268+0000 7fdcd77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.268+0000 7fdcd77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:04.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdce498d640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T04:10:04.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdce498d640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T04:10:04.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdce498d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdce498d640 1 -- 172.21.15.171:0/2651159249 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) -- 0x7fdcc4005370 con 0x7fdce013d620 2024-09-16T04:10:04.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdcd77fe640 1 -- 172.21.15.171:0/2651159249 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v127) ==== 168+0+0 (unknown 2461956091 0 0) 0x7fdcd8015a80 con 0x7fdce013d620 2024-09-16T04:10:04.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdcd77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-16T04:10:04.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.443+0000 7fdcd77fe640 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-16T04:10:04.446 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-16T04:10:04.449 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:10:04.449 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T04:10:04.449 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -x 2024-09-16T04:10:04.450 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-16T04:10:04.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.528+0000 7fcf61b49640 10 monclient: get_monmap_and_config 2024-09-16T04:10:04.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.528+0000 7fcf61b49640 10 monclient: build_initial_monmap 2024-09-16T04:10:04.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.528+0000 7fcf61b49640 10 monclient: monmap: 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:04.529537+0000 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:04.529537+0000 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:04.531 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.529+0000 7fcf61b49640 1 Processor -- start 2024-09-16T04:10:04.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.529+0000 7fcf61b49640 1 -- start start 2024-09-16T04:10:04.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.529+0000 7fcf61b49640 10 monclient: init 2024-09-16T04:10:04.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.529+0000 7fcf61b49640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.529+0000 7fcf61b49640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.529+0000 7fcf61b49640 10 monclient(hunting): picked mon.noname-a con 0x7fcf5c133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 10 monclient(hunting): picked mon.noname-b con 0x7fcf5c1341f0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5c13a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c09da90 con 0x7fcf5c133700 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c09dd70 con 0x7fcf5c13a350 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c09e050 con 0x7fcf5c1341f0 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 10 monclient(hunting): authenticate will time out at 980.035080s 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5b7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fcf5c133700 legacy=0x7fcf5c133ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcf5c13a350 0x7fcf5c13c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56982/0 (socket says 172.21.15.171:56982) 2024-09-16T04:10:04.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5bfff640 1 -- 172.21.15.171:0/554386433 learned_addr learned my addr 172.21.15.171:0/554386433 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242107184 0 0) 0x7fcf5c09dd70 con 0x7fcf5c13a350 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): my global_id is 5560 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf44003520 con 0x7fcf5c13a350 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698744651 0 0) 0x7fcf5c09e050 con 0x7fcf5c1341f0 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient(hunting): my global_id is 5591 2024-09-16T04:10:04.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf5c09dd70 con 0x7fcf5c1341f0 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 429733450 0 0) 0x7fcf44003520 con 0x7fcf5c13a350 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 >> v1:172.21.15.148:6790/0 conn(0x7fcf5c1341f0 legacy=0x7fcf5c138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 >> v1:172.21.15.148:6789/0 conn(0x7fcf5c133700 legacy=0x7fcf5c133ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c09d4f0 con 0x7fcf5c13a350 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:04.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf54001610 con 0x7fcf5c13a350 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 5 monclient: authenticate success, global_id 5560 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: dump: 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.542 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf54001930 con 0x7fcf5c13a350 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 1 -- 172.21.15.171:0/554386433 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf54001d00 con 0x7fcf5c13a350 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf5a7fc640 10 monclient: dump: 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.543 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.530+0000 7fcf61b49640 4 set_mon_vals no callback set 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 10 monclient: shutdown 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 1 -- 172.21.15.171:0/554386433 >> v1:172.21.15.171:6789/0 conn(0x7fcf5c13a350 legacy=0x7fcf5c13c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 1 -- 172.21.15.171:0/554386433 shutdown_connections 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 1 -- 172.21.15.171:0/554386433 >> 172.21.15.171:0/554386433 conn(0x7fcf5c0a5450 msgr2=0x7fcf5c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 1 -- 172.21.15.171:0/554386433 shutdown_connections 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 1 -- 172.21.15.171:0/554386433 wait complete. 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 10 monclient: build_initial_monmap 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.531+0000 7fcf61b49640 10 monclient: monmap: 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 1 Processor -- start 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 1 -- start start 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient: init 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5c133700 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient(hunting): picked mon.noname-b con 0x7fcf5c1341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient(hunting): picked mon.noname-a con 0x7fcf5c13a350 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:04.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c1393b0 con 0x7fcf5c13a350 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c11cbd0 con 0x7fcf5c1341f0 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c072d30 con 0x7fcf5c133700 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf61b49640 10 monclient(hunting): authenticate will time out at 980.037781s 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf5affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcf5c1341f0 0x7fcf5c138ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:56988/0 (socket says 172.21.15.171:56988) 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf5affd640 1 -- 172.21.15.171:0/4280880595 learned_addr learned my addr 172.21.15.171:0/4280880595 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.532+0000 7fcf5bfff640 1 -- 172.21.15.171:0/4280880595 >> v1:172.21.15.148:6789/0 conn(0x7fcf5c13a350 legacy=0x7fcf5c1d39e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252928728 0 0) 0x7fcf5c11cbd0 con 0x7fcf5c1341f0 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): my global_id is 5563 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf48003520 con 0x7fcf5c1341f0 2024-09-16T04:10:04.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317909450 0 0) 0x7fcf5c072d30 con 0x7fcf5c133700 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): my global_id is 5594 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf5c11cbd0 con 0x7fcf5c133700 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3906136590 0 0) 0x7fcf48003520 con 0x7fcf5c1341f0 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf5c072d30 con 0x7fcf5c1341f0 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf50001970 con 0x7fcf5c1341f0 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): dump: 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.548 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1931222643 0 0) 0x7fcf5c072d30 con 0x7fcf5c1341f0 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 >> v1:172.21.15.148:6790/0 conn(0x7fcf5c133700 legacy=0x7fcf5c138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 >> v1:172.21.15.148:6789/0 conn(0x7fcf5c13a350 legacy=0x7fcf5c1d39e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:04.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 monclient: found mon.b 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c11c610 con 0x7fcf5c1341f0 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 5 monclient: authenticate success, global_id 5563 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 10 monclient: _renew_subs 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c11c8f0 con 0x7fcf5c1341f0 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 10 monclient: _renew_subs 2024-09-16T04:10:04.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf61b49640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf5c0a03b0 con 0x7fcf5c1341f0 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf50001c50 con 0x7fcf5c1341f0 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf50003a60 con 0x7fcf5c1341f0 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf59ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.533+0000 7fcf43fff640 10 monclient: dump: 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.551 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcf50003c80 con 0x7fcf5c1341f0 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf61b49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf61b49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf61b49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf61b49640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf3c005080 con 0x7fcf5c1341f0 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(127..127 src has 1..127) ==== 7393+0+0 (unknown 1062140266 0 0) 0x7fcf50047d00 con 0x7fcf5c1341f0 2024-09-16T04:10:04.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.534+0000 7fcf5b7fe640 10 monclient: get_auth_request con 0x7fcf48038fb0 auth_method 0 2024-09-16T04:10:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.539+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fcf50015c80 con 0x7fcf5c1341f0 2024-09-16T04:10:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.539+0000 7fcf43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.539+0000 7fcf43fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:04.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.714+0000 7fcf61b49640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:04.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.714+0000 7fcf61b49640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:04.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.714+0000 7fcf61b49640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.715+0000 7fcf61b49640 1 -- 172.21.15.171:0/4280880595 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) -- 0x7fcf3c005370 con 0x7fcf5c1341f0 2024-09-16T04:10:04.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.715+0000 7fcf43fff640 1 -- 172.21.15.171:0/4280880595 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v127) ==== 163+0+0 (unknown 1319589707 0 0) 0x7fcf50019930 con 0x7fcf5c1341f0 2024-09-16T04:10:04.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.715+0000 7fcf43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-16T04:10:04.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.715+0000 7fcf43fff640 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-16T04:10:04.718 INFO:tasks.workunit.client.0.smithi171.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-16T04:10:04.721 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:10:04.721 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-16T04:10:04.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.803+0000 7fbfe9e47640 10 monclient: get_monmap_and_config 2024-09-16T04:10:04.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.803+0000 7fbfe9e47640 10 monclient: build_initial_monmap 2024-09-16T04:10:04.805 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.803+0000 7fbfe9e47640 10 monclient: monmap: 2024-09-16T04:10:04.805 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:04.804236+0000 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:04.804236+0000 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:04.806 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:04.807 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.816 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 1 Processor -- start 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 1 -- start start 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient: init 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe40685d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe409bfc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe4136e10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:04.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe4099c00 con 0x7fbfe4136e10 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe41373c0 con 0x7fbfe409bfc0 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe409a150 con 0x7fbfe40685d0 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe9e47640 10 monclient(hunting): authenticate will time out at 980.309668s 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbfe409bfc0 0x7fbfe413bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:57000/0 (socket says 172.21.15.171:57000) 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe2ffd640 1 -- 172.21.15.171:0/603103059 learned_addr learned my addr 172.21.15.171:0/603103059 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.804+0000 7fbfe3fff640 1 -- 172.21.15.171:0/603103059 >> v1:172.21.15.148:6789/0 conn(0x7fbfe4136e10 legacy=0x7fbfe413f350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:04.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309432171 0 0) 0x7fbfe41373c0 con 0x7fbfe409bfc0 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): my global_id is 5566 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfcc003520 con 0x7fbfe409bfc0 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316369466 0 0) 0x7fbfe409a150 con 0x7fbfe40685d0 2024-09-16T04:10:04.819 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient(hunting): my global_id is 5597 2024-09-16T04:10:04.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe41373c0 con 0x7fbfe40685d0 2024-09-16T04:10:04.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2310902496 0 0) 0x7fbfcc003520 con 0x7fbfe409bfc0 2024-09-16T04:10:04.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 >> v1:172.21.15.148:6790/0 conn(0x7fbfe40685d0 legacy=0x7fbfe409b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 >> v1:172.21.15.148:6789/0 conn(0x7fbfe4136e10 legacy=0x7fbfe413f350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:04.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:04.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:04.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe4143aa0 con 0x7fbfe409bfc0 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 5 monclient: authenticate success, global_id 5566 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbfd8001610 con 0x7fbfe409bfc0 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:04.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: dump: 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfd8001930 con 0x7fbfe409bfc0 2024-09-16T04:10:04.824 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 1 -- 172.21.15.171:0/603103059 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbfd8001d00 con 0x7fbfe409bfc0 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 4 set_mon_vals no callback set 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe27fc640 10 monclient: dump: 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.825 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 10 monclient: shutdown 2024-09-16T04:10:04.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 1 -- 172.21.15.171:0/603103059 >> v1:172.21.15.171:6789/0 conn(0x7fbfe409bfc0 legacy=0x7fbfe413bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 1 -- 172.21.15.171:0/603103059 shutdown_connections 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.805+0000 7fbfe9e47640 1 -- 172.21.15.171:0/603103059 >> 172.21.15.171:0/603103059 conn(0x7fbfe40a1e90 msgr2=0x7fbfe40a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.806+0000 7fbfe9e47640 1 -- 172.21.15.171:0/603103059 shutdown_connections 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.806+0000 7fbfe9e47640 1 -- 172.21.15.171:0/603103059 wait complete. 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.806+0000 7fbfe9e47640 10 monclient: build_initial_monmap 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.806+0000 7fbfe9e47640 10 monclient: monmap: 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:04.827 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.806+0000 7fbfe9e47640 1 Processor -- start 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 1 -- start start 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient: init 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:04.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe40685d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe409bfc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe4136e10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe413de00 con 0x7fbfe4136e10 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe4140800 con 0x7fbfe409bfc0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe40a00b0 con 0x7fbfe40685d0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe9e47640 10 monclient(hunting): authenticate will time out at 980.312449s 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe2ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbfe409bfc0 0x7fbfe413aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:57012/0 (socket says 172.21.15.171:57012) 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe2ffd640 1 -- 172.21.15.171:0/3334449465 learned_addr learned my addr 172.21.15.171:0/3334449465 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfe3fff640 1 -- 172.21.15.171:0/3334449465 >> v1:172.21.15.148:6789/0 conn(0x7fbfe4136e10 legacy=0x7fbfe41d3a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528082327 0 0) 0x7fbfe4140800 con 0x7fbfe409bfc0 2024-09-16T04:10:04.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): my global_id is 5569 2024-09-16T04:10:04.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd0003520 con 0x7fbfe409bfc0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1881076954 0 0) 0x7fbfd0003520 con 0x7fbfe409bfc0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfe4140800 con 0x7fbfe409bfc0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3656968309 0 0) 0x7fbfe40a00b0 con 0x7fbfe40685d0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): my global_id is 5600 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd0003520 con 0x7fbfe40685d0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbfd80015d0 con 0x7fbfe409bfc0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:04.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): dump: 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.807+0000 7fbfcbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:04.832 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3932661843 0 0) 0x7fbfe4140800 con 0x7fbfe409bfc0 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 >> v1:172.21.15.148:6790/0 conn(0x7fbfe40685d0 legacy=0x7fbfe413a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 >> v1:172.21.15.148:6789/0 conn(0x7fbfe4136e10 legacy=0x7fbfe41d3a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 monclient: found mon.b 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe409c580 con 0x7fbfe409bfc0 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:04.833 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 5 monclient: authenticate success, global_id 5569 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 10 monclient: _renew_subs 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe40a0600 con 0x7fbfe409bfc0 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 10 monclient: _renew_subs 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbfe413b1e0 con 0x7fbfe409bfc0 2024-09-16T04:10:04.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfd8001930 con 0x7fbfe409bfc0 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbfd8004e60 con 0x7fbfe409bfc0 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 10 monclient: dump: 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:04.835 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfe1ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.808+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbfd8010290 con 0x7fbfe409bfc0 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.809+0000 7fbfe9e47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.809+0000 7fbfe9e47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.809+0000 7fbfe9e47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.809+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc4005080 con 0x7fbfe409bfc0 2024-09-16T04:10:04.836 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.809+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(127..127 src has 1..127) ==== 7393+0+0 (unknown 1062140266 0 0) 0x7fbfd8047cb0 con 0x7fbfe409bfc0 2024-09-16T04:10:04.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.809+0000 7fbfe37fe640 10 monclient: get_auth_request con 0x7fbfd0039010 auth_method 0 2024-09-16T04:10:04.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.813+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbfd8015d30 con 0x7fbfe409bfc0 2024-09-16T04:10:04.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.813+0000 7fbfcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:04.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.813+0000 7fbfcbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:04.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.988+0000 7fbfe9e47640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T04:10:04.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.988+0000 7fbfe9e47640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T04:10:04.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.988+0000 7fbfe9e47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:04.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:04.988+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) -- 0x7fbfc4005370 con 0x7fbfe409bfc0 2024-09-16T04:10:05.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.156+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v128) ==== 139+0+0 (unknown 2653420188 0 0) 0x7fbfd8015630 con 0x7fbfe409bfc0 2024-09-16T04:10:05.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.156+0000 7fbfcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T04:10:05.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.156+0000 7fbfcbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:05.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.265+0000 7fbfe9e47640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T04:10:05.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.265+0000 7fbfe9e47640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T04:10:05.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.265+0000 7fbfe9e47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:05.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.265+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) -- 0x7fbfc4005ab0 con 0x7fbfe409bfc0 2024-09-16T04:10:05.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.996+0000 7fbfcbfff640 1 -- 172.21.15.171:0/3334449465 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v129) ==== 139+0+0 (unknown 2051425552 0 0) 0x7fbfd801e180 con 0x7fbfe409bfc0 2024-09-16T04:10:05.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.996+0000 7fbfcbfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-16T04:10:05.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.996+0000 7fbfcbfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:06.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 >> v1:172.21.15.171:6812/7327273 conn(0x7fbfd0039010 legacy=0x7fbfd003b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 10 monclient: shutdown 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 >> v1:172.21.15.171:6789/0 conn(0x7fbfe409bfc0 legacy=0x7fbfe413aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 shutdown_connections 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 >> 172.21.15.171:0/3334449465 conn(0x7fbfe40a1e90 msgr2=0x7fbfe4142410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 shutdown_connections 2024-09-16T04:10:06.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:05.998+0000 7fbfe9e47640 1 -- 172.21.15.171:0/3334449465 wait complete. 2024-09-16T04:10:06.012 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:06.012 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c fooset 2024-09-16T04:10:06.012 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 1 2024-09-16T04:10:06.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.090+0000 7fbd3b0f8640 10 monclient: get_monmap_and_config 2024-09-16T04:10:06.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.090+0000 7fbd3b0f8640 10 monclient: build_initial_monmap 2024-09-16T04:10:06.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.090+0000 7fbd3b0f8640 10 monclient: monmap: 2024-09-16T04:10:06.092 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:06.090938+0000 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:06.090938+0000 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.090+0000 7fbd3b0f8640 1 Processor -- start 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.090+0000 7fbd3b0f8640 1 -- start start 2024-09-16T04:10:06.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.090+0000 7fbd3b0f8640 10 monclient: init 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbd34136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbd34139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbd3413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd340a1340 con 0x7fbd34136b60 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd34137650 con 0x7fbd3413d620 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd340a0aa0 con 0x7fbd34139b10 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd3b0f8640 10 monclient(hunting): authenticate will time out at 981.596512s 2024-09-16T04:10:06.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd38e94640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbd34136b60 legacy=0x7fbd34136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd39695640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbd3413d620 0x7fbd3413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:57034/0 (socket says 172.21.15.171:57034) 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd39695640 1 -- 172.21.15.171:0/2469455121 learned_addr learned my addr 172.21.15.171:0/2469455121 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763351045 0 0) 0x7fbd34137650 con 0x7fbd3413d620 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd337fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd337fe640 10 monclient(hunting): my global_id is 5572 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.091+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd1c003520 con 0x7fbd3413d620 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1086953214 0 0) 0x7fbd1c003520 con 0x7fbd3413d620 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 >> v1:172.21.15.148:6790/0 conn(0x7fbd34139b10 legacy=0x7fbd3413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 >> v1:172.21.15.148:6789/0 conn(0x7fbd34136b60 legacy=0x7fbd34136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:06.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 monclient: found mon.noname-c 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd340a0550 con 0x7fbd3413d620 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd2c001610 con 0x7fbd3413d620 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 5 monclient: authenticate success, global_id 5572 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:06.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: dump: 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd2c001970 con 0x7fbd3413d620 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:06.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 4 set_mon_vals no callback set 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 1 -- 172.21.15.171:0/2469455121 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd2c0048a0 con 0x7fbd3413d620 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd337fe640 10 monclient: dump: 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.104 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 10 monclient: shutdown 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/2469455121 >> v1:172.21.15.171:6789/0 conn(0x7fbd3413d620 legacy=0x7fbd3413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/2469455121 shutdown_connections 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/2469455121 >> 172.21.15.171:0/2469455121 conn(0x7fbd340a98b0 msgr2=0x7fbd340a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/2469455121 shutdown_connections 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.092+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/2469455121 wait complete. 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.093+0000 7fbd3b0f8640 10 monclient: build_initial_monmap 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.093+0000 7fbd3b0f8640 10 monclient: monmap: 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:06.105 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.093+0000 7fbd3b0f8640 1 Processor -- start 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.093+0000 7fbd3b0f8640 1 -- start start 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.093+0000 7fbd3b0f8640 10 monclient: init 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbd34136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:06.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbd34139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbd3413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd34120780 con 0x7fbd34136b60 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd34136700 con 0x7fbd34139b10 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3409ee40 con 0x7fbd3413d620 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 10 monclient(hunting): authenticate will time out at 981.599310s 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd38e94640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fbd34136b60 legacy=0x7fbd341cba10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd33fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbd34139b10 0x7fbd341d0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:57038/0 (socket says 172.21.15.171:57038) 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd33fff640 1 -- 172.21.15.171:0/1142247939 learned_addr learned my addr 172.21.15.171:0/1142247939 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2998357125 0 0) 0x7fbd34136700 con 0x7fbd34139b10 2024-09-16T04:10:06.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): my global_id is 5575 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd20003550 con 0x7fbd34139b10 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160113986 0 0) 0x7fbd3409ee40 con 0x7fbd3413d620 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): my global_id is 5606 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd34136700 con 0x7fbd3413d620 2024-09-16T04:10:06.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1327149231 0 0) 0x7fbd20003550 con 0x7fbd34139b10 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd3409ee40 con 0x7fbd34139b10 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd28001930 con 0x7fbd34139b10 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): dump: 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.110 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2261939643 0 0) 0x7fbd3409ee40 con 0x7fbd34139b10 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 >> v1:172.21.15.148:6790/0 conn(0x7fbd3413d620 legacy=0x7fbd341d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 >> v1:172.21.15.148:6789/0 conn(0x7fbd34136b60 legacy=0x7fbd341cba10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 monclient: found mon.b 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd341204a0 con 0x7fbd34139b10 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:06.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.094+0000 7fbd3b0f8640 5 monclient: authenticate success, global_id 5575 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd3b0f8640 10 monclient: _renew_subs 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd3b0f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3409f280 con 0x7fbd34139b10 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd3b0f8640 10 monclient: _renew_subs 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd3b0f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd3409e3d0 con 0x7fbd34139b10 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd28003610 con 0x7fbd34139b10 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd28003960 con 0x7fbd34139b10 2024-09-16T04:10:06.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd32ffd640 4 set_mon_vals no callback set 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 10 monclient: dump: 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.113 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.095+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbd28003b00 con 0x7fbd34139b10 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.096+0000 7fbd3b0f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.096+0000 7fbd3b0f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.096+0000 7fbd3b0f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.096+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd14005080 con 0x7fbd34139b10 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.096+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(129..129 src has 1..129) ==== 7393+0+0 (unknown 3687590271 0 0) 0x7fbd28047bb0 con 0x7fbd34139b10 2024-09-16T04:10:06.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.096+0000 7fbd38e94640 10 monclient: get_auth_request con 0x7fbd20038f70 auth_method 0 2024-09-16T04:10:06.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.100+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbd28015cb0 con 0x7fbd34139b10 2024-09-16T04:10:06.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.100+0000 7fbd317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.100+0000 7fbd317fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:06.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.276+0000 7fbd3b0f8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:06.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.276+0000 7fbd3b0f8640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:06.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.276+0000 7fbd3b0f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.276+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fbd14005370 con 0x7fbd34139b10 2024-09-16T04:10:06.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.277+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v129) ==== 71+0+14 (unknown 417425193 0 3003798303) 0x7fbd280155b0 con 0x7fbd34139b10 2024-09-16T04:10:06.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.277+0000 7fbd317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:06.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.277+0000 7fbd317fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.383+0000 7fbd3b0f8640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.383+0000 7fbd3b0f8640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.383+0000 7fbd3b0f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.383+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fbd14000bb0 con 0x7fbd34139b10 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.384+0000 7fbd317fa640 1 -- 172.21.15.171:0/1142247939 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v129) ==== 71+0+14 (unknown 417425193 0 3003798303) 0x7fbd28019900 con 0x7fbd34139b10 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.384+0000 7fbd317fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:06.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.384+0000 7fbd317fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:06.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 >> v1:172.21.15.171:6812/7327273 conn(0x7fbd20038f70 legacy=0x7fbd2003b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 10 monclient: shutdown 2024-09-16T04:10:06.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:06.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 >> v1:172.21.15.171:6789/0 conn(0x7fbd34139b10 legacy=0x7fbd341d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 shutdown_connections 2024-09-16T04:10:06.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 >> 172.21.15.171:0/1142247939 conn(0x7fbd340a98b0 msgr2=0x7fbd34139120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:06.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 shutdown_connections 2024-09-16T04:10:06.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.386+0000 7fbd3b0f8640 1 -- 172.21.15.171:0/1142247939 wait complete. 2024-09-16T04:10:06.397 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set rm fooset 2024-09-16T04:10:06.477 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.475+0000 7f8a7715f640 10 monclient: get_monmap_and_config 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.475+0000 7f8a7715f640 10 monclient: build_initial_monmap 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.475+0000 7f8a7715f640 10 monclient: monmap: 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:06.476691+0000 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:06.476691+0000 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:06.478 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.479 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:06.479 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:06.479 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:06.479 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.476+0000 7f8a7715f640 1 Processor -- start 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.476+0000 7f8a7715f640 1 -- start start 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.476+0000 7f8a7715f640 10 monclient: init 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient(hunting): picked mon.noname-b con 0x7f8a70132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:06.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient(hunting): picked mon.noname-a con 0x7f8a70132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:06.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a701332b0 con 0x7f8a70132e60 2024-09-16T04:10:06.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a70134310 con 0x7f8a7009ec20 2024-09-16T04:10:06.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7009d980 con 0x7f8a70132a80 2024-09-16T04:10:06.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:06.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient(hunting): authenticate will time out at 981.982201s 2024-09-16T04:10:06.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8a7009ec20 0x7f8a70133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:57040/0 (socket says 172.21.15.171:57040) 2024-09-16T04:10:06.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6ffff640 1 -- 172.21.15.171:0/2330406711 learned_addr learned my addr 172.21.15.171:0/2330406711 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:06.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a756fc640 1 -- 172.21.15.171:0/2330406711 >> v1:172.21.15.148:6789/0 conn(0x7f8a70132e60 legacy=0x7f8a7013c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:06.489 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2682328939 0 0) 0x7f8a70134310 con 0x7f8a7009ec20 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient(hunting): my global_id is 5578 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a5c003520 con 0x7f8a7009ec20 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1854077754 0 0) 0x7f8a5c003520 con 0x7f8a7009ec20 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 >> v1:172.21.15.148:6790/0 conn(0x7f8a70132a80 legacy=0x7f8a701334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 >> v1:172.21.15.148:6789/0 conn(0x7f8a70132e60 legacy=0x7f8a7013c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.490 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7009dc60 con 0x7f8a7009ec20 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 5 monclient: authenticate success, global_id 5578 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8a58001610 con 0x7f8a7009ec20 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: dump: 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.491 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a58001900 con 0x7f8a7009ec20 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 1 -- 172.21.15.171:0/2330406711 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8a58001d10 con 0x7f8a7009ec20 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:06.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a7715f640 4 set_mon_vals no callback set 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.477+0000 7f8a6f7fe640 10 monclient: dump: 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 10 monclient: shutdown 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 1 -- 172.21.15.171:0/2330406711 >> v1:172.21.15.171:6789/0 conn(0x7f8a7009ec20 legacy=0x7f8a70133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 1 -- 172.21.15.171:0/2330406711 shutdown_connections 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 1 -- 172.21.15.171:0/2330406711 >> 172.21.15.171:0/2330406711 conn(0x7f8a700a52a0 msgr2=0x7f8a700a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 1 -- 172.21.15.171:0/2330406711 shutdown_connections 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.478+0000 7f8a7715f640 1 -- 172.21.15.171:0/2330406711 wait complete. 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient: build_initial_monmap 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient: monmap: 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:06.494 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 1 Processor -- start 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 1 -- start start 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient: init 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.479+0000 7f8a7715f640 10 monclient(hunting): picked mon.noname-c con 0x7f8a70132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:06.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient(hunting): picked mon.noname-b con 0x7f8a70132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7006ceb0 con 0x7f8a7009ec20 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a700a3dd0 con 0x7f8a70132e60 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7009c980 con 0x7f8a70132a80 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient(hunting): authenticate will time out at 981.985067s 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a74efb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8a7009ec20 legacy=0x7f8a701cbad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a756fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8a70132e60 0x7f8a701d3df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:57056/0 (socket says 172.21.15.171:57056) 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a756fc640 1 -- 172.21.15.171:0/3577235552 learned_addr learned my addr 172.21.15.171:0/3577235552 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1673486601 0 0) 0x7f8a700a3dd0 con 0x7f8a70132e60 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): my global_id is 5581 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a50003520 con 0x7f8a70132e60 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1675595498 0 0) 0x7f8a7009c980 con 0x7f8a70132a80 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): my global_id is 5612 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a700a3dd0 con 0x7f8a70132a80 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3922936710 0 0) 0x7f8a50003520 con 0x7f8a70132e60 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a7009c980 con 0x7f8a70132e60 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8a68004440 con 0x7f8a70132e60 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:06.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): dump: 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:06.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1286821005 0 0) 0x7f8a7009c980 con 0x7f8a70132e60 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 >> v1:172.21.15.148:6790/0 conn(0x7f8a70132a80 legacy=0x7f8a701d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 >> v1:172.21.15.148:6789/0 conn(0x7f8a7009ec20 legacy=0x7f8a701cbad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 monclient: found mon.b 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a70122bc0 con 0x7f8a70132e60 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 5 monclient: authenticate success, global_id 5581 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient: _renew_subs 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a70122ea0 con 0x7f8a70132e60 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient: _renew_subs 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a701cc1e0 con 0x7f8a70132e60 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a680046a0 con 0x7f8a70132e60 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8a680049f0 con 0x7f8a70132e60 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6effd640 4 set_mon_vals no callback set 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.480+0000 7f8a6d7fa640 10 monclient: dump: 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:06.500 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8a68004c30 con 0x7f8a70132e60 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a7715f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a7715f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a7715f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a54005080 con 0x7f8a70132e60 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(129..129 src has 1..129) ==== 7393+0+0 (unknown 3687590271 0 0) 0x7f8a68047ee0 con 0x7f8a70132e60 2024-09-16T04:10:06.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.481+0000 7f8a74efb640 10 monclient: get_auth_request con 0x7f8a50038fe0 auth_method 0 2024-09-16T04:10:06.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.486+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8a68019b50 con 0x7f8a70132e60 2024-09-16T04:10:06.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.486+0000 7f8a6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.486+0000 7f8a6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:06.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.663+0000 7f8a7715f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T04:10:06.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.663+0000 7f8a7715f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T04:10:06.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.663+0000 7f8a7715f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:06.665 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:06.663+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) -- 0x7f8a54005370 con 0x7f8a70132e60 2024-09-16T04:10:07.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.177+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v130) ==== 89+0+0 (unknown 3302232522 0 0) 0x7f8a6801e410 con 0x7f8a70132e60 2024-09-16T04:10:07.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.177+0000 7f8a6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T04:10:07.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.177+0000 7f8a6d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:07.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.288+0000 7f8a7715f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T04:10:07.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.288+0000 7f8a7715f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T04:10:07.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.288+0000 7f8a7715f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:07.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:07.288+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) -- 0x7f8a54002210 con 0x7f8a70132e60 2024-09-16T04:10:08.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.181+0000 7f8a6d7fa640 1 -- 172.21.15.171:0/3577235552 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v131) ==== 89+0+0 (unknown 2259037475 0 0) 0x7f8a68044e40 con 0x7f8a70132e60 2024-09-16T04:10:08.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.181+0000 7f8a6d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-16T04:10:08.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.181+0000 7f8a6d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 >> v1:172.21.15.171:6812/7327273 conn(0x7f8a50038fe0 legacy=0x7f8a5003b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 10 monclient: shutdown 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 >> v1:172.21.15.171:6789/0 conn(0x7f8a70132e60 legacy=0x7f8a701d3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 shutdown_connections 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 >> 172.21.15.171:0/3577235552 conn(0x7f8a700a52a0 msgr2=0x7f8a70131c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 shutdown_connections 2024-09-16T04:10:08.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.185+0000 7f8a7715f640 1 -- 172.21.15.171:0/3577235552 wait complete. 2024-09-16T04:10:08.201 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:08.204 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c fooset 2024-09-16T04:10:08.205 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 0 2024-09-16T04:10:08.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.282+0000 7f2c708ba640 10 monclient: get_monmap_and_config 2024-09-16T04:10:08.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.282+0000 7f2c708ba640 10 monclient: build_initial_monmap 2024-09-16T04:10:08.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.282+0000 7f2c708ba640 10 monclient: monmap: 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:08.283389+0000 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:08.283389+0000 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:08.285 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 1 Processor -- start 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 1 -- start start 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient: init 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c1327c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c09eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c139ea0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c09ef30 con 0x7f2c6c139ea0 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c09d880 con 0x7f2c6c09eb20 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c09db60 con 0x7f2c6c1327c0 2024-09-16T04:10:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.283+0000 7f2c708ba640 10 monclient(hunting): authenticate will time out at 983.788931s 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c6ad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2c6c139ea0 legacy=0x7f2c6c13c290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c63fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2c6c09eb20 0x7f2c6c138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42406/0 (socket says 172.21.15.171:42406) 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c63fff640 1 -- 172.21.15.171:0/3736771651 learned_addr learned my addr 172.21.15.171:0/3736771651 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247966783 0 0) 0x7f2c6c09d880 con 0x7f2c6c09eb20 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): my global_id is 5584 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c58003520 con 0x7f2c6c09eb20 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4025903087 0 0) 0x7f2c6c09db60 con 0x7f2c6c1327c0 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient(hunting): my global_id is 5615 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6c09d880 con 0x7f2c6c1327c0 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3715790812 0 0) 0x7f2c58003520 con 0x7f2c6c09eb20 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 >> v1:172.21.15.148:6790/0 conn(0x7f2c6c1327c0 legacy=0x7f2c6c132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 >> v1:172.21.15.148:6789/0 conn(0x7f2c6c139ea0 legacy=0x7f2c6c13c290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:08.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c09de40 con 0x7f2c6c09eb20 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: _finish_auth 0 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c54001590 con 0x7f2c6c09eb20 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c708ba640 5 monclient: authenticate success, global_id 5584 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c708ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: dump: 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.296 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c708ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c540017f0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 1 -- 172.21.15.171:0/3736771651 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c54001b40 con 0x7f2c6c09eb20 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c708ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:08.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c708ba640 4 set_mon_vals no callback set 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.284+0000 7f2c69d9b640 10 monclient: dump: 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 10 monclient: shutdown 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 1 -- 172.21.15.171:0/3736771651 >> v1:172.21.15.171:6789/0 conn(0x7f2c6c09eb20 legacy=0x7f2c6c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 1 -- 172.21.15.171:0/3736771651 shutdown_connections 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 1 -- 172.21.15.171:0/3736771651 >> 172.21.15.171:0/3736771651 conn(0x7f2c6c0a5370 msgr2=0x7f2c6c0a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 1 -- 172.21.15.171:0/3736771651 shutdown_connections 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.285+0000 7f2c708ba640 1 -- 172.21.15.171:0/3736771651 wait complete. 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient: build_initial_monmap 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient: monmap: 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:08.299 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 1 Processor -- start 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 1 -- start start 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient: init 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c09eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c1327c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c139ea0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:08.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c11c050 con 0x7f2c6c139ea0 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c131db0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c11b200 con 0x7f2c6c1327c0 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c708ba640 10 monclient(hunting): authenticate will time out at 983.791897s 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c6a59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2c6c09eb20 0x7f2c6c1316a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42412/0 (socket says 172.21.15.171:42412) 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c6ad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2c6c139ea0 legacy=0x7f2c6c1cfc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.286+0000 7f2c6a59c640 1 -- 172.21.15.171:0/2123950285 learned_addr learned my addr 172.21.15.171:0/2123950285 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660495127 0 0) 0x7f2c6c131db0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): my global_id is 5587 2024-09-16T04:10:08.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c4c003520 con 0x7f2c6c09eb20 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720150677 0 0) 0x7f2c6c11b200 con 0x7f2c6c1327c0 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): my global_id is 5618 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6c131db0 con 0x7f2c6c1327c0 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1622398048 0 0) 0x7f2c4c003520 con 0x7f2c6c09eb20 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c6c11b200 con 0x7f2c6c09eb20 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c5c001970 con 0x7f2c6c09eb20 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.302 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3607619989 0 0) 0x7f2c6c131db0 con 0x7f2c6c1327c0 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c4c003520 con 0x7f2c6c1327c0 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c54001510 con 0x7f2c6c1327c0 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:08.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3682994830 0 0) 0x7f2c6c11b200 con 0x7f2c6c09eb20 2024-09-16T04:10:08.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 >> v1:172.21.15.148:6790/0 conn(0x7f2c6c1327c0 legacy=0x7f2c6c1cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 >> v1:172.21.15.148:6789/0 conn(0x7f2c6c139ea0 legacy=0x7f2c6c1cfc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 monclient: found mon.b 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c06ce80 con 0x7f2c6c09eb20 2024-09-16T04:10:08.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 5 monclient: authenticate success, global_id 5587 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 10 monclient: _renew_subs 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c071b00 con 0x7f2c6c09eb20 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 10 monclient: _renew_subs 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c6c0723a0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c5c001d20 con 0x7f2c6c09eb20 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c5c003a60 con 0x7f2c6c09eb20 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:08.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c6959a640 4 set_mon_vals no callback set 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.287+0000 7f2c62ffd640 10 monclient: dump: 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.288+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2c5c003c80 con 0x7f2c6c09eb20 2024-09-16T04:10:08.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.288+0000 7f2c708ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.288+0000 7f2c708ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.288+0000 7f2c708ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.288+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c50005080 con 0x7f2c6c09eb20 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.288+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(131..131 src has 1..131) ==== 7061+0+0 (unknown 4177889058 0 0) 0x7f2c5c047b30 con 0x7f2c6c09eb20 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.289+0000 7f2c63fff640 10 monclient: get_auth_request con 0x7f2c4c038ef0 auth_method 0 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.293+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2c5c015da0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.293+0000 7f2c62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:08.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.293+0000 7f2c62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.471+0000 7f2c708ba640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.471+0000 7f2c708ba640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.471+0000 7f2c708ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.471+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f2c50005370 con 0x7f2c6c09eb20 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.472+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) ==== 71+0+7 (unknown 738765755 0 1352219128) 0x7f2c5c0156a0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.472+0000 7f2c62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.472+0000 7f2c62ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:08.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.581+0000 7f2c708ba640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.581+0000 7f2c708ba640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.581+0000 7f2c708ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.581+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7f2c50000bb0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.582+0000 7f2c62ffd640 1 -- 172.21.15.171:0/2123950285 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) ==== 71+0+7 (unknown 738765755 0 1352219128) 0x7f2c5c0199f0 con 0x7f2c6c09eb20 2024-09-16T04:10:08.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.582+0000 7f2c62ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.582+0000 7f2c62ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.583+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 >> v1:172.21.15.171:6812/7327273 conn(0x7f2c4c038ef0 legacy=0x7f2c4c03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 10 monclient: shutdown 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 >> v1:172.21.15.171:6789/0 conn(0x7f2c6c09eb20 legacy=0x7f2c6c1316a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 shutdown_connections 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 >> 172.21.15.171:0/2123950285 conn(0x7f2c6c0a5370 msgr2=0x7f2c6c0a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 shutdown_connections 2024-09-16T04:10:08.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.584+0000 7f2c708ba640 1 -- 172.21.15.171:0/2123950285 wait complete. 2024-09-16T04:10:08.595 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:08.596 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep barset 2024-09-16T04:10:08.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.672+0000 7fa10d71f640 10 monclient: get_monmap_and_config 2024-09-16T04:10:08.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.672+0000 7fa10d71f640 10 monclient: build_initial_monmap 2024-09-16T04:10:08.674 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.672+0000 7fa10d71f640 10 monclient: monmap: 2024-09-16T04:10:08.674 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:08.674 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.674 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:08.673339+0000 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:08.673339+0000 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:08.675 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 1 Processor -- start 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 1 -- start start 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient: init 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient(hunting): picked mon.noname-b con 0x7fa1081327c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient(hunting): picked mon.noname-a con 0x7fa10809eb20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient(hunting): picked mon.noname-c con 0x7fa108139ea0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10809ef30 con 0x7fa10809eb20 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10809d880 con 0x7fa108139ea0 2024-09-16T04:10:08.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10809db60 con 0x7fa1081327c0 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.673+0000 7fa10d71f640 10 monclient(hunting): authenticate will time out at 984.178995s 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa0fffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa10809eb20 legacy=0x7fa108138780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1077fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa108139ea0 0x7fa10813c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42422/0 (socket says 172.21.15.171:42422) 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1077fe640 1 -- 172.21.15.171:0/3183242407 learned_addr learned my addr 172.21.15.171:0/3183242407 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2889759666 0 0) 0x7fa10809d880 con 0x7fa108139ea0 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): my global_id is 5590 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0f4003520 con 0x7fa108139ea0 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928129985 0 0) 0x7fa10809db60 con 0x7fa1081327c0 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient(hunting): my global_id is 5621 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa10809d880 con 0x7fa1081327c0 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1505246487 0 0) 0x7fa0f4003520 con 0x7fa108139ea0 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 >> v1:172.21.15.148:6790/0 conn(0x7fa1081327c0 legacy=0x7fa108132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 >> v1:172.21.15.148:6789/0 conn(0x7fa10809eb20 legacy=0x7fa108138780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10809de40 con 0x7fa108139ea0 2024-09-16T04:10:08.684 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa100001990 con 0x7fa108139ea0 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa10d71f640 5 monclient: authenticate success, global_id 5590 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa10d71f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: dump: 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.685 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa10d71f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa100001c40 con 0x7fa108139ea0 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa10d71f640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 1 -- 172.21.15.171:0/3183242407 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa100002010 con 0x7fa108139ea0 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa10d71f640 4 set_mon_vals no callback set 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.674+0000 7fa1067fc640 10 monclient: dump: 2024-09-16T04:10:08.686 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.687 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 10 monclient: shutdown 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 1 -- 172.21.15.171:0/3183242407 >> v1:172.21.15.171:6789/0 conn(0x7fa108139ea0 legacy=0x7fa10813c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 1 -- 172.21.15.171:0/3183242407 shutdown_connections 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 1 -- 172.21.15.171:0/3183242407 >> 172.21.15.171:0/3183242407 conn(0x7fa1080a5370 msgr2=0x7fa1080a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 1 -- 172.21.15.171:0/3183242407 shutdown_connections 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.675+0000 7fa10d71f640 1 -- 172.21.15.171:0/3183242407 wait complete. 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 10 monclient: build_initial_monmap 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 10 monclient: monmap: 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:08.688 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 1 Processor -- start 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 1 -- start start 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 10 monclient: init 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.676+0000 7fa10d71f640 10 monclient(hunting): picked mon.noname-b con 0x7fa10809eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:08.689 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1081327c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient(hunting): picked mon.noname-a con 0x7fa108139ea0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10806ce80 con 0x7fa108139ea0 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10811b8a0 con 0x7fa10809eb20 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa10811bb80 con 0x7fa1081327c0 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient(hunting): authenticate will time out at 984.182108s 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa106ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa10809eb20 0x7fa108131300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42426/0 (socket says 172.21.15.171:42426) 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa106ffd640 1 -- 172.21.15.171:0/3170637077 learned_addr learned my addr 172.21.15.171:0/3170637077 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa1077fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa108139ea0 legacy=0x7fa1081cfa60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444323712 0 0) 0x7fa10811b8a0 con 0x7fa10809eb20 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.690 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): my global_id is 5593 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0e8003520 con 0x7fa10809eb20 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039169337 0 0) 0x7fa10811bb80 con 0x7fa1081327c0 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): my global_id is 5624 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa10811b8a0 con 0x7fa1081327c0 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2640647912 0 0) 0x7fa0e8003520 con 0x7fa10809eb20 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa10811bb80 con 0x7fa10809eb20 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0f8001970 con 0x7fa10809eb20 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:08.691 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1180575544 0 0) 0x7fa10811bb80 con 0x7fa10809eb20 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 >> v1:172.21.15.148:6790/0 conn(0x7fa1081327c0 legacy=0x7fa1081cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.692 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 >> v1:172.21.15.148:6789/0 conn(0x7fa108139ea0 legacy=0x7fa1081cfa60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 monclient: found mon.b 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10811b200 con 0x7fa10809eb20 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 5 monclient: authenticate success, global_id 5593 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient: _renew_subs 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1080a0630 con 0x7fa10809eb20 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient: _renew_subs 2024-09-16T04:10:08.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa108131a10 con 0x7fa10809eb20 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa0f8001d20 con 0x7fa10809eb20 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa0f8003a60 con 0x7fa10809eb20 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa0ff7fe640 10 monclient: dump: 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:08.694 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.677+0000 7fa105ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.678+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa0f8003c80 con 0x7fa10809eb20 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.678+0000 7fa10d71f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.678+0000 7fa10d71f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.678+0000 7fa10d71f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.678+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa108137c70 con 0x7fa10809eb20 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.679+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(131..131 src has 1..131) ==== 7061+0+0 (unknown 4177889058 0 0) 0x7fa0f8047b30 con 0x7fa10809eb20 2024-09-16T04:10:08.695 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.679+0000 7fa0fffff640 10 monclient: get_auth_request con 0x7fa0e8038f60 auth_method 0 2024-09-16T04:10:08.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.683+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa0f8015de0 con 0x7fa10809eb20 2024-09-16T04:10:08.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.683+0000 7fa0ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:08.696 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.683+0000 7fa0ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:08.858 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.856+0000 7fa10d71f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.856+0000 7fa10d71f640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.856+0000 7fa10d71f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.856+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fa10811bec0 con 0x7fa10809eb20 2024-09-16T04:10:08.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.857+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) ==== 71+0+7 (unknown 738765755 0 1352219128) 0x7fa0f80156e0 con 0x7fa10809eb20 2024-09-16T04:10:08.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.857+0000 7fa0ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.857+0000 7fa0ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:08.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa10d71f640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa10d71f640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa10d71f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:08.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fa1080008d0 con 0x7fa10809eb20 2024-09-16T04:10:08.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa0ff7fe640 1 -- 172.21.15.171:0/3170637077 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v131) ==== 71+0+7 (unknown 738765755 0 1352219128) 0x7fa0f801e290 con 0x7fa10809eb20 2024-09-16T04:10:08.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa0ff7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:08.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.963+0000 7fa0ff7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 >> v1:172.21.15.171:6812/7327273 conn(0x7fa0e8038f60 legacy=0x7fa0e803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 10 monclient: shutdown 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 >> v1:172.21.15.171:6789/0 conn(0x7fa10809eb20 legacy=0x7fa108131300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 shutdown_connections 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 >> 172.21.15.171:0/3170637077 conn(0x7fa1080a5370 msgr2=0x7fa1080a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.965+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 shutdown_connections 2024-09-16T04:10:08.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:08.966+0000 7fa10d71f640 1 -- 172.21.15.171:0/3170637077 wait complete. 2024-09-16T04:10:08.977 INFO:tasks.workunit.client.0.smithi171.stdout:barset 2024-09-16T04:10:08.977 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set rm barset 2024-09-16T04:10:09.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.054+0000 7f13a8077640 10 monclient: get_monmap_and_config 2024-09-16T04:10:09.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.054+0000 7f13a8077640 10 monclient: build_initial_monmap 2024-09-16T04:10:09.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.054+0000 7f13a8077640 10 monclient: monmap: 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:09.055560+0000 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:09.055560+0000 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:09.057 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 1 Processor -- start 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 1 -- start start 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 10 monclient: init 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 10 monclient(hunting): picked mon.noname-a con 0x7f13a0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:09.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 10 monclient(hunting): picked mon.noname-b con 0x7f13a009ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 10 monclient(hunting): picked mon.noname-c con 0x7f13a0132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a01332b0 con 0x7f13a0132a80 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.055+0000 7f13a8077640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a0134310 con 0x7f13a0132e60 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a009d980 con 0x7f13a009ec20 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 10 monclient(hunting): authenticate will time out at 984.561019s 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a5e13640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f13a0132a80 legacy=0x7f13a01334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a6614640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f13a0132e60 0x7f13a013c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42428/0 (socket says 172.21.15.171:42428) 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a6614640 1 -- 172.21.15.171:0/2355065981 learned_addr learned my addr 172.21.15.171:0/2355065981 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818024854 0 0) 0x7f13a0134310 con 0x7f13a0132e60 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient(hunting): _negotiate 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:09.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient(hunting): my global_id is 5596 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138c003520 con 0x7f13a0132e60 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2632776658 0 0) 0x7f138c003520 con 0x7f13a0132e60 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 >> v1:172.21.15.148:6790/0 conn(0x7f13a009ec20 legacy=0x7f13a0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 >> v1:172.21.15.148:6789/0 conn(0x7f13a0132a80 legacy=0x7f13a01334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: _finish_hunting 0 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 monclient: found mon.noname-c 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a009dc60 con 0x7f13a0132e60 2024-09-16T04:10:09.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: _finish_auth 0 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: _check_auth_tickets 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 5 monclient: authenticate success, global_id 5596 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1398001990 con 0x7f13a0132e60 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: dump: 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.067 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1398001c00 con 0x7f13a0132e60 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 1 -- 172.21.15.171:0/2355065981 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1398002010 con 0x7f13a0132e60 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a8077640 4 set_mon_vals no callback set 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:09.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.056+0000 7f13a4e11640 10 monclient: dump: 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 10 monclient: shutdown 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 1 -- 172.21.15.171:0/2355065981 >> v1:172.21.15.171:6789/0 conn(0x7f13a0132e60 legacy=0x7f13a013c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 1 -- 172.21.15.171:0/2355065981 shutdown_connections 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 1 -- 172.21.15.171:0/2355065981 >> 172.21.15.171:0/2355065981 conn(0x7f13a00a52a0 msgr2=0x7f13a00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 1 -- 172.21.15.171:0/2355065981 shutdown_connections 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 1 -- 172.21.15.171:0/2355065981 wait complete. 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 10 monclient: build_initial_monmap 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.057+0000 7f13a8077640 10 monclient: monmap: 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:09.070 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 1 Processor -- start 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 1 -- start start 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient: init 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient(hunting): picked mon.noname-b con 0x7f13a009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient(hunting): picked mon.noname-a con 0x7f13a0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient(hunting): picked mon.noname-c con 0x7f13a0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a011db00 con 0x7f13a0132a80 2024-09-16T04:10:09.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a00723b0 con 0x7f13a009ec20 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a009c730 con 0x7f13a0132e60 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a8077640 10 monclient(hunting): authenticate will time out at 984.563674s 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a5e13640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f13a009ec20 0x7f13a0071590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42430/0 (socket says 172.21.15.171:42430) 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a5e13640 1 -- 172.21.15.171:0/3834530675 learned_addr learned my addr 172.21.15.171:0/3834530675 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13a5612640 1 -- 172.21.15.171:0/3834530675 >> v1:172.21.15.148:6789/0 conn(0x7f13a0132a80 legacy=0x7f13a0071ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3776686021 0 0) 0x7f13a00723b0 con 0x7f13a009ec20 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13967fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13967fc640 10 monclient(hunting): my global_id is 5599 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.058+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1384003520 con 0x7f13a009ec20 2024-09-16T04:10:09.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1220344896 0 0) 0x7f13a009c730 con 0x7f13a0132e60 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): my global_id is 5630 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13a00723b0 con 0x7f13a0132e60 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2428108987 0 0) 0x7f1384003520 con 0x7f13a009ec20 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13a009c730 con 0x7f13a009ec20 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f139c004180 con 0x7f13a009ec20 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): dump: 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:09.073 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3038054680 0 0) 0x7f13a009c730 con 0x7f13a009ec20 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 >> v1:172.21.15.148:6790/0 conn(0x7f13a0132e60 legacy=0x7f13a01cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 >> v1:172.21.15.148:6789/0 conn(0x7f13a0132a80 legacy=0x7f13a0071ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:09.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 monclient: found mon.b 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a006ceb0 con 0x7f13a009ec20 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 5 monclient: authenticate success, global_id 5599 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 10 monclient: _renew_subs 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13a011d400 con 0x7f13a009ec20 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 10 monclient: _renew_subs 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:09.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f13a011c170 con 0x7f13a009ec20 2024-09-16T04:10:09.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f139c004430 con 0x7f13a009ec20 2024-09-16T04:10:09.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:09.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f139c004800 con 0x7f13a009ec20 2024-09-16T04:10:09.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:09.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f13967fc640 10 monclient: dump: 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:09.077 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.059+0000 7f1397fff640 4 set_mon_vals no callback set 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f139c004a60 con 0x7f13a009ec20 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13a8077640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13a8077640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13a8077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a00008d0 con 0x7f13a009ec20 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(131..131 src has 1..131) ==== 7061+0+0 (unknown 4177889058 0 0) 0x7f139c047ac0 con 0x7f13a009ec20 2024-09-16T04:10:09.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.060+0000 7f13a5612640 10 monclient: get_auth_request con 0x7f1384038f60 auth_method 0 2024-09-16T04:10:09.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.064+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f139c015d10 con 0x7f13a009ec20 2024-09-16T04:10:09.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.064+0000 7f13967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:09.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.064+0000 7f13967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:09.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.243+0000 7f13a8077640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T04:10:09.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.243+0000 7f13a8077640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T04:10:09.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.243+0000 7f13a8077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:09.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:09.243+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) -- 0x7f13a012e230 con 0x7f13a009ec20 2024-09-16T04:10:10.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.193+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v132) ==== 89+0+0 (unknown 2869787261 0 0) 0x7f139c015610 con 0x7f13a009ec20 2024-09-16T04:10:10.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.193+0000 7f13967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T04:10:10.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.193+0000 7f13967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:10.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.308+0000 7f13a8077640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T04:10:10.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.308+0000 7f13a8077640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T04:10:10.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.308+0000 7f13a8077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:10.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.308+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) -- 0x7f13a01d0b40 con 0x7f13a009ec20 2024-09-16T04:10:10.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.997+0000 7f13967fc640 1 -- 172.21.15.171:0/3834530675 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v133) ==== 89+0+0 (unknown 3917432468 0 0) 0x7f139c019960 con 0x7f13a009ec20 2024-09-16T04:10:10.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.997+0000 7f13967fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-16T04:10:10.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:10.997+0000 7f13967fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:11.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 >> v1:172.21.15.171:6812/7327273 conn(0x7f1384038f60 legacy=0x7f138403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 10 monclient: shutdown 2024-09-16T04:10:11.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:11.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 >> v1:172.21.15.171:6789/0 conn(0x7f13a009ec20 legacy=0x7f13a0071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 shutdown_connections 2024-09-16T04:10:11.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 >> 172.21.15.171:0/3834530675 conn(0x7f13a00a52a0 msgr2=0x7f13a0133320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:11.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 shutdown_connections 2024-09-16T04:10:11.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.000+0000 7f13a8077640 1 -- 172.21.15.171:0/3834530675 wait complete. 2024-09-16T04:10:11.021 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:11.021 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -c barset 2024-09-16T04:10:11.021 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep -q 0 2024-09-16T04:10:11.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.101+0000 7fa9804d2640 10 monclient: get_monmap_and_config 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.101+0000 7fa9804d2640 10 monclient: build_initial_monmap 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.101+0000 7fa9804d2640 10 monclient: monmap: 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:11.102432+0000 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:11.102432+0000 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 1 Processor -- start 2024-09-16T04:10:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 1 -- start start 2024-09-16T04:10:11.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient: init 2024-09-16T04:10:11.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient(hunting): picked mon.noname-a con 0x7fa97809b660 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient(hunting): picked mon.noname-b con 0x7fa9780734b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient(hunting): picked mon.noname-c con 0x7fa978136d70 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa97809c150 con 0x7fa97809b660 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9780998b0 con 0x7fa978136d70 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa978099b90 con 0x7fa9780734b0 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.102+0000 7fa9804d2640 10 monclient(hunting): authenticate will time out at 986.607916s 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97e26e640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa97809b660 legacy=0x7fa97809ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:11.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97ea6f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa978136d70 0x7fa97813f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42442/0 (socket says 172.21.15.171:42442) 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97ea6f640 1 -- 172.21.15.171:0/3281420083 learned_addr learned my addr 172.21.15.171:0/3281420083 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1109329975 0 0) 0x7fa9780998b0 con 0x7fa978136d70 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient(hunting): _negotiate 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient(hunting): my global_id is 5602 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa964003520 con 0x7fa978136d70 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 262304061 0 0) 0x7fa964003520 con 0x7fa978136d70 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 >> v1:172.21.15.148:6790/0 conn(0x7fa9780734b0 legacy=0x7fa97813bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 >> v1:172.21.15.148:6789/0 conn(0x7fa97809b660 legacy=0x7fa97809ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: _finish_hunting 0 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 monclient: found mon.noname-c 2024-09-16T04:10:11.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa978099e70 con 0x7fa978136d70 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: _finish_auth 0 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: _check_auth_tickets 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa970001990 con 0x7fa978136d70 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: dump: 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.114 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa9804d2640 5 monclient: authenticate success, global_id 5602 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa9804d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa970001c80 con 0x7fa978136d70 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:11.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 1 -- 172.21.15.171:0/3281420083 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa970002090 con 0x7fa978136d70 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa9804d2640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa9804d2640 4 set_mon_vals no callback set 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.103+0000 7fa97d26c640 10 monclient: dump: 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.116 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 10 monclient: shutdown 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 1 -- 172.21.15.171:0/3281420083 >> v1:172.21.15.171:6789/0 conn(0x7fa978136d70 legacy=0x7fa97813f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 1 -- 172.21.15.171:0/3281420083 shutdown_connections 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 1 -- 172.21.15.171:0/3281420083 >> 172.21.15.171:0/3281420083 conn(0x7fa9780a1e90 msgr2=0x7fa9780a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 1 -- 172.21.15.171:0/3281420083 shutdown_connections 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 1 -- 172.21.15.171:0/3281420083 wait complete. 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 10 monclient: build_initial_monmap 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.104+0000 7fa9804d2640 10 monclient: monmap: 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:11.117 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 1 Processor -- start 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 1 -- start start 2024-09-16T04:10:11.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient: init 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient(hunting): picked mon.noname-c con 0x7fa9780734b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient(hunting): picked mon.noname-a con 0x7fa97809b660 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient(hunting): picked mon.noname-b con 0x7fa978136d70 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa97809e110 con 0x7fa97809b660 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa97808efe0 con 0x7fa978136d70 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa978092930 con 0x7fa9780734b0 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa9804d2640 10 monclient(hunting): authenticate will time out at 986.610607s 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa97da6d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa97809b660 legacy=0x7fa9781d0c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa97ea6f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa978136d70 0x7fa9781d4380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42456/0 (socket says 172.21.15.171:42456) 2024-09-16T04:10:11.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa97ea6f640 1 -- 172.21.15.171:0/1157140949 learned_addr learned my addr 172.21.15.171:0/1157140949 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530166794 0 0) 0x7fa97808efe0 con 0x7fa978136d70 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): my global_id is 5605 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa95c003520 con 0x7fa978136d70 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993923186 0 0) 0x7fa978092930 con 0x7fa9780734b0 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:11.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 10 monclient(hunting): my global_id is 5636 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.105+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa97808efe0 con 0x7fa9780734b0 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 336271191 0 0) 0x7fa95c003520 con 0x7fa978136d70 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa978092930 con 0x7fa978136d70 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9700017b0 con 0x7fa978136d70 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient(hunting): dump: 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.121 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 244088081 0 0) 0x7fa97808efe0 con 0x7fa9780734b0 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa95c003520 con 0x7fa9780734b0 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 663996794 0 0) 0x7fa978092930 con 0x7fa978136d70 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 >> v1:172.21.15.148:6790/0 conn(0x7fa9780734b0 legacy=0x7fa9781cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 >> v1:172.21.15.148:6789/0 conn(0x7fa97809b660 legacy=0x7fa9781d0c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:11.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 monclient: found mon.b 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa978092c10 con 0x7fa978136d70 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: _finish_auth 0 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 5 monclient: authenticate success, global_id 5605 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 10 monclient: _renew_subs 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa978092ef0 con 0x7fa978136d70 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 10 monclient: _renew_subs 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9781cc390 con 0x7fa978136d70 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa970001c80 con 0x7fa978136d70 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9700051e0 con 0x7fa978136d70 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa96effd640 10 monclient: dump: 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.124 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.106+0000 7fa97ca6b640 4 set_mon_vals no callback set 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa970010610 con 0x7fa978136d70 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa9804d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa9804d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa9804d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa960005080 con 0x7fa978136d70 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(133..133 src has 1..133) ==== 6449+0+0 (unknown 1143600978 0 0) 0x7fa970047c60 con 0x7fa978136d70 2024-09-16T04:10:11.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.107+0000 7fa97e26e640 10 monclient: get_auth_request con 0x7fa95c038e50 auth_method 0 2024-09-16T04:10:11.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.112+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa97001e620 con 0x7fa978136d70 2024-09-16T04:10:11.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.112+0000 7fa96effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:11.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.112+0000 7fa96effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:11.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.286+0000 7fa9804d2640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:11.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.286+0000 7fa9804d2640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:11.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.286+0000 7fa9804d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.286+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fa960005370 con 0x7fa978136d70 2024-09-16T04:10:11.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.287+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) ==== 71+0+0 (unknown 1899167245 0 0) 0x7fa970013e60 con 0x7fa978136d70 2024-09-16T04:10:11.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.287+0000 7fa96effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:11.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.287+0000 7fa96effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:11.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa9804d2640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:11.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa9804d2640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:11.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa9804d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fa960000bb0 con 0x7fa978136d70 2024-09-16T04:10:11.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa96effd640 1 -- 172.21.15.171:0/1157140949 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v133) ==== 71+0+0 (unknown 1899167245 0 0) 0x7fa970045050 con 0x7fa978136d70 2024-09-16T04:10:11.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa96effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:11.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.393+0000 7fa96effd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:11.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.395+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 >> v1:172.21.15.171:6812/7327273 conn(0x7fa95c038e50 legacy=0x7fa95c03b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.395+0000 7fa9804d2640 10 monclient: shutdown 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.395+0000 7fa9804d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.395+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 >> v1:172.21.15.171:6789/0 conn(0x7fa978136d70 legacy=0x7fa9781d4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.395+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 shutdown_connections 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.395+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 >> 172.21.15.171:0/1157140949 conn(0x7fa9780a1e90 msgr2=0x7fa9780988c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.396+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 shutdown_connections 2024-09-16T04:10:11.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.396+0000 7fa9804d2640 1 -- 172.21.15.171:0/1157140949 wait complete. 2024-09-16T04:10:11.407 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.484+0000 7f31c3913640 10 monclient: get_monmap_and_config 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.484+0000 7f31c3913640 10 monclient: build_initial_monmap 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.484+0000 7f31c3913640 10 monclient: monmap: 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:11.485080+0000 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:11.485080+0000 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:11.486 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.487 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:11.487 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:11.487 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:11.487 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.493 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.484+0000 7f31c3913640 1 Processor -- start 2024-09-16T04:10:11.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 1 -- start start 2024-09-16T04:10:11.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient: init 2024-09-16T04:10:11.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:11.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:11.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc0a1340 con 0x7f31bc139b10 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc137650 con 0x7f31bc13d620 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc0a0aa0 con 0x7f31bc136b60 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c3913640 10 monclient(hunting): authenticate will time out at 986.990550s 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c0eae640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f31bc139b10 legacy=0x7f31bc13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c1eb0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f31bc13d620 0x7f31bc13fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42468/0 (socket says 172.21.15.171:42468) 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31c1eb0640 1 -- 172.21.15.171:0/1302015864 learned_addr learned my addr 172.21.15.171:0/1302015864 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2857371971 0 0) 0x7f31bc137650 con 0x7f31bc13d620 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31b3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:11.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31b3fff640 10 monclient(hunting): my global_id is 5608 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.485+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31a8003520 con 0x7f31bc13d620 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 609424150 0 0) 0x7f31a8003520 con 0x7f31bc13d620 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 >> v1:172.21.15.148:6790/0 conn(0x7f31bc136b60 legacy=0x7f31bc136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 >> v1:172.21.15.148:6789/0 conn(0x7f31bc139b10 legacy=0x7f31bc13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 monclient: found mon.noname-c 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc0a0550 con 0x7f31bc13d620 2024-09-16T04:10:11.496 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31b4001990 con 0x7f31bc13d620 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 5 monclient: authenticate success, global_id 5608 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: dump: 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.497 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31b4001cf0 con 0x7f31bc13d620 2024-09-16T04:10:11.498 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 1 -- 172.21.15.171:0/1302015864 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31b4004c20 con 0x7f31bc13d620 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 4 set_mon_vals no callback set 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31b3fff640 10 monclient: dump: 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.499 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 10 monclient: shutdown 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 1 -- 172.21.15.171:0/1302015864 >> v1:172.21.15.171:6789/0 conn(0x7f31bc13d620 legacy=0x7f31bc13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 1 -- 172.21.15.171:0/1302015864 shutdown_connections 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 1 -- 172.21.15.171:0/1302015864 >> 172.21.15.171:0/1302015864 conn(0x7f31bc0a98b0 msgr2=0x7f31bc0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 1 -- 172.21.15.171:0/1302015864 shutdown_connections 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.486+0000 7f31c3913640 1 -- 172.21.15.171:0/1302015864 wait complete. 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 10 monclient: build_initial_monmap 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 10 monclient: monmap: 2024-09-16T04:10:11.500 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 1 Processor -- start 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 1 -- start start 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 10 monclient: init 2024-09-16T04:10:11.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.487+0000 7f31c3913640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc120a10 con 0x7f31bc13d620 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc120cf0 con 0x7f31bc136b60 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc136620 con 0x7f31bc139b10 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient(hunting): authenticate will time out at 986.993048s 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c16af640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f31bc136b60 0x7f31bc09d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42484/0 (socket says 172.21.15.171:42484) 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c16af640 1 -- 172.21.15.171:0/4100735819 learned_addr learned my addr 172.21.15.171:0/4100735819 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c1eb0640 1 -- 172.21.15.171:0/4100735819 >> v1:172.21.15.148:6789/0 conn(0x7f31bc13d620 legacy=0x7f31bc1d37f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:11.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2679337267 0 0) 0x7f31bc120cf0 con 0x7f31bc136b60 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): my global_id is 5611 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31a0003520 con 0x7f31bc136b60 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 995490053 0 0) 0x7f31bc136620 con 0x7f31bc139b10 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): my global_id is 5642 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31bc120cf0 con 0x7f31bc139b10 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 502526727 0 0) 0x7f31a0003520 con 0x7f31bc136b60 2024-09-16T04:10:11.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f31bc136620 con 0x7f31bc136b60 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31ac001930 con 0x7f31bc136b60 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.504 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 259466748 0 0) 0x7f31bc136620 con 0x7f31bc136b60 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 >> v1:172.21.15.148:6790/0 conn(0x7f31bc139b10 legacy=0x7f31bc09dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 >> v1:172.21.15.148:6789/0 conn(0x7f31bc13d620 legacy=0x7f31bc1d37f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 monclient: found mon.b 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc0a0240 con 0x7f31bc136b60 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:11.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 5 monclient: authenticate success, global_id 5611 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient: _renew_subs 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f31bc11fe10 con 0x7f31bc136b60 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f31ac001d00 con 0x7f31bc136b60 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f31ac003a60 con 0x7f31bc136b60 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b1ffb640 10 monclient: dump: 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:11.506 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient: _renew_subs 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31b37fe640 4 set_mon_vals no callback set 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.488+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f31bc0a8bc0 con 0x7f31bc136b60 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f31ac003c00 con 0x7f31bc136b60 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(133..133 src has 1..133) ==== 6449+0+0 (unknown 1143600978 0 0) 0x7f31ac0478a0 con 0x7f31bc136b60 2024-09-16T04:10:11.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31c3913640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:11.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31c3913640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:11.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31c3913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31bc1205d0 con 0x7f31bc136b60 2024-09-16T04:10:11.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.489+0000 7f31c0eae640 10 monclient: get_auth_request con 0x7f31a0038f60 auth_method 0 2024-09-16T04:10:11.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.494+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f31ac015d50 con 0x7f31bc136b60 2024-09-16T04:10:11.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.494+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:11.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.494+0000 7f31b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:11.666 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.664+0000 7f31c3913640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:11.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.664+0000 7f31c3913640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:11.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.664+0000 7f31c3913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:11.667 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:11.664+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7f31bc06bcf0 con 0x7f31bc136b60 2024-09-16T04:10:12.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.198+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v134) ==== 82+0+0 (unknown 3477062080 0 0) 0x7f31ac015650 con 0x7f31bc136b60 2024-09-16T04:10:12.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.198+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:12.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.198+0000 7f31b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:12.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31c3913640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:12.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31c3913640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:12.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31c3913640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7f31bc0a4dc0 con 0x7f31bc136b60 2024-09-16T04:10:12.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31b1ffb640 1 -- 172.21.15.171:0/4100735819 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v134) ==== 115+0+0 (unknown 1783117428 0 0) 0x7f31bc0a4dc0 con 0x7f31bc136b60 2024-09-16T04:10:12.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31b1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:12.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.305+0000 7f31b1ffb640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T04:10:12.309 INFO:tasks.workunit.client.0.smithi171.stderr:compat weight-set already created 2024-09-16T04:10:12.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 >> v1:172.21.15.171:6812/7327273 conn(0x7f31a0038f60 legacy=0x7f31a003b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 10 monclient: shutdown 2024-09-16T04:10:12.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:12.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 >> v1:172.21.15.171:6789/0 conn(0x7f31bc136b60 legacy=0x7f31bc09d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 shutdown_connections 2024-09-16T04:10:12.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 >> 172.21.15.171:0/4100735819 conn(0x7f31bc0a98b0 msgr2=0x7f31bc139170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:12.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 shutdown_connections 2024-09-16T04:10:12.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.307+0000 7f31c3913640 1 -- 172.21.15.171:0/4100735819 wait complete. 2024-09-16T04:10:12.319 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set ls 2024-09-16T04:10:12.319 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '(compat)' 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.397+0000 7fa4ee752640 10 monclient: get_monmap_and_config 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.397+0000 7fa4ee752640 10 monclient: build_initial_monmap 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.397+0000 7fa4ee752640 10 monclient: monmap: 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:12.398055+0000 2024-09-16T04:10:12.399 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:12.398055+0000 2024-09-16T04:10:12.400 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:12.400 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.400 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:12.400 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:12.400 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:12.400 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 1 Processor -- start 2024-09-16T04:10:12.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 1 -- start start 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient: init 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e8136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e8139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e80a1340 con 0x7fa4e813d620 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e8137650 con 0x7fa4e8136b60 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e80a0aa0 con 0x7fa4e8139b10 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4ee752640 10 monclient(hunting): authenticate will time out at 987.903649s 2024-09-16T04:10:12.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa4e8136b60 0x7fa4e8136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42490/0 (socket says 172.21.15.171:42490) 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e7fff640 1 -- 172.21.15.171:0/3787724336 learned_addr learned my addr 172.21.15.171:0/3787724336 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4eccef640 1 -- 172.21.15.171:0/3787724336 >> v1:172.21.15.148:6789/0 conn(0x7fa4e813d620 legacy=0x7fa4e813fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278903199 0 0) 0x7fa4e8137650 con 0x7fa4e8136b60 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e6ffd640 10 monclient(hunting): my global_id is 5614 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.398+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4d4003520 con 0x7fa4e8136b60 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2434961925 0 0) 0x7fa4d4003520 con 0x7fa4e8136b60 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 >> v1:172.21.15.148:6790/0 conn(0x7fa4e8139b10 legacy=0x7fa4e813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 >> v1:172.21.15.148:6789/0 conn(0x7fa4e813d620 legacy=0x7fa4e813fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:12.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 monclient: found mon.noname-c 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e80a0550 con 0x7fa4e8136b60 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4d8001610 con 0x7fa4e8136b60 2024-09-16T04:10:12.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 5 monclient: authenticate success, global_id 5614 2024-09-16T04:10:12.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:12.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:12.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:12.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: dump: 2024-09-16T04:10:12.413 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.414 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.415 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:12.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4d8001970 con 0x7fa4e8136b60 2024-09-16T04:10:12.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:12.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 1 -- 172.21.15.171:0/3787724336 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4d80048a0 con 0x7fa4e8136b60 2024-09-16T04:10:12.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:12.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 4 set_mon_vals no callback set 2024-09-16T04:10:12.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:12.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:12.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4e6ffd640 10 monclient: dump: 2024-09-16T04:10:12.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.418 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.418 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.418 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.418 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.418 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.418 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.419 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.419 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.419 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 10 monclient: shutdown 2024-09-16T04:10:12.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:12.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 1 -- 172.21.15.171:0/3787724336 >> v1:172.21.15.171:6789/0 conn(0x7fa4e8136b60 legacy=0x7fa4e8136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 1 -- 172.21.15.171:0/3787724336 shutdown_connections 2024-09-16T04:10:12.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 1 -- 172.21.15.171:0/3787724336 >> 172.21.15.171:0/3787724336 conn(0x7fa4e80a98b0 msgr2=0x7fa4e80a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:12.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 1 -- 172.21.15.171:0/3787724336 shutdown_connections 2024-09-16T04:10:12.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.399+0000 7fa4ee752640 1 -- 172.21.15.171:0/3787724336 wait complete. 2024-09-16T04:10:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.400+0000 7fa4ee752640 10 monclient: build_initial_monmap 2024-09-16T04:10:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.400+0000 7fa4ee752640 10 monclient: monmap: 2024-09-16T04:10:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:12.423 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.400+0000 7fa4ee752640 1 Processor -- start 2024-09-16T04:10:12.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.400+0000 7fa4ee752640 1 -- start start 2024-09-16T04:10:12.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.400+0000 7fa4ee752640 10 monclient: init 2024-09-16T04:10:12.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:12.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:12.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e8136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:12.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e8139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:12.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:12.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e8120a30 con 0x7fa4e8139b10 2024-09-16T04:10:12.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e8120d10 con 0x7fa4e813d620 2024-09-16T04:10:12.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e80a8b40 con 0x7fa4e8136b60 2024-09-16T04:10:12.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:12.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4ee752640 10 monclient(hunting): authenticate will time out at 987.906356s 2024-09-16T04:10:12.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e77fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fa4e8139b10 legacy=0x7fa4e809db90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:12.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4eccef640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa4e813d620 0x7fa4e81d37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42496/0 (socket says 172.21.15.171:42496) 2024-09-16T04:10:12.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4eccef640 1 -- 172.21.15.171:0/428558586 learned_addr learned my addr 172.21.15.171:0/428558586 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473823218 0 0) 0x7fa4e8120d10 con 0x7fa4e813d620 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): my global_id is 5617 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c8003550 con 0x7fa4e813d620 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590910168 0 0) 0x7fa4e80a8b40 con 0x7fa4e8136b60 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): my global_id is 5648 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4e8120d10 con 0x7fa4e8136b60 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3886179753 0 0) 0x7fa4c8003550 con 0x7fa4e813d620 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4e80a8b40 con 0x7fa4e813d620 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4e0004440 con 0x7fa4e813d620 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): dump: 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.428 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 674587106 0 0) 0x7fa4e8120d10 con 0x7fa4e8136b60 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4c8003550 con 0x7fa4e8136b60 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3770951747 0 0) 0x7fa4e80a8b40 con 0x7fa4e813d620 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 >> v1:172.21.15.148:6790/0 conn(0x7fa4e8136b60 legacy=0x7fa4e809d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 >> v1:172.21.15.148:6789/0 conn(0x7fa4e8139b10 legacy=0x7fa4e809db90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:10:12.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 monclient: found mon.b 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e80a0240 con 0x7fa4e813d620 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.401+0000 7fa4e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 5 monclient: authenticate success, global_id 5617 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 monclient: _renew_subs 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e8120750 con 0x7fa4e813d620 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 monclient: _renew_subs 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa4e809e480 con 0x7fa4e813d620 2024-09-16T04:10:12.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4e00046f0 con 0x7fa4e813d620 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4e0004ac0 con 0x7fa4e813d620 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 10 monclient: dump: 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.431 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e67fc640 4 set_mon_vals no callback set 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa4e0004ce0 con 0x7fa4e813d620 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.402+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4cc005080 con 0x7fa4e813d620 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.403+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(134..134 src has 1..134) ==== 6781+0+0 (unknown 3468784074 0 0) 0x7fa4e0047e50 con 0x7fa4e813d620 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.403+0000 7fa4e7fff640 10 monclient: get_auth_request con 0x7fa4c8038ea0 auth_method 0 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.407+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa4e0016080 con 0x7fa4e813d620 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.407+0000 7fa4e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:12.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.407+0000 7fa4e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:12.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4ee752640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:12.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4ee752640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:12.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4ee752640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fa4cc005370 con 0x7fa4e813d620 2024-09-16T04:10:12.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v134) ==== 71+0+9 (unknown 1605358806 0 1822970295) 0x7fa4e0015980 con 0x7fa4e813d620 2024-09-16T04:10:12.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:12.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.577+0000 7fa4e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:12.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.680+0000 7fa4ee752640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:12.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.680+0000 7fa4ee752640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:12.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.680+0000 7fa4ee752640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.680+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set ls"} v 0) -- 0x7fa4cc000bb0 con 0x7fa4e813d620 2024-09-16T04:10:12.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.681+0000 7fa4e4ff9640 1 -- 172.21.15.171:0/428558586 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v134) ==== 71+0+9 (unknown 1605358806 0 1822970295) 0x7fa4e0019cd0 con 0x7fa4e813d620 2024-09-16T04:10:12.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.681+0000 7fa4e4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-16T04:10:12.683 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.681+0000 7fa4e4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.682+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 >> v1:172.21.15.171:6812/7327273 conn(0x7fa4c8038ea0 legacy=0x7fa4c803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.682+0000 7fa4ee752640 10 monclient: shutdown 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.682+0000 7fa4ee752640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.682+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 >> v1:172.21.15.171:6789/0 conn(0x7fa4e813d620 legacy=0x7fa4e81d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.683+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 shutdown_connections 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.683+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 >> 172.21.15.171:0/428558586 conn(0x7fa4e80a98b0 msgr2=0x7fa4e8139250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.683+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 shutdown_connections 2024-09-16T04:10:12.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.683+0000 7fa4ee752640 1 -- 172.21.15.171:0/428558586 wait complete. 2024-09-16T04:10:12.693 INFO:tasks.workunit.client.0.smithi171.stdout:(compat) 2024-09-16T04:10:12.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.770+0000 7fc1820fa640 10 monclient: get_monmap_and_config 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.770+0000 7fc1820fa640 10 monclient: build_initial_monmap 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.770+0000 7fc1820fa640 10 monclient: monmap: 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:12.771008+0000 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:12.771008+0000 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:12.772 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.773 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:12.773 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:12.773 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:12.773 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.770+0000 7fc1820fa640 1 Processor -- start 2024-09-16T04:10:12.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 1 -- start start 2024-09-16T04:10:12.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient: init 2024-09-16T04:10:12.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:12.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc17c09f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc17c0a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c09e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c0a1260 con 0x7fc17c0a0130 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c09d0c0 con 0x7fc17c09f640 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c09d3a0 con 0x7fc17c09e1d0 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc1820fa640 10 monclient(hunting): authenticate will time out at 988.276657s 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17affd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc17c0a0130 legacy=0x7fc17c138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc17c09f640 0x7fc17c09fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42502/0 (socket says 172.21.15.171:42502) 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17b7fe640 1 -- 172.21.15.171:0/354243536 learned_addr learned my addr 172.21.15.171:0/354243536 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:12.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1798669890 0 0) 0x7fc17c09d0c0 con 0x7fc17c09f640 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.771+0000 7fc17a7fc640 10 monclient(hunting): my global_id is 5620 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc168004630 con 0x7fc17c09f640 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3648582456 0 0) 0x7fc168004630 con 0x7fc17c09f640 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 >> v1:172.21.15.148:6790/0 conn(0x7fc17c09e1d0 legacy=0x7fc17c13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 >> v1:172.21.15.148:6789/0 conn(0x7fc17c0a0130 legacy=0x7fc17c138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c09d680 con 0x7fc17c09f640 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc16c001610 con 0x7fc17c09f640 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 5 monclient: authenticate success, global_id 5620 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: dump: 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.784 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc16c001880 con 0x7fc17c09f640 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:12.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 4 set_mon_vals no callback set 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 1 -- 172.21.15.171:0/354243536 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc16c001c50 con 0x7fc17c09f640 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc17a7fc640 10 monclient: dump: 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.786 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 10 monclient: shutdown 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 1 -- 172.21.15.171:0/354243536 >> v1:172.21.15.171:6789/0 conn(0x7fc17c09f640 legacy=0x7fc17c09fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 1 -- 172.21.15.171:0/354243536 shutdown_connections 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 1 -- 172.21.15.171:0/354243536 >> 172.21.15.171:0/354243536 conn(0x7fc17c0a52a0 msgr2=0x7fc17c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.772+0000 7fc1820fa640 1 -- 172.21.15.171:0/354243536 shutdown_connections 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.773+0000 7fc1820fa640 1 -- 172.21.15.171:0/354243536 wait complete. 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.773+0000 7fc1820fa640 10 monclient: build_initial_monmap 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.773+0000 7fc1820fa640 10 monclient: monmap: 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:12.787 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.773+0000 7fc1820fa640 1 Processor -- start 2024-09-16T04:10:12.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 1 -- start start 2024-09-16T04:10:12.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient: init 2024-09-16T04:10:12.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:12.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:12.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc17c09e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:12.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c09f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:12.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc17c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:12.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c1b6180 con 0x7fc17c09e1d0 2024-09-16T04:10:12.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c1a8ed0 con 0x7fc17c09f640 2024-09-16T04:10:12.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c1a91b0 con 0x7fc17c0a0130 2024-09-16T04:10:12.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:12.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc1820fa640 10 monclient(hunting): authenticate will time out at 988.279536s 2024-09-16T04:10:12.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc17affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc17c09f640 0x7fc17c1a17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42518/0 (socket says 172.21.15.171:42518) 2024-09-16T04:10:12.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc17affd640 1 -- 172.21.15.171:0/2860273564 learned_addr learned my addr 172.21.15.171:0/2860273564 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:12.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc17b7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fc17c09e1d0 legacy=0x7fc17c1a1090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:12.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151242966 0 0) 0x7fc17c1a8ed0 con 0x7fc17c09f640 2024-09-16T04:10:12.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:12.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): my global_id is 5623 2024-09-16T04:10:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc15c003520 con 0x7fc17c09f640 2024-09-16T04:10:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158898150 0 0) 0x7fc17c1a91b0 con 0x7fc17c0a0130 2024-09-16T04:10:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:12.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 10 monclient(hunting): my global_id is 5654 2024-09-16T04:10:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc17c1a8ed0 con 0x7fc17c0a0130 2024-09-16T04:10:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.774+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3030851264 0 0) 0x7fc15c003520 con 0x7fc17c09f640 2024-09-16T04:10:12.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc17c1a91b0 con 0x7fc17c09f640 2024-09-16T04:10:12.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc164004180 con 0x7fc17c09f640 2024-09-16T04:10:12.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:12.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:12.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient(hunting): dump: 2024-09-16T04:10:12.794 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.794 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.795 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.795 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.795 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.795 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.795 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.795 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.796 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.796 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:12.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1758080268 0 0) 0x7fc17c1a91b0 con 0x7fc17c09f640 2024-09-16T04:10:12.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 >> v1:172.21.15.148:6790/0 conn(0x7fc17c0a0130 legacy=0x7fc17c1d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:12.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 >> v1:172.21.15.148:6789/0 conn(0x7fc17c09e1d0 legacy=0x7fc17c1a1090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:12.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:12.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 monclient: found mon.b 2024-09-16T04:10:12.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:12.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c1a1eb0 con 0x7fc17c09f640 2024-09-16T04:10:12.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:12.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:12.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:12.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 5 monclient: authenticate success, global_id 5623 2024-09-16T04:10:12.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 10 monclient: _renew_subs 2024-09-16T04:10:12.798 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c0a3db0 con 0x7fc17c09f640 2024-09-16T04:10:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 10 monclient: _renew_subs 2024-09-16T04:10:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc17c1a8040 con 0x7fc17c09f640 2024-09-16T04:10:12.799 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1640044a0 con 0x7fc17c09f640 2024-09-16T04:10:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc164004870 con 0x7fc17c09f640 2024-09-16T04:10:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:12.800 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 10 monclient: dump: 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:12.801 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc179ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.775+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc164004a10 con 0x7fc17c09f640 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.776+0000 7fc1820fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.776+0000 7fc1820fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.776+0000 7fc1820fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.776+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc160005080 con 0x7fc17c09f640 2024-09-16T04:10:12.802 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.776+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(134..134 src has 1..134) ==== 6781+0+0 (unknown 3468784074 0 0) 0x7fc1640479c0 con 0x7fc17c09f640 2024-09-16T04:10:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.776+0000 7fc17b7fe640 10 monclient: get_auth_request con 0x7fc15c038be0 auth_method 0 2024-09-16T04:10:12.803 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.780+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc164015c90 con 0x7fc17c09f640 2024-09-16T04:10:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.780+0000 7fc173fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:12.804 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.780+0000 7fc173fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:12.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.952+0000 7fc1820fa640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:10:12.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.952+0000 7fc1820fa640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:10:12.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.952+0000 7fc1820fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:12.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:12.952+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fc160005370 con 0x7fc17c09f640 2024-09-16T04:10:13.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.205+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v135) ==== 78+0+0 (unknown 205680241 0 0) 0x7fc16401ad90 con 0x7fc17c09f640 2024-09-16T04:10:13.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.205+0000 7fc173fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:10:13.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.205+0000 7fc173fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:13.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.313+0000 7fc1820fa640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:10:13.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.313+0000 7fc1820fa640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:10:13.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.313+0000 7fc1820fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:13.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:13.313+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fc160002210 con 0x7fc17c09f640 2024-09-16T04:10:14.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.210+0000 7fc173fff640 1 -- 172.21.15.171:0/2860273564 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v136) ==== 78+0+0 (unknown 3376042673 0 0) 0x7fc16401e140 con 0x7fc17c09f640 2024-09-16T04:10:14.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.210+0000 7fc173fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:10:14.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.210+0000 7fc173fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 >> v1:172.21.15.171:6812/7327273 conn(0x7fc15c038be0 legacy=0x7fc15c03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 10 monclient: shutdown 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 >> v1:172.21.15.171:6789/0 conn(0x7fc17c09f640 legacy=0x7fc17c1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 shutdown_connections 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 >> 172.21.15.171:0/2860273564 conn(0x7fc17c0a52a0 msgr2=0x7fc17c0a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 shutdown_connections 2024-09-16T04:10:14.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.214+0000 7fc1820fa640 1 -- 172.21.15.171:0/2860273564 wait complete. 2024-09-16T04:10:14.228 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create cool 2 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.310+0000 7fcbae8eb640 10 monclient: get_monmap_and_config 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.310+0000 7fcbae8eb640 10 monclient: build_initial_monmap 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.310+0000 7fcbae8eb640 10 monclient: monmap: 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:14.311126+0000 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:14.311126+0000 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:14.312 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:14.313 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:14.313 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:14.313 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:14.313 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:14.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 1 Processor -- start 2024-09-16T04:10:14.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 1 -- start start 2024-09-16T04:10:14.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient: init 2024-09-16T04:10:14.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:14.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient(hunting): picked mon.noname-a con 0x7fcba8136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient(hunting): picked mon.noname-b con 0x7fcba8139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient(hunting): picked mon.noname-c con 0x7fcba813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba80a1340 con 0x7fcba8136b60 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba8137650 con 0x7fcba813d620 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba80a0aa0 con 0x7fcba8139b10 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbae8eb640 10 monclient(hunting): authenticate will time out at 989.816675s 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbace88640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcba813d620 0x7fcba813fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42530/0 (socket says 172.21.15.171:42530) 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcbace88640 1 -- 172.21.15.171:0/243813009 learned_addr learned my addr 172.21.15.171:0/243813009 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcba7fff640 1 -- 172.21.15.171:0/243813009 >> v1:172.21.15.148:6789/0 conn(0x7fcba8136b60 legacy=0x7fcba8136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.311+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3606880658 0 0) 0x7fcba8137650 con 0x7fcba813d620 2024-09-16T04:10:14.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient(hunting): my global_id is 5626 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb90003520 con 0x7fcba813d620 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513168906 0 0) 0x7fcb90003520 con 0x7fcba813d620 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 >> v1:172.21.15.148:6790/0 conn(0x7fcba8139b10 legacy=0x7fcba813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 >> v1:172.21.15.148:6789/0 conn(0x7fcba8136b60 legacy=0x7fcba8136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 monclient: found mon.noname-c 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba80a0550 con 0x7fcba813d620 2024-09-16T04:10:14.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcba0001990 con 0x7fcba813d620 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 5 monclient: authenticate success, global_id 5626 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: dump: 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:14.322 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcba0001cf0 con 0x7fcba813d620 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 1 -- 172.21.15.171:0/243813009 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcba0004c20 con 0x7fcba813d620 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 4 set_mon_vals no callback set 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcba6ffd640 10 monclient: dump: 2024-09-16T04:10:14.323 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 10 monclient: shutdown 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 1 -- 172.21.15.171:0/243813009 >> v1:172.21.15.171:6789/0 conn(0x7fcba813d620 legacy=0x7fcba813fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 1 -- 172.21.15.171:0/243813009 shutdown_connections 2024-09-16T04:10:14.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 1 -- 172.21.15.171:0/243813009 >> 172.21.15.171:0/243813009 conn(0x7fcba80a98b0 msgr2=0x7fcba80a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 1 -- 172.21.15.171:0/243813009 shutdown_connections 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.312+0000 7fcbae8eb640 1 -- 172.21.15.171:0/243813009 wait complete. 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.313+0000 7fcbae8eb640 10 monclient: build_initial_monmap 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.313+0000 7fcbae8eb640 10 monclient: monmap: 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:14.325 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.313+0000 7fcbae8eb640 1 Processor -- start 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.313+0000 7fcbae8eb640 1 -- start start 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.313+0000 7fcbae8eb640 10 monclient: init 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient(hunting): picked mon.noname-a con 0x7fcba8136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient(hunting): picked mon.noname-c con 0x7fcba8139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient(hunting): picked mon.noname-b con 0x7fcba813d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba813ad10 con 0x7fcba8136b60 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba80a2b30 con 0x7fcba813d620 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcba80a0240 con 0x7fcba8139b10 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbae8eb640 10 monclient(hunting): authenticate will time out at 989.819401s 2024-09-16T04:10:14.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba7fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fcba8136b60 legacy=0x7fcba8139ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbace88640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcba813d620 0x7fcba81d37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42538/0 (socket says 172.21.15.171:42538) 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcbace88640 1 -- 172.21.15.171:0/3846866983 learned_addr learned my addr 172.21.15.171:0/3846866983 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996902327 0 0) 0x7fcba80a2b30 con 0x7fcba813d620 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): my global_id is 5629 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb94003550 con 0x7fcba813d620 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 369123326 0 0) 0x7fcba80a0240 con 0x7fcba8139b10 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): my global_id is 5660 2024-09-16T04:10:14.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba80a2b30 con 0x7fcba8139b10 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3820372331 0 0) 0x7fcb94003550 con 0x7fcba813d620 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcba80a0240 con 0x7fcba813d620 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcba00017b0 con 0x7fcba813d620 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): dump: 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:14.328 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.314+0000 7fcba4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1806595045 0 0) 0x7fcba80a0240 con 0x7fcba813d620 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 >> v1:172.21.15.148:6790/0 conn(0x7fcba8139b10 legacy=0x7fcba813a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 >> v1:172.21.15.148:6789/0 conn(0x7fcba8136b60 legacy=0x7fcba8139ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 monclient: found mon.b 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba813b2d0 con 0x7fcba813d620 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:10:14.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 5 monclient: authenticate success, global_id 5629 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 10 monclient: _renew_subs 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcba809aea0 con 0x7fcba813d620 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 10 monclient: _renew_subs 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcba813c9d0 con 0x7fcba813d620 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcba0001cf0 con 0x7fcba813d620 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcba00051d0 con 0x7fcba813d620 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:14.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba67fc640 4 set_mon_vals no callback set 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 10 monclient: dump: 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.315+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcba0010600 con 0x7fcba813d620 2024-09-16T04:10:14.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.316+0000 7fcbae8eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.316+0000 7fcbae8eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.316+0000 7fcbae8eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.316+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb88005080 con 0x7fcba813d620 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.316+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(136..136 src has 1..136) ==== 6449+0+0 (unknown 1537231772 0 0) 0x7fcba0047bd0 con 0x7fcba813d620 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.316+0000 7fcba7fff640 10 monclient: get_auth_request con 0x7fcb94038bc0 auth_method 0 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.320+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fcba00160e0 con 0x7fcba813d620 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.320+0000 7fcba4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:14.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.320+0000 7fcba4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:14.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.485+0000 7fcbae8eb640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T04:10:14.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.485+0000 7fcbae8eb640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T04:10:14.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.485+0000 7fcbae8eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:14.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:14.485+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) -- 0x7fcb88005b80 con 0x7fcba813d620 2024-09-16T04:10:15.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.216+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v137) ==== 111+0+0 (unknown 2016388275 0 0) 0x7fcba001e590 con 0x7fcba813d620 2024-09-16T04:10:15.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.216+0000 7fcba4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T04:10:15.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.216+0000 7fcba4ff9640 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-16T04:10:15.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.319+0000 7fcbae8eb640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T04:10:15.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.319+0000 7fcbae8eb640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T04:10:15.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.319+0000 7fcbae8eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.319+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) -- 0x7fcb88001090 con 0x7fcba813d620 2024-09-16T04:10:15.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.320+0000 7fcba4ff9640 1 -- 172.21.15.171:0/3846866983 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v137) ==== 118+0+0 (unknown 3687984175 0 0) 0x7fcba0013db0 con 0x7fcba813d620 2024-09-16T04:10:15.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.320+0000 7fcba4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-16T04:10:15.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.320+0000 7fcba4ff9640 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-16T04:10:15.322 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'cool' already exists 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 >> v1:172.21.15.171:6812/7327273 conn(0x7fcb94038bc0 legacy=0x7fcb9403b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 10 monclient: shutdown 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 >> v1:172.21.15.171:6789/0 conn(0x7fcba813d620 legacy=0x7fcba81d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 shutdown_connections 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 >> 172.21.15.171:0/3846866983 conn(0x7fcba80a98b0 msgr2=0x7fcba8132ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 shutdown_connections 2024-09-16T04:10:15.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.322+0000 7fcbae8eb640 1 -- 172.21.15.171:0/3846866983 wait complete. 2024-09-16T04:10:15.333 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool create cold 2 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.411+0000 7f28a845c640 10 monclient: get_monmap_and_config 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.411+0000 7f28a845c640 10 monclient: build_initial_monmap 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.411+0000 7f28a845c640 10 monclient: monmap: 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:15.412643+0000 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:15.412643+0000 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:15.414 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:15.415 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.412+0000 7f28a845c640 1 Processor -- start 2024-09-16T04:10:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.412+0000 7f28a845c640 1 -- start start 2024-09-16T04:10:15.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.412+0000 7f28a845c640 10 monclient: init 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient(hunting): picked mon.noname-a con 0x7f28a009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient(hunting): picked mon.noname-b con 0x7f28a00a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient(hunting): picked mon.noname-c con 0x7f28a009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a00a1260 con 0x7f28a009f640 2024-09-16T04:10:15.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a009d0c0 con 0x7f28a009e1d0 2024-09-16T04:10:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a009d3a0 con 0x7f28a00a0130 2024-09-16T04:10:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient(hunting): authenticate will time out at 990.918245s 2024-09-16T04:10:15.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a61f8640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f28a009f640 legacy=0x7f28a009fa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a69f9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f28a009e1d0 0x7f28a013bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42550/0 (socket says 172.21.15.171:42550) 2024-09-16T04:10:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a69f9640 1 -- 172.21.15.171:0/2796349910 learned_addr learned my addr 172.21.15.171:0/2796349910 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631528754 0 0) 0x7f28a009d0c0 con 0x7f28a009e1d0 2024-09-16T04:10:15.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): _negotiate 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): my global_id is 5632 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f288c003520 con 0x7f28a009e1d0 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187177243 0 0) 0x7f28a009d3a0 con 0x7f28a00a0130 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): _negotiate 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:15.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient(hunting): my global_id is 5663 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28a009d0c0 con 0x7f28a00a0130 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1359902077 0 0) 0x7f288c003520 con 0x7f28a009e1d0 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 >> v1:172.21.15.148:6790/0 conn(0x7f28a00a0130 legacy=0x7f28a0138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 >> v1:172.21.15.148:6789/0 conn(0x7f28a009f640 legacy=0x7f28a009fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: _finish_hunting 0 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 monclient: found mon.noname-c 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a009d680 con 0x7f28a009e1d0 2024-09-16T04:10:15.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: _finish_auth 0 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: _check_auth_tickets 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2898001990 con 0x7f28a009e1d0 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 5 monclient: authenticate success, global_id 5632 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: dump: 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:15.430 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2898001c40 con 0x7f28a009e1d0 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 1 -- 172.21.15.171:0/2796349910 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2898002010 con 0x7f28a009e1d0 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a845c640 4 set_mon_vals no callback set 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.413+0000 7f28a51f6640 10 monclient: dump: 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:15.431 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 10 monclient: shutdown 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 1 -- 172.21.15.171:0/2796349910 >> v1:172.21.15.171:6789/0 conn(0x7f28a009e1d0 legacy=0x7f28a013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 1 -- 172.21.15.171:0/2796349910 shutdown_connections 2024-09-16T04:10:15.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 1 -- 172.21.15.171:0/2796349910 >> 172.21.15.171:0/2796349910 conn(0x7f28a00a52a0 msgr2=0x7f28a00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 1 -- 172.21.15.171:0/2796349910 shutdown_connections 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.414+0000 7f28a845c640 1 -- 172.21.15.171:0/2796349910 wait complete. 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient: build_initial_monmap 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient: monmap: 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:15.433 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 1 Processor -- start 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 1 -- start start 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient: init 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient(hunting): picked mon.noname-b con 0x7f28a009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient(hunting): picked mon.noname-a con 0x7f28a009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient(hunting): picked mon.noname-c con 0x7f28a00a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a01c5940 con 0x7f28a009f640 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a006ceb0 con 0x7f28a009e1d0 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a006fcc0 con 0x7f28a00a0130 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a845c640 10 monclient(hunting): authenticate will time out at 990.920954s 2024-09-16T04:10:15.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a61f8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f28a009e1d0 0x7f28a01c5230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42562/0 (socket says 172.21.15.171:42562) 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.415+0000 7f28a61f8640 1 -- 172.21.15.171:0/390778116 learned_addr learned my addr 172.21.15.171:0/390778116 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a59f7640 1 -- 172.21.15.171:0/390778116 >> v1:172.21.15.148:6789/0 conn(0x7f28a009f640 legacy=0x7f28a01d0560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771503020 0 0) 0x7f28a006ceb0 con 0x7f28a009e1d0 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): my global_id is 5635 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2884003520 con 0x7f28a009e1d0 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2235585885 0 0) 0x7f28a006fcc0 con 0x7f28a00a0130 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): my global_id is 5666 2024-09-16T04:10:15.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28a006ceb0 con 0x7f28a00a0130 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4089353381 0 0) 0x7f2884003520 con 0x7f28a009e1d0 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28a006fcc0 con 0x7f28a009e1d0 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f289c0041d0 con 0x7f28a009e1d0 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.436 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1077240093 0 0) 0x7f28a006ceb0 con 0x7f28a00a0130 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2884003520 con 0x7f28a00a0130 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2898001df0 con 0x7f28a00a0130 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.437 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1207747153 0 0) 0x7f28a006fcc0 con 0x7f28a009e1d0 2024-09-16T04:10:15.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 >> v1:172.21.15.148:6790/0 conn(0x7f28a00a0130 legacy=0x7f28a01d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 >> v1:172.21.15.148:6789/0 conn(0x7f28a009f640 legacy=0x7f28a01d0560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 monclient: found mon.b 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a01acb90 con 0x7f28a009e1d0 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 5 monclient: authenticate success, global_id 5635 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 10 monclient: _renew_subs 2024-09-16T04:10:15.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28a01c8b30 con 0x7f28a009e1d0 2024-09-16T04:10:15.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 10 monclient: _renew_subs 2024-09-16T04:10:15.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f28a01ab640 con 0x7f28a009e1d0 2024-09-16T04:10:15.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f289c004430 con 0x7f28a009e1d0 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f289c004780 con 0x7f28a009e1d0 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f28a49f5640 4 set_mon_vals no callback set 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.416+0000 7f2896ffd640 10 monclient: dump: 2024-09-16T04:10:15.441 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:15.442 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:15.442 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.442 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:15.442 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:15.442 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f289c004920 con 0x7f28a009e1d0 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f28a845c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f28a845c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f28a845c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2888005080 con 0x7f28a009e1d0 2024-09-16T04:10:15.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(137..137 src has 1..137) ==== 6792+0+0 (unknown 156155081 0 0) 0x7f289c047940 con 0x7f28a009e1d0 2024-09-16T04:10:15.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.417+0000 7f28a59f7640 10 monclient: get_auth_request con 0x7f2884038f60 auth_method 0 2024-09-16T04:10:15.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.422+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f289c015b60 con 0x7f28a009e1d0 2024-09-16T04:10:15.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.422+0000 7f2896ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:15.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.422+0000 7f2896ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:15.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.587+0000 7f28a845c640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T04:10:15.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.587+0000 7f28a845c640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T04:10:15.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.587+0000 7f28a845c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:15.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:15.587+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) -- 0x7f2888005b80 con 0x7f28a009e1d0 2024-09-16T04:10:16.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.220+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v138) ==== 111+0+0 (unknown 2029814857 0 0) 0x7f289c0197b0 con 0x7f28a009e1d0 2024-09-16T04:10:16.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.220+0000 7f2896ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T04:10:16.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.220+0000 7f2896ffd640 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.324+0000 7f28a845c640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.324+0000 7f28a845c640 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.324+0000 7f28a845c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.324+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) -- 0x7f2888005760 con 0x7f28a009e1d0 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.325+0000 7f2896ffd640 1 -- 172.21.15.171:0/390778116 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v138) ==== 118+0+0 (unknown 1919541335 0 0) 0x7f289c015460 con 0x7f28a009e1d0 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.325+0000 7f2896ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.325+0000 7f2896ffd640 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-16T04:10:16.327 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'cold' already exists 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 >> v1:172.21.15.171:6812/7327273 conn(0x7f2884038f60 legacy=0x7f288403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 10 monclient: shutdown 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 >> v1:172.21.15.171:6789/0 conn(0x7f28a009e1d0 legacy=0x7f28a01c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 shutdown_connections 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 >> 172.21.15.171:0/390778116 conn(0x7f28a00a52a0 msgr2=0x7f28a00709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 shutdown_connections 2024-09-16T04:10:16.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.327+0000 7f28a845c640 1 -- 172.21.15.171:0/390778116 wait complete. 2024-09-16T04:10:16.338 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool set cold size 2 2024-09-16T04:10:16.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.416+0000 7f7e2c50a640 10 monclient: get_monmap_and_config 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.416+0000 7f7e2c50a640 10 monclient: build_initial_monmap 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.416+0000 7f7e2c50a640 10 monclient: monmap: 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:16.417678+0000 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:16.417678+0000 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:16.419 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:16.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.417+0000 7f7e2c50a640 1 Processor -- start 2024-09-16T04:10:16.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.417+0000 7f7e2c50a640 1 -- start start 2024-09-16T04:10:16.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.417+0000 7f7e2c50a640 10 monclient: init 2024-09-16T04:10:16.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:16.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:16.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient(hunting): picked mon.noname-c con 0x7f7e24133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient(hunting): picked mon.noname-a con 0x7f7e241341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2413a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2409da90 con 0x7f7e241341f0 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2409dd70 con 0x7f7e24133700 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2409e050 con 0x7f7e2413a350 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient(hunting): authenticate will time out at 991.923225s 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e29aa5640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7e241341f0 legacy=0x7f7e24138c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2a2a6640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7e24133700 0x7f7e24133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42586/0 (socket says 172.21.15.171:42586) 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2a2a6640 1 -- 172.21.15.171:0/2246790261 learned_addr learned my addr 172.21.15.171:0/2246790261 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569800352 0 0) 0x7f7e2409dd70 con 0x7f7e24133700 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): _negotiate 2024-09-16T04:10:16.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): my global_id is 5638 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e10003520 con 0x7f7e24133700 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3588024814 0 0) 0x7f7e2409e050 con 0x7f7e2413a350 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): _negotiate 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient(hunting): my global_id is 5669 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e2409dd70 con 0x7f7e2413a350 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1583343271 0 0) 0x7f7e10003520 con 0x7f7e24133700 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 >> v1:172.21.15.148:6790/0 conn(0x7f7e2413a350 legacy=0x7f7e2413c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 >> v1:172.21.15.148:6789/0 conn(0x7f7e241341f0 legacy=0x7f7e24138c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:16.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: _finish_hunting 0 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 monclient: found mon.noname-c 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2409d4f0 con 0x7f7e24133700 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: _finish_auth 0 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: _check_auth_tickets 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 5 monclient: authenticate success, global_id 5638 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e20001610 con 0x7f7e24133700 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: dump: 2024-09-16T04:10:16.429 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e20001930 con 0x7f7e24133700 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:16.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 1 -- 172.21.15.171:0/2246790261 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e20001d00 con 0x7f7e24133700 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e2c50a640 4 set_mon_vals no callback set 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.418+0000 7f7e292a4640 10 monclient: dump: 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:16.431 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 10 monclient: shutdown 2024-09-16T04:10:16.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 1 -- 172.21.15.171:0/2246790261 >> v1:172.21.15.171:6789/0 conn(0x7f7e24133700 legacy=0x7f7e24133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 1 -- 172.21.15.171:0/2246790261 shutdown_connections 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 1 -- 172.21.15.171:0/2246790261 >> 172.21.15.171:0/2246790261 conn(0x7f7e240a5450 msgr2=0x7f7e240a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 1 -- 172.21.15.171:0/2246790261 shutdown_connections 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.419+0000 7f7e2c50a640 1 -- 172.21.15.171:0/2246790261 wait complete. 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient: build_initial_monmap 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient: monmap: 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:16.433 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 1 Processor -- start 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 1 -- start start 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient: init 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient(hunting): picked mon.noname-c con 0x7f7e24133700 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient(hunting): picked mon.noname-a con 0x7f7e241341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2413a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e240a48a0 con 0x7f7e241341f0 2024-09-16T04:10:16.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2409c0b0 con 0x7f7e2413a350 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2406f870 con 0x7f7e24133700 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e2c50a640 10 monclient(hunting): authenticate will time out at 991.925929s 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.420+0000 7f7e29aa5640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f7e241341f0 legacy=0x7f7e241d0820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2aaa7640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7e2413a350 0x7f7e241d3f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42588/0 (socket says 172.21.15.171:42588) 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2aaa7640 1 -- 172.21.15.171:0/3342251537 learned_addr learned my addr 172.21.15.171:0/3342251537 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555092400 0 0) 0x7f7e2409c0b0 con 0x7f7e2413a350 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): my global_id is 5641 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e08003520 con 0x7f7e2413a350 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1412948345 0 0) 0x7f7e2406f870 con 0x7f7e24133700 2024-09-16T04:10:16.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): my global_id is 5672 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e2409c0b0 con 0x7f7e24133700 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 318656005 0 0) 0x7f7e08003520 con 0x7f7e2413a350 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e2406f870 con 0x7f7e2413a350 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e1c001c50 con 0x7f7e2413a350 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): dump: 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.436 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3064069696 0 0) 0x7f7e2406f870 con 0x7f7e2413a350 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 >> v1:172.21.15.148:6790/0 conn(0x7f7e24133700 legacy=0x7f7e24132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 >> v1:172.21.15.148:6789/0 conn(0x7f7e241341f0 legacy=0x7f7e241d0820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 monclient: found mon.b 2024-09-16T04:10:16.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e24120830 con 0x7f7e2413a350 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 5 monclient: authenticate success, global_id 5641 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 10 monclient: _renew_subs 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e2411f940 con 0x7f7e2413a350 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 10 monclient: _renew_subs 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e24132ff0 con 0x7f7e2413a350 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e1c002020 con 0x7f7e2413a350 2024-09-16T04:10:16.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e1c003de0 con 0x7f7e2413a350 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e1affd640 10 monclient: dump: 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:16.439 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.421+0000 7f7e28aa3640 4 set_mon_vals no callback set 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7e1c004000 con 0x7f7e2413a350 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(138..138 src has 1..138) ==== 7135+0+0 (unknown 2001054715 0 0) 0x7f7e1c047f80 con 0x7f7e2413a350 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e2a2a6640 10 monclient: get_auth_request con 0x7f7e08038f60 auth_method 0 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e2c50a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e2c50a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e2c50a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.422+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0c005080 con 0x7f7e2413a350 2024-09-16T04:10:16.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.427+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7e1c016010 con 0x7f7e2413a350 2024-09-16T04:10:16.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.427+0000 7f7e1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:16.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.427+0000 7f7e1affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:16.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.592+0000 7f7e2c50a640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T04:10:16.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.592+0000 7f7e2c50a640 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T04:10:16.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.592+0000 7f7e2c50a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:16.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:16.592+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) -- 0x7f7e0c005b80 con 0x7f7e2413a350 2024-09-16T04:10:17.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.225+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v139) ==== 124+0+0 (unknown 2752074814 0 0) 0x7f7e1c019cc0 con 0x7f7e2413a350 2024-09-16T04:10:17.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.225+0000 7f7e1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T04:10:17.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.225+0000 7f7e1affd640 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-16T04:10:17.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.328+0000 7f7e2c50a640 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T04:10:17.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.328+0000 7f7e2c50a640 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T04:10:17.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.328+0000 7f7e2c50a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:17.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:17.328+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) -- 0x7f7e0c005760 con 0x7f7e2413a350 2024-09-16T04:10:18.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.230+0000 7f7e1affd640 1 -- 172.21.15.171:0/3342251537 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v140) ==== 124+0+0 (unknown 3984607602 0 0) 0x7f7e1c01e580 con 0x7f7e2413a350 2024-09-16T04:10:18.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.230+0000 7f7e1affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-16T04:10:18.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.230+0000 7f7e1affd640 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-16T04:10:18.232 INFO:tasks.workunit.client.0.smithi171.stderr:set pool 12 size to 2 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 >> v1:172.21.15.171:6812/7327273 conn(0x7f7e08038f60 legacy=0x7f7e0803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 10 monclient: shutdown 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 >> v1:172.21.15.171:6789/0 conn(0x7f7e2413a350 legacy=0x7f7e241d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 shutdown_connections 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 >> 172.21.15.171:0/3342251537 conn(0x7f7e240a5450 msgr2=0x7f7e2409f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 shutdown_connections 2024-09-16T04:10:18.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.233+0000 7f7e2c50a640 1 -- 172.21.15.171:0/3342251537 wait complete. 2024-09-16T04:10:18.246 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T04:10:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.328+0000 7fab61742640 10 monclient: get_monmap_and_config 2024-09-16T04:10:18.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.328+0000 7fab61742640 10 monclient: build_initial_monmap 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.328+0000 7fab61742640 10 monclient: monmap: 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:18.329557+0000 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:18.329557+0000 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:18.331 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 1 Processor -- start 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 1 -- start start 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 10 monclient: init 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 10 monclient(hunting): picked mon.noname-c con 0x7fab5c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 10 monclient(hunting): picked mon.noname-a con 0x7fab5c136530 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:18.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.329+0000 7fab61742640 10 monclient(hunting): picked mon.noname-b con 0x7fab5c13a040 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c0a1260 con 0x7fab5c136530 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c09d0c0 con 0x7fab5c09e1d0 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c09d3a0 con 0x7fab5c13a040 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 10 monclient(hunting): authenticate will time out at 993.835035s 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab5c09e1d0 0x7fab5c0a1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39366/0 (socket says 172.21.15.171:39366) 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5affd640 1 -- 172.21.15.171:0/820188984 learned_addr learned my addr 172.21.15.171:0/820188984 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab53fff640 1 -- 172.21.15.171:0/820188984 >> v1:172.21.15.148:6789/0 conn(0x7fab5c136530 legacy=0x7fab5c138920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283799642 0 0) 0x7fab5c09d0c0 con 0x7fab5c09e1d0 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): my global_id is 5644 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab48003520 con 0x7fab5c09e1d0 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417889854 0 0) 0x7fab5c09d3a0 con 0x7fab5c13a040 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient(hunting): my global_id is 5675 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab5c09d0c0 con 0x7fab5c13a040 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 734605300 0 0) 0x7fab48003520 con 0x7fab5c09e1d0 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 >> v1:172.21.15.148:6790/0 conn(0x7fab5c13a040 legacy=0x7fab5c13c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 >> v1:172.21.15.148:6789/0 conn(0x7fab5c136530 legacy=0x7fab5c138920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:18.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c09d680 con 0x7fab5c09e1d0 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab4c001610 con 0x7fab5c09e1d0 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 5 monclient: authenticate success, global_id 5644 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: dump: 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:18.342 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab4c0018c0 con 0x7fab5c09e1d0 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:18.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab61742640 4 set_mon_vals no callback set 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 1 -- 172.21.15.171:0/820188984 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab4c001c90 con 0x7fab5c09e1d0 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.330+0000 7fab5a7fc640 10 monclient: dump: 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.344 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 10 monclient: shutdown 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 1 -- 172.21.15.171:0/820188984 >> v1:172.21.15.171:6789/0 conn(0x7fab5c09e1d0 legacy=0x7fab5c0a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 1 -- 172.21.15.171:0/820188984 shutdown_connections 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 1 -- 172.21.15.171:0/820188984 >> 172.21.15.171:0/820188984 conn(0x7fab5c0a52a0 msgr2=0x7fab5c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 1 -- 172.21.15.171:0/820188984 shutdown_connections 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 1 -- 172.21.15.171:0/820188984 wait complete. 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 10 monclient: build_initial_monmap 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.331+0000 7fab61742640 10 monclient: monmap: 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:18.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 1 Processor -- start 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 1 -- start start 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient: init 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:18.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient(hunting): picked mon.noname-b con 0x7fab5c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient(hunting): picked mon.noname-a con 0x7fab5c136530 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient(hunting): picked mon.noname-c con 0x7fab5c13a040 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c124000 con 0x7fab5c136530 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c122870 con 0x7fab5c09e1d0 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c1cb540 con 0x7fab5c13a040 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab61742640 10 monclient(hunting): authenticate will time out at 993.837847s 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab53fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fab5c136530 legacy=0x7fab5c1d05f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab5affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab5c09e1d0 0x7fab5c135630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39368/0 (socket says 172.21.15.171:39368) 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.332+0000 7fab5affd640 1 -- 172.21.15.171:0/2538448336 learned_addr learned my addr 172.21.15.171:0/2538448336 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3984279584 0 0) 0x7fab5c122870 con 0x7fab5c09e1d0 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:18.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): my global_id is 5647 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab3c003520 con 0x7fab5c09e1d0 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174502111 0 0) 0x7fab5c1cb540 con 0x7fab5c13a040 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): my global_id is 5678 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab5c122870 con 0x7fab5c13a040 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1107795241 0 0) 0x7fab3c003520 con 0x7fab5c09e1d0 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab5c1cb540 con 0x7fab5c09e1d0 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab4c001c10 con 0x7fab5c09e1d0 2024-09-16T04:10:18.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): dump: 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:18.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 861678594 0 0) 0x7fab5c122870 con 0x7fab5c13a040 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab3c003520 con 0x7fab5c13a040 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab54001c30 con 0x7fab5c13a040 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient(hunting): dump: 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:18.350 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2567677424 0 0) 0x7fab5c1cb540 con 0x7fab5c09e1d0 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 >> v1:172.21.15.148:6790/0 conn(0x7fab5c13a040 legacy=0x7fab5c1d3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 >> v1:172.21.15.148:6789/0 conn(0x7fab5c136530 legacy=0x7fab5c1d05f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 monclient: found mon.b 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c135d40 con 0x7fab5c09e1d0 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:18.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 5 monclient: authenticate success, global_id 5647 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 10 monclient: _renew_subs 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c1cbdf0 con 0x7fab5c09e1d0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 10 monclient: _renew_subs 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab5c1d5440 con 0x7fab5c09e1d0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab4c004c40 con 0x7fab5c09e1d0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab4c005050 con 0x7fab5c09e1d0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:18.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab537fe640 10 monclient: dump: 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.333+0000 7fab59ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fab4c0052f0 con 0x7fab5c09e1d0 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab61742640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:18.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab61742640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab61742640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(140..140 src has 1..140) ==== 7135+0+0 (unknown 4184553254 0 0) 0x7fab4c047bc0 con 0x7fab5c09e1d0 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c09c840 con 0x7fab5c09e1d0 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.334+0000 7fab53fff640 10 monclient: get_auth_request con 0x7fab3c038ea0 auth_method 0 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.339+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fab5c09c840 con 0x7fab5c09e1d0 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.339+0000 7fab537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:18.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.339+0000 7fab537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:18.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.511+0000 7fab61742640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:18.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.511+0000 7fab61742640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:18.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.511+0000 7fab61742640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:18.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:18.511+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fab5c135ff0 con 0x7fab5c09e1d0 2024-09-16T04:10:19.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.237+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v141) ==== 82+0+0 (unknown 1165000671 0 0) 0x7fab4c015e90 con 0x7fab5c09e1d0 2024-09-16T04:10:19.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.237+0000 7fab537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:19.239 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.237+0000 7fab537fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:19.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab61742640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:19.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab61742640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:19.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab61742640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fab5c06ceb0 con 0x7fab5c09e1d0 2024-09-16T04:10:19.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab537fe640 1 -- 172.21.15.171:0/2538448336 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v141) ==== 115+0+0 (unknown 3192864329 0 0) 0x7fab4c015790 con 0x7fab5c09e1d0 2024-09-16T04:10:19.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab537fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:10:19.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.345+0000 7fab537fe640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T04:10:19.349 INFO:tasks.workunit.client.0.smithi171.stderr:compat weight-set already created 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 >> v1:172.21.15.171:6812/7327273 conn(0x7fab3c038ea0 legacy=0x7fab3c03b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 10 monclient: shutdown 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 >> v1:172.21.15.171:6789/0 conn(0x7fab5c09e1d0 legacy=0x7fab5c135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 shutdown_connections 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 >> 172.21.15.171:0/2538448336 conn(0x7fab5c0a52a0 msgr2=0x7fab5c072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 shutdown_connections 2024-09-16T04:10:19.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.348+0000 7fab61742640 1 -- 172.21.15.171:0/2538448336 wait complete. 2024-09-16T04:10:19.360 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create cool flat 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.438+0000 7fc328e53640 10 monclient: get_monmap_and_config 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.438+0000 7fc328e53640 10 monclient: build_initial_monmap 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.438+0000 7fc328e53640 10 monclient: monmap: 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:19.440091+0000 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:19.440091+0000 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:19.441 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:19.442 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:19.442 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:19.442 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:19.442 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.438+0000 7fc328e53640 1 Processor -- start 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.438+0000 7fc328e53640 1 -- start start 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.438+0000 7fc328e53640 10 monclient: init 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient(hunting): picked mon.noname-c con 0x7fc32409f7c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient(hunting): picked mon.noname-b con 0x7fc32409fba0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:19.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient(hunting): picked mon.noname-a con 0x7fc3241395d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3240a1490 con 0x7fc3241395d0 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32409d1b0 con 0x7fc32409f7c0 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32409d490 con 0x7fc32409fba0 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc328e53640 10 monclient(hunting): authenticate will time out at 994.945598s 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc32409f7c0 0x7fc32409de70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39384/0 (socket says 172.21.15.171:39384) 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32259c640 1 -- 172.21.15.171:0/3040130428 learned_addr learned my addr 172.21.15.171:0/3040130428 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc322d9d640 1 -- 172.21.15.171:0/3040130428 >> v1:172.21.15.148:6789/0 conn(0x7fc3241395d0 legacy=0x7fc32413b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1671037677 0 0) 0x7fc32409d1b0 con 0x7fc32409f7c0 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32159a640 10 monclient(hunting): _negotiate 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32159a640 10 monclient(hunting): my global_id is 5650 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.439+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc30c004630 con 0x7fc32409f7c0 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 289909264 0 0) 0x7fc30c004630 con 0x7fc32409f7c0 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 >> v1:172.21.15.148:6790/0 conn(0x7fc32409fba0 legacy=0x7fc3240a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 >> v1:172.21.15.148:6789/0 conn(0x7fc3241395d0 legacy=0x7fc32413b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: _finish_hunting 0 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 monclient: found mon.noname-c 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32409bca0 con 0x7fc32409f7c0 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: _finish_auth 0 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:19.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc314001610 con 0x7fc32409f7c0 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 5 monclient: authenticate success, global_id 5650 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: dump: 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:19.451 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc314001970 con 0x7fc32409f7c0 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 1 -- 172.21.15.171:0/3040130428 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3140048a0 con 0x7fc32409f7c0 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 4 set_mon_vals no callback set 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc32159a640 10 monclient: dump: 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:19.452 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 10 monclient: shutdown 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 1 -- 172.21.15.171:0/3040130428 >> v1:172.21.15.171:6789/0 conn(0x7fc32409f7c0 legacy=0x7fc32409de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 1 -- 172.21.15.171:0/3040130428 shutdown_connections 2024-09-16T04:10:19.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 1 -- 172.21.15.171:0/3040130428 >> 172.21.15.171:0/3040130428 conn(0x7fc3240a5450 msgr2=0x7fc3240a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 1 -- 172.21.15.171:0/3040130428 shutdown_connections 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.440+0000 7fc328e53640 1 -- 172.21.15.171:0/3040130428 wait complete. 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.441+0000 7fc328e53640 10 monclient: build_initial_monmap 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.441+0000 7fc328e53640 10 monclient: monmap: 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:19.454 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.441+0000 7fc328e53640 1 Processor -- start 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.441+0000 7fc328e53640 1 -- start start 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.441+0000 7fc328e53640 10 monclient: init 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient(hunting): picked mon.noname-c con 0x7fc32409f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient(hunting): picked mon.noname-a con 0x7fc32409fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient(hunting): picked mon.noname-b con 0x7fc3241395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc324120b10 con 0x7fc32409fba0 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32411f100 con 0x7fc3241395d0 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3240a3f20 con 0x7fc32409f7c0 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:19.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc328e53640 10 monclient(hunting): authenticate will time out at 994.948395s 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc322d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc3241395d0 0x7fc3241d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39394/0 (socket says 172.21.15.171:39394) 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc322d9d640 1 -- 172.21.15.171:0/2640941966 learned_addr learned my addr 172.21.15.171:0/2640941966 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc321d9b640 1 -- 172.21.15.171:0/2640941966 >> v1:172.21.15.148:6789/0 conn(0x7fc32409fba0 legacy=0x7fc3241d0ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3243741107 0 0) 0x7fc32411f100 con 0x7fc3241395d0 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): my global_id is 5653 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc304003520 con 0x7fc3241395d0 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936313708 0 0) 0x7fc3240a3f20 con 0x7fc32409f7c0 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.456 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): my global_id is 5684 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc32411f100 con 0x7fc32409f7c0 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1704796734 0 0) 0x7fc304003520 con 0x7fc3241395d0 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3240a3f20 con 0x7fc3241395d0 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc31c004440 con 0x7fc3241395d0 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.457 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.442+0000 7fc312ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2491548260 0 0) 0x7fc3240a3f20 con 0x7fc3241395d0 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 >> v1:172.21.15.148:6790/0 conn(0x7fc32409f7c0 legacy=0x7fc3241cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 >> v1:172.21.15.148:6789/0 conn(0x7fc32409fba0 legacy=0x7fc3241d0ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 monclient: found mon.b 2024-09-16T04:10:19.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32409c0b0 con 0x7fc3241395d0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 5 monclient: authenticate success, global_id 5653 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 monclient: _renew_subs 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc324120830 con 0x7fc3241395d0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 monclient: _renew_subs 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc32411db90 con 0x7fc3241395d0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc31c0046a0 con 0x7fc3241395d0 2024-09-16T04:10:19.459 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc31c0049f0 con 0x7fc3241395d0 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc320d99640 4 set_mon_vals no callback set 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 10 monclient: dump: 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:19.460 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc31c004c30 con 0x7fc3241395d0 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.443+0000 7fc328e53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.444+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc308005080 con 0x7fc3241395d0 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.444+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(141..141 src has 1..141) ==== 7467+0+0 (unknown 3857894836 0 0) 0x7fc31c047f40 con 0x7fc3241395d0 2024-09-16T04:10:19.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.444+0000 7fc32259c640 10 monclient: get_auth_request con 0x7fc304038fb0 auth_method 0 2024-09-16T04:10:19.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.448+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc31c015f20 con 0x7fc3241395d0 2024-09-16T04:10:19.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.448+0000 7fc312ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.448+0000 7fc312ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:19.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.619+0000 7fc328e53640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T04:10:19.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.619+0000 7fc328e53640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T04:10:19.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.619+0000 7fc328e53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:19.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:19.619+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) -- 0x7fc308005370 con 0x7fc3241395d0 2024-09-16T04:10:20.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.247+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v142) ==== 107+0+0 (unknown 3739112647 0 0) 0x7fc31c015820 con 0x7fc3241395d0 2024-09-16T04:10:20.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.247+0000 7fc312ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T04:10:20.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.247+0000 7fc312ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:20.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc328e53640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T04:10:20.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc328e53640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T04:10:20.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc328e53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) -- 0x7fc308005bd0 con 0x7fc3241395d0 2024-09-16T04:10:20.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc312ffd640 1 -- 172.21.15.171:0/2640941966 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v142) ==== 149+0+0 (unknown 1146471745 0 0) 0x7fc31c019b70 con 0x7fc3241395d0 2024-09-16T04:10:20.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc312ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-16T04:10:20.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.356+0000 7fc312ffd640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-16T04:10:20.360 INFO:tasks.workunit.client.0.smithi171.stderr:weight-set for pool 'cool' already created 2024-09-16T04:10:20.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 >> v1:172.21.15.171:6812/7327273 conn(0x7fc304038fb0 legacy=0x7fc30403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 10 monclient: shutdown 2024-09-16T04:10:20.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:20.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 >> v1:172.21.15.171:6789/0 conn(0x7fc3241395d0 legacy=0x7fc3241d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 shutdown_connections 2024-09-16T04:10:20.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 >> 172.21.15.171:0/2640941966 conn(0x7fc3240a5450 msgr2=0x7fc324138e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:20.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 shutdown_connections 2024-09-16T04:10:20.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.358+0000 7fc328e53640 1 -- 172.21.15.171:0/2640941966 wait complete. 2024-09-16T04:10:20.370 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create cold positional 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.449+0000 7f5cf5a5e640 10 monclient: get_monmap_and_config 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.449+0000 7f5cf5a5e640 10 monclient: build_initial_monmap 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.449+0000 7f5cf5a5e640 10 monclient: monmap: 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:20.450877+0000 2024-09-16T04:10:20.452 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:20.450877+0000 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.449+0000 7f5cf5a5e640 1 Processor -- start 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.449+0000 7f5cf5a5e640 1 -- start start 2024-09-16T04:10:20.453 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.449+0000 7f5cf5a5e640 10 monclient: init 2024-09-16T04:10:20.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:20.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:20.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient(hunting): picked mon.noname-b con 0x7f5cf0136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:20.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient(hunting): picked mon.noname-c con 0x7f5cf0137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:20.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient(hunting): picked mon.noname-a con 0x7f5cf01375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf009b2c0 con 0x7f5cf01375a0 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf009b5a0 con 0x7f5cf0137140 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf00a6640 con 0x7f5cf0136d60 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient(hunting): authenticate will time out at 995.956271s 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cef7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f5cf01375a0 legacy=0x7f5cf013f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5ce67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5cf0137140 0x7f5cf013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39402/0 (socket says 172.21.15.171:39402) 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5ce67fc640 1 -- 172.21.15.171:0/3501572860 learned_addr learned my addr 172.21.15.171:0/3501572860 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1675055183 0 0) 0x7f5cf009b5a0 con 0x7f5cf0137140 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): my global_id is 5656 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cdc003520 con 0x7f5cf0137140 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815392213 0 0) 0x7f5cf00a6640 con 0x7f5cf0136d60 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient(hunting): my global_id is 5687 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf009b5a0 con 0x7f5cf0136d60 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3619941192 0 0) 0x7f5cdc003520 con 0x7f5cf0137140 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 >> v1:172.21.15.148:6790/0 conn(0x7f5cf0136d60 legacy=0x7f5cf009bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 >> v1:172.21.15.148:6789/0 conn(0x7f5cf01375a0 legacy=0x7f5cf013f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:20.462 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf0143b20 con 0x7f5cf0137140 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5cd8001610 con 0x7f5cf0137140 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 5 monclient: authenticate success, global_id 5656 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: dump: 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:20.463 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cd80018c0 con 0x7f5cf0137140 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 1 -- 172.21.15.171:0/3501572860 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5cd8001c90 con 0x7f5cf0137140 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:20.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cf5a5e640 4 set_mon_vals no callback set 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.450+0000 7f5cee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cee7fc640 10 monclient: dump: 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.465 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 10 monclient: shutdown 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/3501572860 >> v1:172.21.15.171:6789/0 conn(0x7f5cf0137140 legacy=0x7f5cf013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/3501572860 shutdown_connections 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/3501572860 >> 172.21.15.171:0/3501572860 conn(0x7f5cf00a9ae0 msgr2=0x7f5cf00a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/3501572860 shutdown_connections 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.451+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/3501572860 wait complete. 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient: build_initial_monmap 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient: monmap: 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:20.466 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 1 Processor -- start 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 1 -- start start 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient: init 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:20.467 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient(hunting): picked mon.noname-a con 0x7f5cf0136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient(hunting): picked mon.noname-b con 0x7f5cf0137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 10 monclient(hunting): picked mon.noname-c con 0x7f5cf01375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.452+0000 7f5cf5a5e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf012eab0 con 0x7f5cf0136d60 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf0130b80 con 0x7f5cf0137140 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf00a87d0 con 0x7f5cf01375a0 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 10 monclient(hunting): authenticate will time out at 995.959042s 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5ceeffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f5cf0136d60 legacy=0x7f5cf012dc90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5ce67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5cf0137140 0x7f5cf012e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39410/0 (socket says 172.21.15.171:39410) 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5ce67fc640 1 -- 172.21.15.171:0/2104531454 learned_addr learned my addr 172.21.15.171:0/2104531454 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719365001 0 0) 0x7f5cf0130b80 con 0x7f5cf0137140 2024-09-16T04:10:20.468 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): my global_id is 5659 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd0003520 con 0x7f5cf0137140 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677941865 0 0) 0x7f5cf00a87d0 con 0x7f5cf01375a0 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): my global_id is 5690 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf0130b80 con 0x7f5cf01375a0 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 833509744 0 0) 0x7f5cd0003520 con 0x7f5cf0137140 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cf00a87d0 con 0x7f5cf0137140 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5cd8001c10 con 0x7f5cf0137140 2024-09-16T04:10:20.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): dump: 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:20.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2906200368 0 0) 0x7f5cf00a87d0 con 0x7f5cf0137140 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 >> v1:172.21.15.148:6790/0 conn(0x7f5cf01375a0 legacy=0x7f5cf01d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 >> v1:172.21.15.148:6789/0 conn(0x7f5cf0136d60 legacy=0x7f5cf012dc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: _finish_hunting 0 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 monclient: found mon.b 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00a82a0 con 0x7f5cf0137140 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: _finish_auth 0 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:20.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 5 monclient: authenticate success, global_id 5659 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 10 monclient: _renew_subs 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf01305c0 con 0x7f5cf0137140 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 10 monclient: _renew_subs 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5cf012f830 con 0x7f5cf0137140 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cd8004c40 con 0x7f5cf0137140 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5cd8004f90 con 0x7f5cf0137140 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cedffb640 4 set_mon_vals no callback set 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:20.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.453+0000 7f5cecff9640 10 monclient: dump: 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.473 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5cd80051b0 con 0x7f5cf0137140 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5cf5a5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5cf5a5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5cf5a5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(142..142 src has 1..142) ==== 7799+0+0 (unknown 2936103556 0 0) 0x7f5cd8047fe0 con 0x7f5cf0137140 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd4005080 con 0x7f5cf0137140 2024-09-16T04:10:20.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.454+0000 7f5ceeffd640 10 monclient: get_auth_request con 0x7f5cd0038f10 auth_method 0 2024-09-16T04:10:20.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.459+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5cd8015e70 con 0x7f5cf0137140 2024-09-16T04:10:20.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.459+0000 7f5cecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:20.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.459+0000 7f5cecff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:20.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.632+0000 7f5cf5a5e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T04:10:20.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.632+0000 7f5cf5a5e640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T04:10:20.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.632+0000 7f5cf5a5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:20.635 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:20.632+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) -- 0x7f5cd4005370 con 0x7f5cf0137140 2024-09-16T04:10:20.710 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:10:20.710 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:10:20.711 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:10:20.711 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:10:20.714 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:10:20.714 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:10:20.715 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.838+0000 7f5c42598640 10 monclient: get_monmap_and_config 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.838+0000 7f5c42598640 10 monclient: build_initial_monmap 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.838+0000 7f5c42598640 10 monclient: monmap: 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:20.839209+0000 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:20.839209+0000 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.840 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:20.841 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:20.841 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:20.841 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 1 Processor -- start 2024-09-16T04:10:20.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 1 -- start start 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient: init 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3c0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a0e00 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a10e0 con 0x7f5c3c0a2410 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c0a4970 con 0x7f5c3c13d060 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:20.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c42598640 10 monclient(hunting): authenticate will time out at 1041.464613s 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5c3c0a1b10 0x7f5c3c13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48634/0 (socket says 172.21.15.148:48634) 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3b7fe640 1 -- 172.21.15.148:0/2280034117 learned_addr learned my addr 172.21.15.148:0/2280034117 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793749367 0 0) 0x7f5c3c0a0e00 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): my global_id is 64098 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c24003520 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1655926084 0 0) 0x7f5c3c0a4970 con 0x7f5c3c13d060 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 10 monclient(hunting): my global_id is 5693 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.839+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c0a0e00 con 0x7f5c3c13d060 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587926499 0 0) 0x7f5c3c0a10e0 con 0x7f5c3c0a2410 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient(hunting): my global_id is 5662 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c0a4970 con 0x7f5c3c0a2410 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2709576852 0 0) 0x7f5c3c0a0e00 con 0x7f5c3c13d060 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 >> v1:172.21.15.171:6789/0 conn(0x7f5c3c0a2410 legacy=0x7f5c3c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 >> v1:172.21.15.148:6789/0 conn(0x7f5c3c0a1b10 legacy=0x7f5c3c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 monclient: found mon.noname-b 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:20.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c06dfd0 con 0x7f5c3c13d060 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c34001970 con 0x7f5c3c13d060 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 5 monclient: authenticate success, global_id 5693 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: dump: 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.852 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c34001c50 con 0x7f5c3c13d060 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 1 -- 172.21.15.148:0/2280034117 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c34002060 con 0x7f5c3c13d060 2024-09-16T04:10:20.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 4 set_mon_vals no callback set 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c3affd640 10 monclient: dump: 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.854 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 10 monclient: shutdown 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 1 -- 172.21.15.148:0/2280034117 >> v1:172.21.15.148:6790/0 conn(0x7f5c3c13d060 legacy=0x7f5c3c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 1 -- 172.21.15.148:0/2280034117 shutdown_connections 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 1 -- 172.21.15.148:0/2280034117 >> 172.21.15.148:0/2280034117 conn(0x7f5c3c0a97b0 msgr2=0x7f5c3c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 1 -- 172.21.15.148:0/2280034117 shutdown_connections 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.840+0000 7f5c42598640 1 -- 172.21.15.148:0/2280034117 wait complete. 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.841+0000 7f5c42598640 10 monclient: build_initial_monmap 2024-09-16T04:10:20.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.841+0000 7f5c42598640 10 monclient: monmap: 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:20.856 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.841+0000 7f5c42598640 1 Processor -- start 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 1 -- start start 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient: init 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient(hunting): picked mon.noname-c con 0x7f5c3c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient(hunting): picked mon.noname-b con 0x7f5c3c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c09e9f0 con 0x7f5c3c0a2410 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c137f90 con 0x7f5c3c13d060 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c3c09f2d0 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c42598640 10 monclient(hunting): authenticate will time out at 1041.467516s 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c3bfff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5c3c0a1b10 0x7f5c3c1328c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:45208/0 (socket says 172.21.15.148:45208) 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c3bfff640 1 -- 172.21.15.148:0/1123093555 learned_addr learned my addr 172.21.15.148:0/1123093555 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313685006 0 0) 0x7f5c3c09f2d0 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): my global_id is 5696 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c28003520 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1533617705 0 0) 0x7f5c3c09e9f0 con 0x7f5c3c0a2410 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): my global_id is 64101 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c09f2d0 con 0x7f5c3c0a2410 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312321910 0 0) 0x7f5c3c137f90 con 0x7f5c3c13d060 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 10 monclient(hunting): my global_id is 5665 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.842+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c09e9f0 con 0x7f5c3c13d060 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2160552411 0 0) 0x7f5c28003520 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c3c137f90 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c2c001930 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:20.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): dump: 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3079878198 0 0) 0x7f5c3c09f2d0 con 0x7f5c3c0a2410 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c28003520 con 0x7f5c3c0a2410 2024-09-16T04:10:20.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 152845559 0 0) 0x7f5c3c09e9f0 con 0x7f5c3c13d060 2024-09-16T04:10:20.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c3c09f2d0 con 0x7f5c3c13d060 2024-09-16T04:10:20.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c34001870 con 0x7f5c3c13d060 2024-09-16T04:10:20.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:20.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:20.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): dump: 2024-09-16T04:10:20.862 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:20.862 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.862 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.862 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c30001190 con 0x7f5c3c0a2410 2024-09-16T04:10:20.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:20.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:20.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient(hunting): dump: 2024-09-16T04:10:20.864 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:20.864 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.864 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.864 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3416601089 0 0) 0x7f5c3c137f90 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 >> v1:172.21.15.171:6789/0 conn(0x7f5c3c13d060 legacy=0x7f5c3c1d4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 >> v1:172.21.15.148:6789/0 conn(0x7f5c3c0a2410 legacy=0x7f5c3c1d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 monclient: found mon.c 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c13cc20 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: _finish_auth 0 2024-09-16T04:10:20.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 5 monclient: authenticate success, global_id 5696 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 10 monclient: _renew_subs 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c0a0130 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 10 monclient: _renew_subs 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c3c09f770 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c2c001d00 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c3a7fc640 4 set_mon_vals no callback set 2024-09-16T04:10:20.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c2c003a60 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.843+0000 7f5c38ff9640 10 monclient: dump: 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:20.868 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5c2c003c80 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c42598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:20.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c42598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c42598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3c140590 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7799+0+0 (unknown 2936103556 0 0) 0x7f5c2c0459d0 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.844+0000 7f5c3b7fe640 10 monclient: get_auth_request con 0x7f5c28039300 auth_method 0 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.849+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5c2c015ab0 con 0x7f5c3c0a1b10 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.849+0000 7f5c38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:20.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:20.849+0000 7f5c38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.078+0000 7f5c42598640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.078+0000 7f5c42598640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.078+0000 7f5c42598640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.078+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5c3c0a46a0 con 0x7f5c3c0a1b10 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.079+0000 7f5c38ff9640 1 -- 172.21.15.148:0/1123093555 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3652644885) 0x7f5c2c003f70 con 0x7f5c3c0a1b10 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.079+0000 7f5c38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.079+0000 7f5c38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:21.081 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:21.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 >> v1:172.21.15.171:6812/7327273 conn(0x7f5c28039300 legacy=0x7f5c2803b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 10 monclient: shutdown 2024-09-16T04:10:21.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 >> v1:172.21.15.148:6790/0 conn(0x7f5c3c0a1b10 legacy=0x7f5c3c1328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 shutdown_connections 2024-09-16T04:10:21.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 >> 172.21.15.148:0/1123093555 conn(0x7f5c3c0a97b0 msgr2=0x7f5c3c0a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 shutdown_connections 2024-09-16T04:10:21.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.081+0000 7f5c42598640 1 -- 172.21.15.148:0/1123093555 wait complete. 2024-09-16T04:10:21.094 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:10:21.094 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:10:21.094 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.192+0000 7fe1fd191640 10 monclient: get_monmap_and_config 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.192+0000 7fe1fd191640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.192+0000 7fe1fd191640 10 monclient: monmap: 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:21.193070+0000 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:21.193070+0000 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.194 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:21.195 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:21.195 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.192+0000 7fe1fd191640 1 Processor -- start 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.192+0000 7fe1fd191640 1 -- start start 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.192+0000 7fe1fd191640 10 monclient: init 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f80a0e00 con 0x7fe1f80a2410 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f80a10e0 con 0x7fe1f813d060 2024-09-16T04:10:21.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f80a4970 con 0x7fe1f80a1b10 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 10 monclient(hunting): authenticate will time out at 1041.818352s 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f659c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe1f80a1b10 0x7fe1f813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:45218/0 (socket says 172.21.15.148:45218) 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f6d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe1f80a2410 0x7fe1f80a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48644/0 (socket says 172.21.15.148:48644) 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f659c640 1 -- 172.21.15.148:0/2073782250 learned_addr learned my addr 172.21.15.148:0/2073782250 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795302668 0 0) 0x7fe1f80a0e00 con 0x7fe1f80a2410 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): my global_id is 64104 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e0003520 con 0x7fe1f80a2410 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006774501 0 0) 0x7fe1f80a4970 con 0x7fe1f80a1b10 2024-09-16T04:10:21.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): my global_id is 5699 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1f80a0e00 con 0x7fe1f80a1b10 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228331802 0 0) 0x7fe1f80a10e0 con 0x7fe1f813d060 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient(hunting): my global_id is 5668 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1f80a4970 con 0x7fe1f813d060 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3345995767 0 0) 0x7fe1e0003520 con 0x7fe1f80a2410 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 >> v1:172.21.15.148:6790/0 conn(0x7fe1f80a1b10 legacy=0x7fe1f813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 >> v1:172.21.15.171:6789/0 conn(0x7fe1f813d060 legacy=0x7fe1f813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f806dfd0 con 0x7fe1f80a2410 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 5 monclient: authenticate success, global_id 64104 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1e8001610 con 0x7fe1f80a2410 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1f5d9b640 10 monclient: dump: 2024-09-16T04:10:21.207 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.193+0000 7fe1fd191640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1e8001930 con 0x7fe1f80a2410 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1f5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1f5d9b640 1 -- 172.21.15.148:0/2073782250 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1e8001d40 con 0x7fe1f80a2410 2024-09-16T04:10:21.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 4 set_mon_vals no callback set 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1f5d9b640 10 monclient: dump: 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.209 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 10 monclient: shutdown 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 1 -- 172.21.15.148:0/2073782250 >> v1:172.21.15.148:6789/0 conn(0x7fe1f80a2410 legacy=0x7fe1f80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 1 -- 172.21.15.148:0/2073782250 shutdown_connections 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 1 -- 172.21.15.148:0/2073782250 >> 172.21.15.148:0/2073782250 conn(0x7fe1f80a97b0 msgr2=0x7fe1f80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 1 -- 172.21.15.148:0/2073782250 shutdown_connections 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.194+0000 7fe1fd191640 1 -- 172.21.15.148:0/2073782250 wait complete. 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.195+0000 7fe1fd191640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.195+0000 7fe1fd191640 10 monclient: monmap: 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:21.210 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.195+0000 7fe1fd191640 1 Processor -- start 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.195+0000 7fe1fd191640 1 -- start start 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.195+0000 7fe1fd191640 10 monclient: init 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient(hunting): picked mon.noname-a con 0x7fe1f80a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient(hunting): picked mon.noname-c con 0x7fe1f80a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient(hunting): picked mon.noname-b con 0x7fe1f813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f813cb20 con 0x7fe1f80a1b10 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f80a0130 con 0x7fe1f813d060 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f81387f0 con 0x7fe1f80a2410 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1fd191640 10 monclient(hunting): authenticate will time out at 1041.821249s 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1f6d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe1f80a1b10 0x7fe1f813c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48656/0 (socket says 172.21.15.148:48656) 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1f6d9d640 1 -- 172.21.15.148:0/3712120374 learned_addr learned my addr 172.21.15.148:0/3712120374 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287597861 0 0) 0x7fe1f81387f0 con 0x7fe1f80a2410 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): my global_id is 5702 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1e4003520 con 0x7fe1f80a2410 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167137909 0 0) 0x7fe1f813cb20 con 0x7fe1f80a1b10 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): my global_id is 64107 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1f81387f0 con 0x7fe1f80a1b10 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201248412 0 0) 0x7fe1f80a0130 con 0x7fe1f813d060 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): my global_id is 5671 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1f813cb20 con 0x7fe1f813d060 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 507734151 0 0) 0x7fe1e4003520 con 0x7fe1f80a2410 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1f80a0130 con 0x7fe1f80a2410 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1ec0018b0 con 0x7fe1f80a2410 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.214 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2925023169 0 0) 0x7fe1f81387f0 con 0x7fe1f80a1b10 2024-09-16T04:10:21.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1e4003520 con 0x7fe1f80a1b10 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1e80015d0 con 0x7fe1f80a1b10 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.216 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2354698830 0 0) 0x7fe1f813cb20 con 0x7fe1f813d060 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1f81387f0 con 0x7fe1f813d060 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1f0001bb0 con 0x7fe1f813d060 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.217 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3167909740 0 0) 0x7fe1f80a0130 con 0x7fe1f80a2410 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.171:6789/0 conn(0x7fe1f813d060 legacy=0x7fe1f81d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.196+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.148:6789/0 conn(0x7fe1f80a1b10 legacy=0x7fe1f813c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 1 monclient: found mon.c 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f813a800 con 0x7fe1f80a2410 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 5 monclient: authenticate success, global_id 5702 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 10 monclient: _renew_subs 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f8139350 con 0x7fe1f80a2410 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 10 monclient: _renew_subs 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1f81d4bd0 con 0x7fe1f80a2410 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe1ec0047e0 con 0x7fe1f80a2410 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1ec004b30 con 0x7fe1f80a2410 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1f559a640 4 set_mon_vals no callback set 2024-09-16T04:10:21.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 10 monclient: dump: 2024-09-16T04:10:21.220 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.220 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.220 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe1ec00fdc0 con 0x7fe1f80a2410 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.197+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1d80065c0 con 0x7fe1d8002860 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7799+0+0 (unknown 2936103556 0 0) 0x7fe1ec047b00 con 0x7fe1f80a2410 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1f759e640 10 monclient: get_auth_request con 0x7fe1e4039100 auth_method 0 2024-09-16T04:10:21.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3618927 0 0) 0x7fe1d80065c0 con 0x7fe1d8002860 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient(hunting): my global_id is 64110 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1f80a0130 con 0x7fe1d8002860 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4121451960 0 0) 0x7fe1f80a0130 con 0x7fe1d8002860 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1d80065c0 con 0x7fe1d8002860 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1e8004da0 con 0x7fe1d8002860 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient: dump: 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.222 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3844355067 0 0) 0x7fe1d80065c0 con 0x7fe1d8002860 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1d800a150 con 0x7fe1d8002860 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe1e8006320 con 0x7fe1d8002860 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:21.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.148:6789/0 conn(0x7fe1d8002860 legacy=0x7fe1d8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.198+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.148:6789/0 conn(0x7fe1d8002860 legacy=0x7fe1d8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.204+0000 7fe1fd191640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:21.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.204+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe1d8009e40 con 0x7fe1d8009100 2024-09-16T04:10:21.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2615894308 0 0) 0x7fe1d8009e40 con 0x7fe1d8009100 2024-09-16T04:10:21.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient(hunting): my global_id is 64113 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d80065c0 con 0x7fe1d8009100 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1047353602 0 0) 0x7fe1d80065c0 con 0x7fe1d8009100 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1d8009e40 con 0x7fe1d8009100 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1e8005090 con 0x7fe1d8009100 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient: dump: 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.226 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4275626050 0 0) 0x7fe1d8009e40 con 0x7fe1d8009100 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1d800c3d0 con 0x7fe1d8009100 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 2607829493) 0x7fe1e80064d0 con 0x7fe1d8009100 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.148:6789/0 conn(0x7fe1d8009100 legacy=0x7fe1d8009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.205+0000 7fe1df7fe640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.148:6789/0 conn(0x7fe1d8009100 legacy=0x7fe1d8009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 22, 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 403, 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:10:21.228 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.229 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.230 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:21.231 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.232 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:21.233 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.234 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.235 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.236 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:21.237 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.239 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:21.239 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.171:6812/7327273 conn(0x7fe1e4039100 legacy=0x7fe1e403b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 10 monclient: shutdown 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 >> v1:172.21.15.148:6790/0 conn(0x7fe1f80a2410 legacy=0x7fe1f81cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1f759e640 1 -- 172.21.15.148:0/3712120374 reap_dead start 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1f759e640 1 -- 172.21.15.148:0/3712120374 reap_dead start 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 shutdown_connections 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 >> 172.21.15.148:0/3712120374 conn(0x7fe1f80a97b0 msgr2=0x7fe1f80a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 shutdown_connections 2024-09-16T04:10:21.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.206+0000 7fe1fd191640 1 -- 172.21.15.148:0/3712120374 wait complete. 2024-09-16T04:10:21.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:21.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:10:21.241 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.337+0000 7ff84047b640 10 monclient: get_monmap_and_config 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.337+0000 7ff84047b640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.337+0000 7ff84047b640 10 monclient: monmap: 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:21.338276+0000 2024-09-16T04:10:21.339 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:21.338276+0000 2024-09-16T04:10:21.340 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.340 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.340 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.340 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:21.340 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:21.340 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.337+0000 7ff84047b640 1 Processor -- start 2024-09-16T04:10:21.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.337+0000 7ff84047b640 1 -- start start 2024-09-16T04:10:21.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.337+0000 7ff84047b640 10 monclient: init 2024-09-16T04:10:21.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient(hunting): picked mon.noname-c con 0x7ff83809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient(hunting): picked mon.noname-b con 0x7ff83809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient(hunting): picked mon.noname-a con 0x7ff83809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8380a1010 con 0x7ff83809cc30 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83809f030 con 0x7ff83809bb70 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83809dc60 con 0x7ff83809c660 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff84047b640 10 monclient(hunting): authenticate will time out at 1041.963583s 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83ea18640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff83809cc30 0x7ff83813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48674/0 (socket says 172.21.15.148:48674) 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83da16640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff83809c660 0x7ff838138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:45242/0 (socket says 172.21.15.148:45242) 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83ea18640 1 -- 172.21.15.148:0/2204686900 learned_addr learned my addr 172.21.15.148:0/2204686900 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273508006 0 0) 0x7ff8380a1010 con 0x7ff83809cc30 2024-09-16T04:10:21.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): my global_id is 64116 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff824003520 con 0x7ff83809cc30 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425867183 0 0) 0x7ff83809dc60 con 0x7ff83809c660 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 10 monclient(hunting): my global_id is 5705 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.338+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8380a1010 con 0x7ff83809c660 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 388912476 0 0) 0x7ff824003520 con 0x7ff83809cc30 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 >> v1:172.21.15.148:6790/0 conn(0x7ff83809c660 legacy=0x7ff838138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 >> v1:172.21.15.171:6789/0 conn(0x7ff83809bb70 legacy=0x7ff83809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 monclient: found mon.noname-a 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83809df40 con 0x7ff83809cc30 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff830001990 con 0x7ff83809cc30 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 5 monclient: authenticate success, global_id 64116 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: dump: 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff830001b80 con 0x7ff83809cc30 2024-09-16T04:10:21.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 1 -- 172.21.15.148:0/2204686900 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff830001f90 con 0x7ff83809cc30 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 4 set_mon_vals no callback set 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff83d215640 10 monclient: dump: 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 10 monclient: shutdown 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 1 -- 172.21.15.148:0/2204686900 >> v1:172.21.15.148:6789/0 conn(0x7ff83809cc30 legacy=0x7ff83813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 1 -- 172.21.15.148:0/2204686900 shutdown_connections 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 1 -- 172.21.15.148:0/2204686900 >> 172.21.15.148:0/2204686900 conn(0x7ff8380a5640 msgr2=0x7ff8380a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 1 -- 172.21.15.148:0/2204686900 shutdown_connections 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.339+0000 7ff84047b640 1 -- 172.21.15.148:0/2204686900 wait complete. 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.340+0000 7ff84047b640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.340+0000 7ff84047b640 10 monclient: monmap: 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.355 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.340+0000 7ff84047b640 1 Processor -- start 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.340+0000 7ff84047b640 1 -- start start 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.340+0000 7ff84047b640 10 monclient: init 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient(hunting): picked mon.noname-a con 0x7ff83809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient(hunting): picked mon.noname-b con 0x7ff83809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient(hunting): picked mon.noname-c con 0x7ff83809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff838135630 con 0x7ff83809bb70 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff838135910 con 0x7ff83809c660 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff838132840 con 0x7ff83809cc30 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff83e217640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff83809bb70 0x7ff838131a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48686/0 (socket says 172.21.15.148:48686) 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff83e217640 1 -- 172.21.15.148:0/1889865222 learned_addr learned my addr 172.21.15.148:0/1889865222 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff84047b640 10 monclient(hunting): authenticate will time out at 1041.966380s 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557209681 0 0) 0x7ff838135630 con 0x7ff83809bb70 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): my global_id is 64119 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff81c003520 con 0x7ff83809bb70 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155687612 0 0) 0x7ff838132840 con 0x7ff83809cc30 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): my global_id is 5708 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff838135630 con 0x7ff83809cc30 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2571808271 0 0) 0x7ff838135910 con 0x7ff83809c660 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): my global_id is 5677 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff838132840 con 0x7ff83809c660 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3178587808 0 0) 0x7ff838135630 con 0x7ff83809cc30 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff838135910 con 0x7ff83809cc30 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff830001ed0 con 0x7ff83809cc30 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): dump: 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.359 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2731885239 0 0) 0x7ff81c003520 con 0x7ff83809bb70 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff838135630 con 0x7ff83809bb70 2024-09-16T04:10:21.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff8340018c0 con 0x7ff83809bb70 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): dump: 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.361 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3740196733 0 0) 0x7ff838132840 con 0x7ff83809c660 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff81c003520 con 0x7ff83809c660 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff8280010b0 con 0x7ff83809c660 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.341+0000 7ff82effd640 10 monclient(hunting): dump: 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.362 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1869827869 0 0) 0x7ff838135910 con 0x7ff83809cc30 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.171:6789/0 conn(0x7ff83809c660 legacy=0x7ff838132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.148:6789/0 conn(0x7ff83809bb70 legacy=0x7ff838131a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 monclient: found mon.c 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff838134220 con 0x7ff83809cc30 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 5 monclient: authenticate success, global_id 5708 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 10 monclient: _renew_subs 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff83813b820 con 0x7ff83809cc30 2024-09-16T04:10:21.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 10 monclient: _renew_subs 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff838135350 con 0x7ff83809cc30 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff830001b80 con 0x7ff83809cc30 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff830005110 con 0x7ff83809cc30 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 10 monclient: dump: 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.365 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff83ca14640 4 set_mon_vals no callback set 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff8300103a0 con 0x7ff83809cc30 2024-09-16T04:10:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.342+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff8200065c0 con 0x7ff820002860 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7799+0+0 (unknown 2936103556 0 0) 0x7ff830048090 con 0x7ff83809cc30 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1829808257 0 0) 0x7ff8200065c0 con 0x7ff820002860 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient(hunting): my global_id is 5711 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff838135910 con 0x7ff820002860 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff83da16640 10 monclient: get_auth_request con 0x7ff81c039140 auth_method 0 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3959947842 0 0) 0x7ff838135910 con 0x7ff820002860 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8200065c0 con 0x7ff820002860 2024-09-16T04:10:21.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff834003aa0 con 0x7ff820002860 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 10 monclient: dump: 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.368 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1888864009 0 0) 0x7ff8200065c0 con 0x7ff820002860 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.343+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff82000a150 con 0x7ff820002860 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.344+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff8340057a0 con 0x7ff820002860 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.344+0000 7ff82effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.344+0000 7ff82effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:21.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.344+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.148:6790/0 conn(0x7ff820002860 legacy=0x7ff820004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.344+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.148:6790/0 conn(0x7ff820002860 legacy=0x7ff820004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff84047b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff820009e20 con 0x7ff8200090e0 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3768983011 0 0) 0x7ff820009e20 con 0x7ff8200090e0 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient(hunting): my global_id is 5714 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8200065c0 con 0x7ff8200090e0 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3379199787 0 0) 0x7ff8200065c0 con 0x7ff8200090e0 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff820009e20 con 0x7ff8200090e0 2024-09-16T04:10:21.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff8340041e0 con 0x7ff8200090e0 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient: dump: 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.373 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 526422460 0 0) 0x7ff820009e20 con 0x7ff8200090e0 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff82000c390 con 0x7ff8200090e0 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 2671055376) 0x7ff834004000 con 0x7ff8200090e0 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:21.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.148:6790/0 conn(0x7ff8200090e0 legacy=0x7ff820009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.350+0000 7ff82effd640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.148:6790/0 conn(0x7ff8200090e0 legacy=0x7ff820009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:10:21.375 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 22, 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 245428, 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:21.376 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.377 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.378 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:21.379 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:21.380 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.381 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:21.382 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:21.383 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.384 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:21.385 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.386 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.387 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:10:21.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:10:21.389 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.390 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.390 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:21.390 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:21.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.171:6812/7327273 conn(0x7ff81c039140 legacy=0x7ff81c03b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 10 monclient: shutdown 2024-09-16T04:10:21.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 >> v1:172.21.15.148:6790/0 conn(0x7ff83809cc30 legacy=0x7ff8381d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff83ea18640 1 -- 172.21.15.148:0/1889865222 reap_dead start 2024-09-16T04:10:21.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff83ea18640 1 -- 172.21.15.148:0/1889865222 reap_dead start 2024-09-16T04:10:21.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 shutdown_connections 2024-09-16T04:10:21.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 >> 172.21.15.148:0/1889865222 conn(0x7ff8380a5640 msgr2=0x7ff83809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 shutdown_connections 2024-09-16T04:10:21.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.351+0000 7ff84047b640 1 -- 172.21.15.148:0/1889865222 wait complete. 2024-09-16T04:10:21.392 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:21.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:10:21.393 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.489+0000 7f930e882640 10 monclient: get_monmap_and_config 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.489+0000 7f930e882640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.489+0000 7f930e882640 10 monclient: monmap: 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:21.490232+0000 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:21.490232+0000 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.491 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.492 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.492 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:21.492 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:21.492 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 1 Processor -- start 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 1 -- start start 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient: init 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient(hunting): picked mon.noname-c con 0x7f93080a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient(hunting): picked mon.noname-b con 0x7f93080a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient(hunting): picked mon.noname-a con 0x7f930813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93080a0e00 con 0x7f930813d060 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93080a10e0 con 0x7f93080a2410 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93080a4970 con 0x7f93080a1b10 2024-09-16T04:10:21.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930e882640 10 monclient(hunting): authenticate will time out at 1042.115809s 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930ce1f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f930813d060 0x7f930813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48692/0 (socket says 172.21.15.148:48692) 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f92fffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f93080a1b10 0x7f930813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:45286/0 (socket says 172.21.15.148:45286) 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.490+0000 7f930ce1f640 1 -- 172.21.15.148:0/42571812 learned_addr learned my addr 172.21.15.148:0/42571812 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364763941 0 0) 0x7f93080a4970 con 0x7f93080a1b10 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): my global_id is 5717 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92f4003520 con 0x7f93080a1b10 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100881013 0 0) 0x7f93080a0e00 con 0x7f930813d060 2024-09-16T04:10:21.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): my global_id is 64122 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93080a4970 con 0x7f930813d060 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468307286 0 0) 0x7f93080a10e0 con 0x7f93080a2410 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient(hunting): my global_id is 5680 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93080a0e00 con 0x7f93080a2410 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 564407414 0 0) 0x7f93080a4970 con 0x7f930813d060 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 >> v1:172.21.15.148:6790/0 conn(0x7f93080a1b10 legacy=0x7f930813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 >> v1:172.21.15.171:6789/0 conn(0x7f93080a2410 legacy=0x7f93080a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 monclient: found mon.noname-a 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930806dfd0 con 0x7f930813d060 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9300001910 con 0x7f930813d060 2024-09-16T04:10:21.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f930e882640 5 monclient: authenticate success, global_id 64122 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f930e882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: dump: 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.506 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f930e882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9300001bb0 con 0x7f930813d060 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f930e882640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 1 -- 172.21.15.148:0/42571812 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9300001fc0 con 0x7f930813d060 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f930e882640 4 set_mon_vals no callback set 2024-09-16T04:10:21.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.491+0000 7f93077fe640 10 monclient: dump: 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.508 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.509 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 10 monclient: shutdown 2024-09-16T04:10:21.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 1 -- 172.21.15.148:0/42571812 >> v1:172.21.15.148:6789/0 conn(0x7f930813d060 legacy=0x7f930813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 1 -- 172.21.15.148:0/42571812 shutdown_connections 2024-09-16T04:10:21.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 1 -- 172.21.15.148:0/42571812 >> 172.21.15.148:0/42571812 conn(0x7f93080a97b0 msgr2=0x7f93080a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 1 -- 172.21.15.148:0/42571812 shutdown_connections 2024-09-16T04:10:21.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 1 -- 172.21.15.148:0/42571812 wait complete. 2024-09-16T04:10:21.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.492+0000 7f930e882640 10 monclient: monmap: 2024-09-16T04:10:21.512 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 1 Processor -- start 2024-09-16T04:10:21.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 1 -- start start 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient: init 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient(hunting): picked mon.noname-b con 0x7f93080a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient(hunting): picked mon.noname-c con 0x7f93080a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient(hunting): picked mon.noname-a con 0x7f930813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9308137870 con 0x7f930813d060 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9308137b50 con 0x7f93080a1b10 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f930809d8c0 con 0x7f93080a2410 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930e882640 10 monclient(hunting): authenticate will time out at 1042.118554s 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f92fffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f93080a2410 0x7f930809d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:45292/0 (socket says 172.21.15.148:45292) 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f92fffff640 1 -- 172.21.15.148:0/2601624341 learned_addr learned my addr 172.21.15.148:0/2601624341 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f930ce1f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f930813d060 0x7f93081d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48700/0 (socket says 172.21.15.148:48700) 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3267958460 0 0) 0x7f930809d8c0 con 0x7f93080a2410 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): my global_id is 5720 2024-09-16T04:10:21.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92e8003550 con 0x7f93080a2410 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2455343432 0 0) 0x7f9308137870 con 0x7f930813d060 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): my global_id is 64125 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f930809d8c0 con 0x7f930813d060 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459770584 0 0) 0x7f9308137b50 con 0x7f93080a1b10 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 10 monclient(hunting): my global_id is 5683 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.493+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9308137870 con 0x7f93080a1b10 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 524058414 0 0) 0x7f92e8003550 con 0x7f93080a2410 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9308137b50 con 0x7f93080a2410 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92f00019e0 con 0x7f93080a2410 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): dump: 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.517 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3654762004 0 0) 0x7f930809d8c0 con 0x7f930813d060 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92e8003550 con 0x7f930813d060 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9300001f00 con 0x7f930813d060 2024-09-16T04:10:21.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): dump: 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.519 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2674754516 0 0) 0x7f9308137870 con 0x7f93080a1b10 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f930809d8c0 con 0x7f93080a1b10 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92f8001870 con 0x7f93080a1b10 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient(hunting): dump: 2024-09-16T04:10:21.520 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1072507105 0 0) 0x7f9308137b50 con 0x7f93080a2410 2024-09-16T04:10:21.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.171:6789/0 conn(0x7f93080a1b10 legacy=0x7f930809caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.148:6789/0 conn(0x7f930813d060 legacy=0x7f93081d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 monclient: found mon.c 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f930813cb00 con 0x7f93080a2410 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 5 monclient: authenticate success, global_id 5720 2024-09-16T04:10:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 10 monclient: _renew_subs 2024-09-16T04:10:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9308136bc0 con 0x7f93080a2410 2024-09-16T04:10:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 10 monclient: _renew_subs 2024-09-16T04:10:21.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f930809efd0 con 0x7f93080a2410 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92f0001cf0 con 0x7f93080a2410 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92f0004c30 con 0x7f93080a2410 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f9306ffd640 4 set_mon_vals no callback set 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.494+0000 7f93057fa640 10 monclient: dump: 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.524 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f930e882640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92ec0065c0 con 0x7f92ec002860 2024-09-16T04:10:21.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f92f000fec0 con 0x7f93080a2410 2024-09-16T04:10:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7799+0+0 (unknown 2936103556 0 0) 0x7f92f0047b00 con 0x7f93080a2410 2024-09-16T04:10:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f930ce1f640 10 monclient: get_auth_request con 0x7f92e8039130 auth_method 0 2024-09-16T04:10:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837002330 0 0) 0x7f92ec0065c0 con 0x7f92ec002860 2024-09-16T04:10:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient(hunting): my global_id is 5686 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9308137b50 con 0x7f92ec002860 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3503501195 0 0) 0x7f9308137b50 con 0x7f92ec002860 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92ec0065c0 con 0x7f92ec002860 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92f80035f0 con 0x7f92ec002860 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 10 monclient: dump: 2024-09-16T04:10:21.527 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.528 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2136042004 0 0) 0x7f92ec0065c0 con 0x7f92ec002860 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.495+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92ec00a150 con 0x7f92ec002860 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.496+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f92f80059a0 con 0x7f92ec002860 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.496+0000 7f93057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.496+0000 7f93057fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:21.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.496+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.171:6789/0 conn(0x7f92ec002860 legacy=0x7f92ec004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.496+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.171:6789/0 conn(0x7f92ec002860 legacy=0x7f92ec004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.502+0000 7f930e882640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:21.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.502+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92ec009e40 con 0x7f92ec009100 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287234182 0 0) 0x7f92ec009e40 con 0x7f92ec009100 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient(hunting): my global_id is 5689 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92ec0065c0 con 0x7f92ec009100 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2894961987 0 0) 0x7f92ec0065c0 con 0x7f92ec009100 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92ec009e40 con 0x7f92ec009100 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92f8003d00 con 0x7f92ec009100 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient: dump: 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.533 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3411753899 0 0) 0x7f92ec009e40 con 0x7f92ec009100 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92ec00c3d0 con 0x7f92ec009100 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 2282901047) 0x7f92f8005b50 con 0x7f92ec009100 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.171:6789/0 conn(0x7f92ec009100 legacy=0x7f92ec009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f93057fa640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.171:6789/0 conn(0x7f92ec009100 legacy=0x7f92ec009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.534 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 22, 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 245541, 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:10:21.535 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.536 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.537 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:21.538 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:21.539 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:21.540 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.541 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.542 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.543 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:21.544 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:21.545 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:21.546 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.547 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:21.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:21.549 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:21.549 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:21.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.171:6812/7327273 conn(0x7f92e8039130 legacy=0x7f92e803b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f930e882640 10 monclient: shutdown 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f930e882640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 >> v1:172.21.15.148:6790/0 conn(0x7f93080a2410 legacy=0x7f930809d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.503+0000 7f930ce1f640 1 -- 172.21.15.148:0/2601624341 reap_dead start 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.504+0000 7f930ce1f640 1 -- 172.21.15.148:0/2601624341 reap_dead start 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.504+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 shutdown_connections 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.504+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 >> 172.21.15.148:0/2601624341 conn(0x7f93080a97b0 msgr2=0x7f9308135880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.504+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 shutdown_connections 2024-09-16T04:10:21.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.504+0000 7f930e882640 1 -- 172.21.15.148:0/2601624341 wait complete. 2024-09-16T04:10:21.552 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:21.552 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:10:21.552 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:10:21.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.648+0000 7f98e9f5f640 10 monclient: get_monmap_and_config 2024-09-16T04:10:21.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.648+0000 7f98e9f5f640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.648+0000 7f98e9f5f640 10 monclient: monmap: 2024-09-16T04:10:21.650 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.650 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:21.649457+0000 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:21.649457+0000 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:21.651 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 1 Processor -- start 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 1 -- start start 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient: init 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient(hunting): picked mon.noname-a con 0x7f98e40a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient(hunting): picked mon.noname-c con 0x7f98e409b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient(hunting): picked mon.noname-b con 0x7f98e409b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e409e040 con 0x7f98e40a0e90 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e409d660 con 0x7f98e409b170 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e409c0d0 con 0x7f98e409b5c0 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e9f5f640 10 monclient(hunting): authenticate will time out at 1042.274783s 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e37fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f98e40a0e90 0x7f98e409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48708/0 (socket says 172.21.15.148:48708) 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.649+0000 7f98e37fe640 1 -- 172.21.15.148:0/2336062949 learned_addr learned my addr 172.21.15.148:0/2336062949 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1529775308 0 0) 0x7f98e409c0d0 con 0x7f98e409b5c0 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): my global_id is 5723 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98cc003440 con 0x7f98e409b5c0 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2177381673 0 0) 0x7f98d4001040 con 0x7f98e40a0e90 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): my global_id is 64128 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e409c0d0 con 0x7f98e40a0e90 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e37fe640 1 -- 172.21.15.148:0/2336062949 >> v1:172.21.15.148:6789/0 conn(0x7f98e40a0e90 legacy=0x7f98e409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T04:10:21.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e37fe640 1 -- 172.21.15.148:0/2336062949 >> v1:172.21.15.148:6789/0 conn(0x7f98e40a0e90 legacy=0x7f98e409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e37fe640 1 --1- 172.21.15.148:0/2336062949 >> v1:172.21.15.148:6789/0 conn(0x7f98e40a0e90 0x7f98e409aa60 :-1 s=OPENED pgs=51 cs=1 l=1).handle_message read tag failed 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e37fe640 1 --1- 172.21.15.148:0/2336062949 >> v1:172.21.15.148:6789/0 conn(0x7f98e40a0e90 0x7f98e409aa60 :-1 s=OPENED pgs=51 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3571054185 0 0) 0x7f98d8001110 con 0x7f98e409b170 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient(hunting): my global_id is 5692 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98d4001040 con 0x7f98e409b170 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 496864073 0 0) 0x7f98cc003440 con 0x7f98e409b5c0 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 >> v1:172.21.15.171:6789/0 conn(0x7f98e409b170 legacy=0x7f98e41383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 >> v1:172.21.15.148:6789/0 conn(0x7f98e40a0e90 legacy=0x7f98e409aa60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 monclient: found mon.noname-b 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e409d380 con 0x7f98e409b5c0 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f98dc001990 con 0x7f98e409b5c0 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 5 monclient: authenticate success, global_id 5723 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: dump: 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:21.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98dc001bf0 con 0x7f98e409b5c0 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 4 set_mon_vals no callback set 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 1 -- 172.21.15.148:0/2336062949 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f98dc001f40 con 0x7f98e409b5c0 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e27fc640 10 monclient: dump: 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 10 monclient: shutdown 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.650+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2336062949 >> v1:172.21.15.148:6790/0 conn(0x7f98e409b5c0 legacy=0x7f98e413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2336062949 shutdown_connections 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2336062949 >> 172.21.15.148:0/2336062949 conn(0x7f98e40a5640 msgr2=0x7f98e40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2336062949 shutdown_connections 2024-09-16T04:10:21.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2336062949 wait complete. 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 10 monclient: build_initial_monmap 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 10 monclient: monmap: 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:21.668 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:21.669 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.651+0000 7f98e9f5f640 1 Processor -- start 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 1 -- start start 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient: init 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient(hunting): picked mon.noname-a con 0x7f98e409b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient(hunting): picked mon.noname-c con 0x7f98e409b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient(hunting): picked mon.noname-b con 0x7f98e40a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:21.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e41c1b30 con 0x7f98e409b170 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e41a6310 con 0x7f98e40a0e90 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e41a8d80 con 0x7f98e409b5c0 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e9f5f640 10 monclient(hunting): authenticate will time out at 1042.277583s 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e2ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f98e409b5c0 0x7f98e41cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:45302/0 (socket says 172.21.15.148:45302) 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98e2ffd640 1 -- 172.21.15.148:0/2104981754 learned_addr learned my addr 172.21.15.148:0/2104981754 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433776156 0 0) 0x7f98e41a8d80 con 0x7f98e409b5c0 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): my global_id is 5726 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98d0003520 con 0x7f98e409b5c0 2024-09-16T04:10:21.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2837797127 0 0) 0x7f98e41c1b30 con 0x7f98e409b170 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 10 monclient(hunting): my global_id is 64131 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.652+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e41a8d80 con 0x7f98e409b170 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741055208 0 0) 0x7f98e41a6310 con 0x7f98e40a0e90 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): my global_id is 5695 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98e41c1b30 con 0x7f98e40a0e90 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1437018026 0 0) 0x7f98d0003520 con 0x7f98e409b5c0 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98e41a6310 con 0x7f98e409b5c0 2024-09-16T04:10:21.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f98d8001a40 con 0x7f98e409b5c0 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): dump: 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.673 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3394202565 0 0) 0x7f98e41a8d80 con 0x7f98e409b170 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98d0003520 con 0x7f98e409b170 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f98d40019f0 con 0x7f98e409b170 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient(hunting): dump: 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.674 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 556083309 0 0) 0x7f98e41a6310 con 0x7f98e409b5c0 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 >> v1:172.21.15.171:6789/0 conn(0x7f98e40a0e90 legacy=0x7f98e41cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 >> v1:172.21.15.148:6789/0 conn(0x7f98e409b170 legacy=0x7f98e41c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 monclient: found mon.c 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e41a6ab0 con 0x7f98e409b5c0 2024-09-16T04:10:21.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 5 monclient: authenticate success, global_id 5726 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 10 monclient: _renew_subs 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f98e41a8aa0 con 0x7f98e409b5c0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 10 monclient: _renew_subs 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f98e41c3e90 con 0x7f98e409b5c0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98d8001d00 con 0x7f98e409b5c0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f98d8003a60 con 0x7f98e409b5c0 2024-09-16T04:10:21.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98e1ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.653+0000 7f98cbfff640 10 monclient: dump: 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:21.677 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f98d8003a60 con 0x7f98e409b5c0 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98e9f5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98e9f5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98e9f5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c4005080 con 0x7f98e409b5c0 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(142..142 src has 1..142) ==== 7799+0+0 (unknown 2936103556 0 0) 0x7f98d8047af0 con 0x7f98e409b5c0 2024-09-16T04:10:21.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.654+0000 7f98e37fe640 10 monclient: get_auth_request con 0x7f98d0038e50 auth_method 0 2024-09-16T04:10:21.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.659+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f98e409d660 con 0x7f98e409b5c0 2024-09-16T04:10:21.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.659+0000 7f98cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:21.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.659+0000 7f98cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:21.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.839+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v143) ==== 113+0+0 (unknown 2670999652 0 0) 0x7f5cd801e320 con 0x7f5cf0137140 2024-09-16T04:10:21.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.839+0000 7f5cecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T04:10:21.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.839+0000 7f5cecff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.896+0000 7f98e9f5f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:10:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.896+0000 7f98e9f5f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:10:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.896+0000 7f98e9f5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:21.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.896+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f98c4005b80 con 0x7f98e409b5c0 2024-09-16T04:10:21.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f98cbfff640 1 -- 172.21.15.148:0/2104981754 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f98d8015a60 con 0x7f98e409b5c0 2024-09-16T04:10:21.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f98cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:10:21.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f98cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:21.902 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.902 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=1493876593,config=692619927,health=3204684937,kv=1777156452,logm=1565979533}) 2024-09-16T04:10:21.902 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=1,health=12,kv=10,logm=64} crc {auth=1493876593,config=1228700967,health=2440873570,kv=2270513034,logm=299607463}) 2024-09-16T04:10:21.902 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.902 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=1493876593,config=692619927,health=3204684937,kv=1777156452,logm=1565979533}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.897+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=1,health=12,kv=10,logm=64} crc {auth=1493876593,config=1228700967,health=862539094,kv=2270513034,logm=874252516}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.898+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.898+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135870502}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.898+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1172548191}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.898+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.898+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135870502}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.898+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834546469}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026390146}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3333331982}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026390146}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822965119}) 2024-09-16T04:10:21.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.904 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020629002}) 2024-09-16T04:10:21.904 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1443773240}) 2024-09-16T04:10:21.904 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.904 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020629002}) 2024-09-16T04:10:21.904 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1482621934}) 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 >> v1:172.21.15.171:6812/7327273 conn(0x7f98d0038e50 legacy=0x7f98d003b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f98e9f5f640 10 monclient: shutdown 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f98e9f5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.899+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 >> v1:172.21.15.148:6790/0 conn(0x7f98e409b5c0 legacy=0x7f98e41cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 shutdown_connections 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 >> 172.21.15.148:0/2104981754 conn(0x7f98e40a5640 msgr2=0x7f98e40a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 shutdown_connections 2024-09-16T04:10:21.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f98e9f5f640 1 -- 172.21.15.148:0/2104981754 wait complete. 2024-09-16T04:10:21.908 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.908 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181328194}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275564129}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181328194}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.900+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=305741578}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.901+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.901+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341639959}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.901+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3108012427}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.901+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.901+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341639959}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.901+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834300795}) 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.909 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886737475}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1688148763}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886737475}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1183181030}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621588389}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912871366}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1621588389}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.902+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4163378341}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.903+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.903+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035436034}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.903+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3614614975}) 2024-09-16T04:10:21.910 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.903+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.903+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035436034}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.903+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3148294761}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.904+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.904+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513803491}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.904+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2311600655}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.904+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.904+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513803491}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.904+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2393242566}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.905+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.905+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277551493}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.905+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179331025}) 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.905+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.911 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.905+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277551493}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.905+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552271623}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336981293}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=923609273}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336981293}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601533281}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044598992}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618166457}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2044598992}) 2024-09-16T04:10:21.912 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.906+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608362844}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.907+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.907+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156827837}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.907+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684274115}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.907+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.907+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156827837}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.907+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3411642471}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.908+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.908+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72186412}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.908+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939689624}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.908+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.908+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72186412}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.908+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763932799}) 2024-09-16T04:10:21.915 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957635947}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336984034}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957635947}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468322747}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854522262}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478062207}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854522262}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.909+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251266403}) 2024-09-16T04:10:21.916 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.910+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.917 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.910+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {logm=2169265118,mdsmap=1097804592,mgr=44478376,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=295471672,mon_config_key=2589496441,monmap=3236500405,osd_metadata=2220131780,osd_pg_creating=2680133514,osd_snap=841190635,osdmap=261432947}) 2024-09-16T04:10:21.917 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.910+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=906659010,mdsmap=1097804592,mgr=3517470555,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2907157848,mon_config_key=2589496441,monmap=2633035914,osd_metadata=2220131780,osd_pg_creating=2680133514,osd_snap=841190635,osdmap=3065595400}) 2024-09-16T04:10:21.917 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.910+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.917 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.910+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=49} crc {logm=2169265118,mdsmap=1097804592,mgr=44478376,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=295471672,mon_config_key=2589496441,monmap=3236500405,osd_metadata=2220131780,osd_pg_creating=2680133514,osd_snap=841190635,osdmap=261432947}) 2024-09-16T04:10:21.917 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=906659010,mdsmap=1097804592,mgr=3983781359,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2519793717,mon_config_key=888502937,monmap=2633035914,osd_metadata=308551371,osd_pg_creating=2680133514,osd_snap=841190635,osdmap=1983598445}) 2024-09-16T04:10:21.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:10:21.923 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.923 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1220476095}) 2024-09-16T04:10:21.923 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3661298024}) 2024-09-16T04:10:21.923 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.923 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1220476095}) 2024-09-16T04:10:21.923 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.911+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1532446092}) 2024-09-16T04:10:21.924 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.912+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.924 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.912+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=3824622638}) 2024-09-16T04:10:21.924 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.912+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=71} crc {osdmap=743386553}) 2024-09-16T04:10:21.925 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.912+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:21.925 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.912+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=3824622638}) 2024-09-16T04:10:21.925 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:10:21.912+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=2358734085}) 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.955+0000 7f5cf5a5e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.955+0000 7f5cf5a5e640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.955+0000 7f5cf5a5e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.955+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) -- 0x7f5cd4005ab0 con 0x7f5cf0137140 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.956+0000 7f5cecff9640 1 -- 172.21.15.171:0/2104531454 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v143) ==== 155+0+0 (unknown 514428508 0 0) 0x7f5cd8045d90 con 0x7f5cf0137140 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.956+0000 7f5cecff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-16T04:10:21.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.956+0000 7f5cecff9640 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-16T04:10:21.960 INFO:tasks.workunit.client.0.smithi171.stderr:weight-set for pool 'cold' already created 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 >> v1:172.21.15.171:6812/7327273 conn(0x7f5cd0038f10 legacy=0x7f5cd003b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 10 monclient: shutdown 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 >> v1:172.21.15.171:6789/0 conn(0x7f5cf0137140 legacy=0x7f5cf012e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 shutdown_connections 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 >> 172.21.15.171:0/2104531454 conn(0x7f5cf00a9ae0 msgr2=0x7f5cf0072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:21.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 shutdown_connections 2024-09-16T04:10:21.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:21.958+0000 7f5cf5a5e640 1 -- 172.21.15.171:0/2104531454 wait complete. 2024-09-16T04:10:21.971 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T04:10:22.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.048+0000 7f6a073fd640 10 monclient: get_monmap_and_config 2024-09-16T04:10:22.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.048+0000 7f6a073fd640 10 monclient: build_initial_monmap 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.048+0000 7f6a073fd640 10 monclient: monmap: 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:22.050689+0000 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:22.050689+0000 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:22.052 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.049+0000 7f6a073fd640 1 Processor -- start 2024-09-16T04:10:22.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.049+0000 7f6a073fd640 1 -- start start 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.049+0000 7f6a073fd640 10 monclient: init 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0009f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a000a1260 con 0x7f6a000a0130 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0009d0c0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0009d3a0 con 0x7f6a0009f640 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 10 monclient(hunting): authenticate will time out at 997.556328s 2024-09-16T04:10:22.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a0599a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6a0009e1d0 0x7f6a0013bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39412/0 (socket says 172.21.15.171:39412) 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a0599a640 1 -- 172.21.15.171:0/966224462 learned_addr learned my addr 172.21.15.171:0/966224462 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175257621 0 0) 0x7f6a0009d0c0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): my global_id is 5698 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69ec004630 con 0x7f6a0009e1d0 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917666308 0 0) 0x7f6a000a1260 con 0x7f6a000a0130 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): my global_id is 64134 2024-09-16T04:10:22.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0009d0c0 con 0x7f6a000a0130 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826060195 0 0) 0x7f6a0009d3a0 con 0x7f6a0009f640 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient(hunting): my global_id is 5729 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a000a1260 con 0x7f6a0009f640 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1570402948 0 0) 0x7f6a0009d0c0 con 0x7f6a000a0130 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 >> v1:172.21.15.148:6790/0 conn(0x7f6a0009f640 legacy=0x7f6a0009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 >> v1:172.21.15.171:6789/0 conn(0x7f6a0009e1d0 legacy=0x7f6a0013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:22.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 monclient: found mon.noname-a 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0009d680 con 0x7f6a000a0130 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69f00015f0 con 0x7f6a000a0130 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 5 monclient: authenticate success, global_id 64134 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:22.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f69f7fff640 10 monclient: dump: 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:22.065 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.050+0000 7f6a073fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69f0001860 con 0x7f6a000a0130 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f69f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f69f7fff640 1 -- 172.21.15.171:0/966224462 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69f0001c70 con 0x7f6a000a0130 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 4 set_mon_vals no callback set 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f69f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f69f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f69f7fff640 10 monclient: dump: 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:22.066 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 10 monclient: shutdown 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 1 -- 172.21.15.171:0/966224462 >> v1:172.21.15.148:6789/0 conn(0x7f6a000a0130 legacy=0x7f6a00138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 1 -- 172.21.15.171:0/966224462 shutdown_connections 2024-09-16T04:10:22.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 1 -- 172.21.15.171:0/966224462 >> 172.21.15.171:0/966224462 conn(0x7f6a000a52a0 msgr2=0x7f6a000a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 1 -- 172.21.15.171:0/966224462 shutdown_connections 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.051+0000 7f6a073fd640 1 -- 172.21.15.171:0/966224462 wait complete. 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.052+0000 7f6a073fd640 10 monclient: build_initial_monmap 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.052+0000 7f6a073fd640 10 monclient: monmap: 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:22.068 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.052+0000 7f6a073fd640 1 Processor -- start 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.052+0000 7f6a073fd640 1 -- start start 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.052+0000 7f6a073fd640 10 monclient: init 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0009e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient(hunting): picked mon.noname-b con 0x7f6a000a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a001a56f0 con 0x7f6a0009f640 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a001a3ce0 con 0x7f6a000a0130 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a000a3cb0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient(hunting): authenticate will time out at 997.559217s 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a0599a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6a000a0130 0x7f6a001cf8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39416/0 (socket says 172.21.15.171:39416) 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a0599a640 1 -- 172.21.15.171:0/309680098 learned_addr learned my addr 172.21.15.171:0/309680098 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78331492 0 0) 0x7f6a001a3ce0 con 0x7f6a000a0130 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): my global_id is 5701 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69e4003520 con 0x7f6a000a0130 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162217499 0 0) 0x7f6a000a3cb0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:22.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): my global_id is 5732 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a001a3ce0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128993414 0 0) 0x7f6a001a56f0 con 0x7f6a0009f640 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): my global_id is 64137 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a000a3cb0 con 0x7f6a0009f640 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3821681974 0 0) 0x7f69e4003520 con 0x7f6a000a0130 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a001a56f0 con 0x7f6a000a0130 2024-09-16T04:10:22.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69f8001360 con 0x7f6a000a0130 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:22.072 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 845629248 0 0) 0x7f6a001a3ce0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69e4003520 con 0x7f6a0009e1d0 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3433854578 0 0) 0x7f6a000a3cb0 con 0x7f6a0009f640 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a001a3ce0 con 0x7f6a0009f640 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69fc0013f0 con 0x7f6a0009e1d0 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:22.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69f00035f0 con 0x7f6a0009f640 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:22.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:22.075 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2082893561 0 0) 0x7f6a001a56f0 con 0x7f6a000a0130 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 >> v1:172.21.15.148:6790/0 conn(0x7f6a0009e1d0 legacy=0x7f6a001c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 >> v1:172.21.15.148:6789/0 conn(0x7f6a0009f640 legacy=0x7f6a001cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 monclient: found mon.b 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0006ceb0 con 0x7f6a000a0130 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:22.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f69f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 5 monclient: authenticate success, global_id 5701 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient: _renew_subs 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.053+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a001a3400 con 0x7f6a000a0130 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69f8001650 con 0x7f6a000a0130 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69f8004c30 con 0x7f6a000a0130 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f5ffb640 10 monclient: dump: 2024-09-16T04:10:22.077 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f6a073fd640 10 monclient: _renew_subs 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f69f77fe640 4 set_mon_vals no callback set 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f6a073fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.054+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a001a4180 con 0x7f6a000a0130 2024-09-16T04:10:22.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f69f800fec0 con 0x7f6a000a0130 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f6a073fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f6a073fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f6a073fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a000008d0 con 0x7f6a000a0130 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(143..143 src has 1..143) ==== 8271+0+0 (unknown 3233160924 0 0) 0x7f69f8047f10 con 0x7f6a000a0130 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.055+0000 7f6a05199640 10 monclient: get_auth_request con 0x7f69e4039210 auth_method 0 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.059+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f69f8015b80 con 0x7f6a000a0130 2024-09-16T04:10:22.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.060+0000 7f69f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:22.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.060+0000 7f69f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:22.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.227+0000 7f6a073fd640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:22.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.227+0000 7f6a073fd640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:22.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.227+0000 7f6a073fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:22.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.227+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f6a001a2770 con 0x7f6a000a0130 2024-09-16T04:10:22.896 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.893+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v144) ==== 123+0+0 (unknown 3808716133 0 0) 0x7f69f8015480 con 0x7f6a000a0130 2024-09-16T04:10:22.896 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.893+0000 7f69f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:22.896 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.893+0000 7f69f5ffb640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T04:10:22.922 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:10:22.922 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:23.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.999+0000 7f6a073fd640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:23.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.999+0000 7f6a073fd640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:23.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.999+0000 7f6a073fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:23.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:22.999+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f6a001a29e0 con 0x7f6a000a0130 2024-09-16T04:10:23.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.024+0000 7f2eeb791640 10 monclient: get_monmap_and_config 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.024+0000 7f2eeb791640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.025+0000 7f2eeb791640 10 monclient: monmap: 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:23.025929+0000 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:23.025929+0000 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:23.027 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.025+0000 7f2eeb791640 1 Processor -- start 2024-09-16T04:10:23.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.025+0000 7f2eeb791640 1 -- start start 2024-09-16T04:10:23.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.025+0000 7f2eeb791640 10 monclient: init 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee40733a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee4073780 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee40a3410 con 0x7f2ee413d010 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee40a3190 con 0x7f2ee40733a0 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee409e4b0 con 0x7f2ee4073780 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2eeb791640 10 monclient(hunting): authenticate will time out at 1043.651397s 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2ee9d2e640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2ee413d010 0x7f2ee413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46466/0 (socket says 172.21.15.148:46466) 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2ee9d2e640 1 -- 172.21.15.148:0/2407183125 learned_addr learned my addr 172.21.15.148:0/2407183125 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2ee8d2c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2ee4073780 0x7f2ee413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:42834/0 (socket says 172.21.15.148:42834) 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741002666 0 0) 0x7f2ee40a3410 con 0x7f2ee413d010 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): my global_id is 64140 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ed0003520 con 0x7f2ee413d010 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177920785 0 0) 0x7f2ee409e4b0 con 0x7f2ee4073780 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): my global_id is 5735 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ee40a3410 con 0x7f2ee4073780 2024-09-16T04:10:23.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603029134 0 0) 0x7f2ee40a3190 con 0x7f2ee40733a0 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient(hunting): my global_id is 5704 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ee409e4b0 con 0x7f2ee40733a0 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2357177449 0 0) 0x7f2ed0003520 con 0x7f2ee413d010 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 >> v1:172.21.15.148:6790/0 conn(0x7f2ee4073780 legacy=0x7f2ee413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 >> v1:172.21.15.171:6789/0 conn(0x7f2ee40733a0 legacy=0x7f2ee409ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.026+0000 7f2edbfff640 1 monclient: found mon.noname-a 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee406e700 con 0x7f2ee413d010 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2edc001990 con 0x7f2ee413d010 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 5 monclient: authenticate success, global_id 64140 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: dump: 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.038 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2edc001c40 con 0x7f2ee413d010 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 1 -- 172.21.15.148:0/2407183125 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2edc002010 con 0x7f2ee413d010 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 4 set_mon_vals no callback set 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2edbfff640 10 monclient: dump: 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 10 monclient: shutdown 2024-09-16T04:10:23.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 1 -- 172.21.15.148:0/2407183125 >> v1:172.21.15.148:6789/0 conn(0x7f2ee413d010 legacy=0x7f2ee413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 1 -- 172.21.15.148:0/2407183125 shutdown_connections 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 1 -- 172.21.15.148:0/2407183125 >> 172.21.15.148:0/2407183125 conn(0x7f2ee40a78b0 msgr2=0x7f2ee40a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 1 -- 172.21.15.148:0/2407183125 shutdown_connections 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.027+0000 7f2eeb791640 1 -- 172.21.15.148:0/2407183125 wait complete. 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.028+0000 7f2eeb791640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.028+0000 7f2eeb791640 10 monclient: monmap: 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.044 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.045 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.045 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:23.045 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:23.045 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.028+0000 7f2eeb791640 1 Processor -- start 2024-09-16T04:10:23.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.028+0000 7f2eeb791640 1 -- start start 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.028+0000 7f2eeb791640 10 monclient: init 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee40733a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee4073780 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee413d010 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee4140740 con 0x7f2ee413d010 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee40741d0 con 0x7f2ee4073780 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee4122590 con 0x7f2ee40733a0 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ee952d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2ee40733a0 0x7f2ee4121770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:42848/0 (socket says 172.21.15.148:42848) 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ee952d640 1 -- 172.21.15.148:0/2694746224 learned_addr learned my addr 172.21.15.148:0/2694746224 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2eeb791640 10 monclient(hunting): authenticate will time out at 1043.654303s 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341939556 0 0) 0x7f2ee4122590 con 0x7f2ee40733a0 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): my global_id is 5738 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec8003520 con 0x7f2ee40733a0 2024-09-16T04:10:23.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 821624622 0 0) 0x7f2ee406e700 con 0x7f2ee413d010 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): my global_id is 64143 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ee4122590 con 0x7f2ee413d010 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3497921659 0 0) 0x7f2ee40a3410 con 0x7f2ee4073780 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): my global_id is 5707 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ee406e700 con 0x7f2ee4073780 2024-09-16T04:10:23.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2588876148 0 0) 0x7f2ec8003520 con 0x7f2ee40733a0 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ee40a3410 con 0x7f2ee40733a0 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2ee00018f0 con 0x7f2ee40733a0 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.049 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1205178332 0 0) 0x7f2ee4122590 con 0x7f2ee413d010 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ec8003520 con 0x7f2ee413d010 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2edc001df0 con 0x7f2ee413d010 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.050 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 992852318 0 0) 0x7f2ee406e700 con 0x7f2ee4073780 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ee4122590 con 0x7f2ee4073780 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2ed40018b0 con 0x7f2ee4073780 2024-09-16T04:10:23.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:23.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 10 monclient(hunting): dump: 2024-09-16T04:10:23.052 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.052 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.052 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.052 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.053 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.053 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.053 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.053 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.053 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.053 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3850687551 0 0) 0x7f2ee40a3410 con 0x7f2ee40733a0 2024-09-16T04:10:23.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.029+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 >> v1:172.21.15.171:6789/0 conn(0x7f2ee4073780 legacy=0x7f2ee4121e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 >> v1:172.21.15.148:6789/0 conn(0x7f2ee413d010 legacy=0x7f2ee41d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 1 monclient: found mon.c 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee40a4dc0 con 0x7f2ee40733a0 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 5 monclient: authenticate success, global_id 5738 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 10 monclient: _renew_subs 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee4122e00 con 0x7f2ee40733a0 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ee0001ce0 con 0x7f2ee40733a0 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2ee0003a60 con 0x7f2ee40733a0 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 10 monclient: dump: 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.056 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 10 monclient: _renew_subs 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ee4123cf0 con 0x7f2ee40733a0 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2edb7fe640 4 set_mon_vals no callback set 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.030+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2ee0003a60 con 0x7f2ee40733a0 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.031+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f2ee0047e50 con 0x7f2ee40733a0 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.031+0000 7f2eeb791640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.031+0000 7f2eeb791640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.031+0000 7f2eeb791640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.031+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee4124370 con 0x7f2ee40733a0 2024-09-16T04:10:23.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.031+0000 7f2ee8d2c640 10 monclient: get_auth_request con 0x7f2ec80391a0 auth_method 0 2024-09-16T04:10:23.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.035+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2ee0015b40 con 0x7f2ee40733a0 2024-09-16T04:10:23.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.035+0000 7f2ed9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.035+0000 7f2ed9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:23.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2eeb791640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:23.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2eeb791640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:23.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2eeb791640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ee4062720 con 0x7f2ee40733a0 2024-09-16T04:10:23.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2ed9ffb640 1 -- 172.21.15.148:0/2694746224 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3460517252) 0x7f2ee0015440 con 0x7f2ee40733a0 2024-09-16T04:10:23.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2ed9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:23.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.262+0000 7f2ed9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:23.265 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:23.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 >> v1:172.21.15.171:6812/7327273 conn(0x7f2ec80391a0 legacy=0x7f2ec803b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 10 monclient: shutdown 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 >> v1:172.21.15.148:6790/0 conn(0x7f2ee40733a0 legacy=0x7f2ee4121770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 shutdown_connections 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 >> 172.21.15.148:0/2694746224 conn(0x7f2ee40a78b0 msgr2=0x7f2ee409ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 shutdown_connections 2024-09-16T04:10:23.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.264+0000 7f2eeb791640 1 -- 172.21.15.148:0/2694746224 wait complete. 2024-09-16T04:10:23.277 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:10:23.277 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:10:23.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:10:23.277 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:23.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.374+0000 7f5736f49640 10 monclient: get_monmap_and_config 2024-09-16T04:10:23.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.374+0000 7f5736f49640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.374+0000 7f5736f49640 10 monclient: monmap: 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:23.375636+0000 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:23.375636+0000 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:23.377 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 1 Processor -- start 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 1 -- start start 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient: init 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient(hunting): picked mon.noname-b con 0x7f573009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient(hunting): picked mon.noname-c con 0x7f573009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient(hunting): picked mon.noname-a con 0x7f573009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57300a1010 con 0x7f573009cc30 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f573009f030 con 0x7f573009c660 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f573009dc60 con 0x7f573009bb70 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.375+0000 7f5736f49640 10 monclient(hunting): authenticate will time out at 1044.000935s 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f5734ce5640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f573009bb70 0x7f573009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:42864/0 (socket says 172.21.15.148:42864) 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f5734ce5640 1 -- 172.21.15.148:0/3425467761 learned_addr learned my addr 172.21.15.148:0/3425467761 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f57354e6640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f573009cc30 0x7f573013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46490/0 (socket says 172.21.15.148:46490) 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456924233 0 0) 0x7f57300a1010 con 0x7f573009cc30 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): my global_id is 64146 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5720003520 con 0x7f573009cc30 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046839009 0 0) 0x7f573009dc60 con 0x7f573009bb70 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): my global_id is 5741 2024-09-16T04:10:23.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57300a1010 con 0x7f573009bb70 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104458085 0 0) 0x7f573009f030 con 0x7f573009c660 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient(hunting): my global_id is 5710 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f573009dc60 con 0x7f573009c660 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3058177097 0 0) 0x7f5720003520 con 0x7f573009cc30 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 >> v1:172.21.15.148:6790/0 conn(0x7f573009bb70 legacy=0x7f573009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 >> v1:172.21.15.171:6789/0 conn(0x7f573009c660 legacy=0x7f5730138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573009df40 con 0x7f573009cc30 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f5736f49640 5 monclient: authenticate success, global_id 64146 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f57280018c0 con 0x7f573009cc30 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: dump: 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.390 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f5736f49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5728001af0 con 0x7f573009cc30 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 1 -- 172.21.15.148:0/3425467761 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5728001f00 con 0x7f573009cc30 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f5736f49640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f5736f49640 4 set_mon_vals no callback set 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.376+0000 7f572f7fe640 10 monclient: dump: 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 10 monclient: shutdown 2024-09-16T04:10:23.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 1 -- 172.21.15.148:0/3425467761 >> v1:172.21.15.148:6789/0 conn(0x7f573009cc30 legacy=0x7f573013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 1 -- 172.21.15.148:0/3425467761 shutdown_connections 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 1 -- 172.21.15.148:0/3425467761 >> 172.21.15.148:0/3425467761 conn(0x7f57300a5640 msgr2=0x7f57300a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 1 -- 172.21.15.148:0/3425467761 shutdown_connections 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 1 -- 172.21.15.148:0/3425467761 wait complete. 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.377+0000 7f5736f49640 10 monclient: monmap: 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.393 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 1 Processor -- start 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 1 -- start start 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient: init 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient(hunting): picked mon.noname-c con 0x7f573009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient(hunting): picked mon.noname-b con 0x7f573009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient(hunting): picked mon.noname-a con 0x7f573009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5730132390 con 0x7f573009cc30 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5730122bc0 con 0x7f573009c660 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5730124eb0 con 0x7f573009bb70 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5736f49640 10 monclient(hunting): authenticate will time out at 1044.003738s 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5734ce5640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f573009bb70 0x7f5730131490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:42866/0 (socket says 172.21.15.148:42866) 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f5734ce5640 1 -- 172.21.15.148:0/2569856719 learned_addr learned my addr 172.21.15.148:0/2569856719 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.378+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232957407 0 0) 0x7f5730124eb0 con 0x7f573009bb70 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): my global_id is 5744 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5714003520 con 0x7f573009bb70 2024-09-16T04:10:23.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728271284 0 0) 0x7f5730132390 con 0x7f573009cc30 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): my global_id is 64149 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5730124eb0 con 0x7f573009cc30 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 216033544 0 0) 0x7f5730122bc0 con 0x7f573009c660 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): my global_id is 5713 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5730132390 con 0x7f573009c660 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1153567841 0 0) 0x7f5714003520 con 0x7f573009bb70 2024-09-16T04:10:23.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5730122bc0 con 0x7f573009bb70 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f57240012b0 con 0x7f573009bb70 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): dump: 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.397 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1876843836 0 0) 0x7f5730124eb0 con 0x7f573009cc30 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5714003520 con 0x7f573009cc30 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5728001e40 con 0x7f573009cc30 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient(hunting): dump: 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.398 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 436590393 0 0) 0x7f5730132390 con 0x7f573009c660 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5730124eb0 con 0x7f573009c660 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2967391715 0 0) 0x7f5730122bc0 con 0x7f573009bb70 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.171:6789/0 conn(0x7f573009c660 legacy=0x7f57301d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.148:6789/0 conn(0x7f573009cc30 legacy=0x7f57301d3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 monclient: found mon.c 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5730124610 con 0x7f573009bb70 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 5 monclient: authenticate success, global_id 5744 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 10 monclient: _renew_subs 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5730131ba0 con 0x7f573009bb70 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 10 monclient: _renew_subs 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5730123900 con 0x7f573009bb70 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5724001680 con 0x7f573009bb70 2024-09-16T04:10:23.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5724004c30 con 0x7f573009bb70 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572effd640 4 set_mon_vals no callback set 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.379+0000 7f572d7fa640 10 monclient: dump: 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.401 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f5736f49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f57100065c0 con 0x7f5710002860 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f572400fec0 con 0x7f573009bb70 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f5724047ed0 con 0x7f573009bb70 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791781320 0 0) 0x7f57100065c0 con 0x7f5710002860 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient(hunting): my global_id is 64152 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5730122bc0 con 0x7f5710002860 2024-09-16T04:10:23.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f57354e6640 10 monclient: get_auth_request con 0x7f5714039030 auth_method 0 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3666433412 0 0) 0x7f5730122bc0 con 0x7f5710002860 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57100065c0 con 0x7f5710002860 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f571c003790 con 0x7f5710002860 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.380+0000 7f572d7fa640 10 monclient: dump: 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.403 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 42029334 0 0) 0x7f57100065c0 con 0x7f5710002860 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f571000a150 con 0x7f5710002860 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f571c005a10 con 0x7f5710002860 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.148:6789/0 conn(0x7f5710002860 legacy=0x7f5710004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.381+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.148:6789/0 conn(0x7f5710002860 legacy=0x7f5710004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f5736f49640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:23.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5710009e50 con 0x7f5710009110 2024-09-16T04:10:23.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286822000 0 0) 0x7f5710009e50 con 0x7f5710009110 2024-09-16T04:10:23.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient(hunting): my global_id is 64155 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57100065c0 con 0x7f5710009110 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2370691031 0 0) 0x7f57100065c0 con 0x7f5710009110 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5710009e50 con 0x7f5710009110 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f571c005120 con 0x7f5710009110 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 10 monclient: dump: 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.407 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2507961457 0 0) 0x7f5710009e50 con 0x7f5710009110 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.387+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f571000c3d0 con 0x7f5710009110 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3977 (unknown 0 0 2393815680) 0x7f571c004f40 con 0x7f5710009110 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f572d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f572d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.148:6789/0 conn(0x7f5710009110 legacy=0x7f57100095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f572d7fa640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.148:6789/0 conn(0x7f5710009110 legacy=0x7f57100095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 22, 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 2586, 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:10:23.409 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.410 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.411 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:23.412 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:23.413 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.414 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:23.415 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:23.416 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.418 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.419 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:23.420 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:23.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.171:6812/7327273 conn(0x7f5714039030 legacy=0x7f571403b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 10 monclient: shutdown 2024-09-16T04:10:23.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 >> v1:172.21.15.148:6790/0 conn(0x7f573009bb70 legacy=0x7f5730131490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f57354e6640 1 -- 172.21.15.148:0/2569856719 reap_dead start 2024-09-16T04:10:23.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f57354e6640 1 -- 172.21.15.148:0/2569856719 reap_dead start 2024-09-16T04:10:23.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 shutdown_connections 2024-09-16T04:10:23.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 >> 172.21.15.148:0/2569856719 conn(0x7f57300a5640 msgr2=0x7f57300a29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 shutdown_connections 2024-09-16T04:10:23.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.388+0000 7f5736f49640 1 -- 172.21.15.148:0/2569856719 wait complete. 2024-09-16T04:10:23.423 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:23.423 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:10:23.423 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:10:23.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.519+0000 7f479910a640 10 monclient: get_monmap_and_config 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.519+0000 7f479910a640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.519+0000 7f479910a640 10 monclient: monmap: 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:23.520832+0000 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:23.520832+0000 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:23.522 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 1 Processor -- start 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 1 -- start start 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 10 monclient: init 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 10 monclient(hunting): picked mon.noname-a con 0x7f47940a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 10 monclient(hunting): picked mon.noname-b con 0x7f479409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.520+0000 7f479910a640 10 monclient(hunting): picked mon.noname-c con 0x7f479409b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f479409e040 con 0x7f47940a0e90 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f479409d660 con 0x7f479409b5c0 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f479409c0d0 con 0x7f479409b170 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 10 monclient(hunting): authenticate will time out at 1044.146037s 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4792d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f47940a0e90 0x7f479409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46522/0 (socket says 172.21.15.148:46522) 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4792d9d640 1 -- 172.21.15.148:0/4073410148 learned_addr learned my addr 172.21.15.148:0/4073410148 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479259c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f479409b170 0x7f47941383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:42882/0 (socket says 172.21.15.148:42882) 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 295454328 0 0) 0x7f4784001490 con 0x7f47940a0e90 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): my global_id is 64158 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f477c004630 con 0x7f47940a0e90 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268652421 0 0) 0x7f479409c0d0 con 0x7f479409b170 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): my global_id is 5747 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4784001490 con 0x7f479409b170 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419064894 0 0) 0x7f478c001110 con 0x7f479409b5c0 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient(hunting): my global_id is 5716 2024-09-16T04:10:23.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f479409c0d0 con 0x7f479409b5c0 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 856700076 0 0) 0x7f477c004630 con 0x7f47940a0e90 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 >> v1:172.21.15.148:6790/0 conn(0x7f479409b170 legacy=0x7f47941383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 >> v1:172.21.15.171:6789/0 conn(0x7f479409b5c0 legacy=0x7f479413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 monclient: found mon.noname-a 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479409d380 con 0x7f47940a0e90 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4784001b60 con 0x7f47940a0e90 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 5 monclient: authenticate success, global_id 64158 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: dump: 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.535 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4784001e10 con 0x7f47940a0e90 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 1 -- 172.21.15.148:0/4073410148 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4784004c20 con 0x7f47940a0e90 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f479910a640 4 set_mon_vals no callback set 2024-09-16T04:10:23.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.521+0000 7f4791d9b640 10 monclient: dump: 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 10 monclient: shutdown 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 1 -- 172.21.15.148:0/4073410148 >> v1:172.21.15.148:6789/0 conn(0x7f47940a0e90 legacy=0x7f479409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 1 -- 172.21.15.148:0/4073410148 shutdown_connections 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 1 -- 172.21.15.148:0/4073410148 >> 172.21.15.148:0/4073410148 conn(0x7f47940a5640 msgr2=0x7f47940a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 1 -- 172.21.15.148:0/4073410148 shutdown_connections 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.522+0000 7f479910a640 1 -- 172.21.15.148:0/4073410148 wait complete. 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient: monmap: 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:23.538 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 1 Processor -- start 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 1 -- start start 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient: init 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient(hunting): picked mon.noname-a con 0x7f479409b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient(hunting): picked mon.noname-c con 0x7f479409b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 10 monclient(hunting): picked mon.noname-b con 0x7f47940a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.523+0000 7f479910a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47941c3b60 con 0x7f479409b170 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47941aadb0 con 0x7f47940a0e90 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47941a8ac0 con 0x7f479409b5c0 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 10 monclient(hunting): authenticate will time out at 1044.149040s 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f4792d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f479409b170 0x7f47941c3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46534/0 (socket says 172.21.15.148:46534) 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f4792d9d640 1 -- 172.21.15.148:0/1671657214 learned_addr learned my addr 172.21.15.148:0/1671657214 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090948930 0 0) 0x7f47941c3b60 con 0x7f479409b170 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): my global_id is 64161 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4780003520 con 0x7f479409b170 2024-09-16T04:10:23.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257503157 0 0) 0x7f47941a8ac0 con 0x7f479409b5c0 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): my global_id is 5750 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47941c3b60 con 0x7f479409b5c0 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181101410 0 0) 0x7f47941aadb0 con 0x7f47940a0e90 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): my global_id is 5719 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47941a8ac0 con 0x7f47940a0e90 2024-09-16T04:10:23.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3091683248 0 0) 0x7f47941c3b60 con 0x7f479409b5c0 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47941aadb0 con 0x7f479409b5c0 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3985577772 0 0) 0x7f4780003520 con 0x7f479409b170 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47941c3b60 con 0x7f479409b170 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4788001250 con 0x7f479409b5c0 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.542 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4784001980 con 0x7f479409b170 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.543 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2731269604 0 0) 0x7f47941a8ac0 con 0x7f47940a0e90 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4780003520 con 0x7f47940a0e90 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f478c001a80 con 0x7f47940a0e90 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.544 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 77322600 0 0) 0x7f47941aadb0 con 0x7f479409b5c0 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.171:6789/0 conn(0x7f47940a0e90 legacy=0x7f47941d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.148:6789/0 conn(0x7f479409b170 legacy=0x7f47941c3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 monclient: found mon.c 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47941aaad0 con 0x7f479409b5c0 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 5 monclient: authenticate success, global_id 5750 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 10 monclient: _renew_subs 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47941a9960 con 0x7f479409b5c0 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 10 monclient: _renew_subs 2024-09-16T04:10:23.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f47941c9440 con 0x7f479409b5c0 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f47880047e0 con 0x7f479409b5c0 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4788004b30 con 0x7f479409b5c0 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f479159a640 4 set_mon_vals no callback set 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.524+0000 7f477b7fe640 10 monclient: dump: 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.547 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.525+0000 7f479910a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.525+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f47740065c0 con 0x7f4774002860 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.525+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f478800fdc0 con 0x7f479409b5c0 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f4788047ca0 con 0x7f479409b5c0 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330693805 0 0) 0x7f47740065c0 con 0x7f4774002860 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.548 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient(hunting): my global_id is 5753 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47941aadb0 con 0x7f4774002860 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f479359e640 10 monclient: get_auth_request con 0x7f47800391f0 auth_method 0 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2862701253 0 0) 0x7f47941aadb0 con 0x7f4774002860 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47740065c0 con 0x7f4774002860 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4784001e10 con 0x7f4774002860 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient: dump: 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.549 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1092798568 0 0) 0x7f47740065c0 con 0x7f4774002860 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f477400a150 con 0x7f4774002860 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f47840070a0 con 0x7f4774002860 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.148:6790/0 conn(0x7f4774002860 legacy=0x7f4774004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.526+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.148:6790/0 conn(0x7f4774002860 legacy=0x7f4774004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.532+0000 7f479910a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:23.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.532+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4774009e70 con 0x7f4774009130 2024-09-16T04:10:23.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679165567 0 0) 0x7f4774009e70 con 0x7f4774009130 2024-09-16T04:10:23.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient(hunting): my global_id is 5756 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47740065c0 con 0x7f4774009130 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 662972344 0 0) 0x7f47740065c0 con 0x7f4774009130 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4774009e70 con 0x7f4774009130 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4784005550 con 0x7f4774009130 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient: dump: 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.553 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 292361796 0 0) 0x7f4774009e70 con 0x7f4774009130 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f477400c3f0 con 0x7f4774009130 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 1551586025) 0x7f47840068c0 con 0x7f4774009130 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.148:6790/0 conn(0x7f4774009130 legacy=0x7f47740095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f477b7fe640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.148:6790/0 conn(0x7f4774009130 legacy=0x7f47740095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 22, 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 247611, 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:10:23.555 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.556 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.557 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:23.558 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.559 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:23.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:23.561 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.562 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.563 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.564 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.565 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:23.566 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.533+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.171:6812/7327273 conn(0x7f47800391f0 legacy=0x7f478003b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 10 monclient: shutdown 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 >> v1:172.21.15.148:6790/0 conn(0x7f479409b5c0 legacy=0x7f47941ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479359e640 1 -- 172.21.15.148:0/1671657214 reap_dead start 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479359e640 1 -- 172.21.15.148:0/1671657214 reap_dead start 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 shutdown_connections 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 >> 172.21.15.148:0/1671657214 conn(0x7f47940a5640 msgr2=0x7f4794071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 shutdown_connections 2024-09-16T04:10:23.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.534+0000 7f479910a640 1 -- 172.21.15.148:0/1671657214 wait complete. 2024-09-16T04:10:23.569 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:23.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:10:23.569 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:10:23.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.665+0000 7f7ad421e640 10 monclient: get_monmap_and_config 2024-09-16T04:10:23.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.665+0000 7f7ad421e640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.665+0000 7f7ad421e640 10 monclient: monmap: 2024-09-16T04:10:23.667 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.667 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.667 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:23.666350+0000 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:23.666350+0000 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:23.668 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.665+0000 7f7ad421e640 1 Processor -- start 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 1 -- start start 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient: init 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient(hunting): picked mon.noname-a con 0x7f7acc134870 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient(hunting): picked mon.noname-b con 0x7f7acc09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient(hunting): picked mon.noname-c con 0x7f7acc13ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc135360 con 0x7f7acc134870 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc134440 con 0x7f7acc13ae80 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc09e650 con 0x7f7acc09e240 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad421e640 10 monclient(hunting): authenticate will time out at 1044.291767s 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad1fba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7acc134870 0x7f7acc134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46538/0 (socket says 172.21.15.148:46538) 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.666+0000 7f7ad1fba640 1 -- 172.21.15.148:0/4170274685 learned_addr learned my addr 172.21.15.148:0/4170274685 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266195701 0 0) 0x7f7acc135360 con 0x7f7acc134870 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): my global_id is 64164 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ab8003520 con 0x7f7acc134870 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755751950 0 0) 0x7f7acc09e650 con 0x7f7acc09e240 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient(hunting): my global_id is 5759 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc135360 con 0x7f7acc09e240 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4136366610 0 0) 0x7f7ab8003520 con 0x7f7acc134870 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 >> v1:172.21.15.148:6790/0 conn(0x7f7acc09e240 legacy=0x7f7acc139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 >> v1:172.21.15.171:6789/0 conn(0x7f7acc13ae80 legacy=0x7f7acc13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 monclient: found mon.noname-a 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc073230 con 0x7f7acc134870 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 5 monclient: authenticate success, global_id 64164 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7abc001610 con 0x7f7acc134870 2024-09-16T04:10:23.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: dump: 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.684 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7abc001870 con 0x7f7acc134870 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 1 -- 172.21.15.148:0/4170274685 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7abc001c80 con 0x7f7acc134870 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 4 set_mon_vals no callback set 2024-09-16T04:10:23.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad0fb8640 10 monclient: dump: 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 10 monclient: shutdown 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 1 -- 172.21.15.148:0/4170274685 >> v1:172.21.15.148:6789/0 conn(0x7f7acc134870 legacy=0x7f7acc134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.667+0000 7f7ad421e640 1 -- 172.21.15.148:0/4170274685 shutdown_connections 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 1 -- 172.21.15.148:0/4170274685 >> 172.21.15.148:0/4170274685 conn(0x7f7acc0a5640 msgr2=0x7f7acc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 1 -- 172.21.15.148:0/4170274685 shutdown_connections 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 1 -- 172.21.15.148:0/4170274685 wait complete. 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 10 monclient: build_initial_monmap 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 10 monclient: monmap: 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:23.687 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 1 Processor -- start 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 1 -- start start 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.668+0000 7f7ad421e640 10 monclient: init 2024-09-16T04:10:23.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient(hunting): picked mon.noname-c con 0x7f7acc09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient(hunting): picked mon.noname-a con 0x7f7acc134870 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient(hunting): picked mon.noname-b con 0x7f7acc13ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc139f00 con 0x7f7acc134870 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc136210 con 0x7f7acc13ae80 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc09c990 con 0x7f7acc09e240 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad421e640 10 monclient(hunting): authenticate will time out at 1044.294415s 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad1fba640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7acc09e240 0x7f7acc09bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:42932/0 (socket says 172.21.15.148:42932) 2024-09-16T04:10:23.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ad1fba640 1 -- 172.21.15.148:0/1528962068 learned_addr learned my addr 172.21.15.148:0/1528962068 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047075111 0 0) 0x7f7acc09c990 con 0x7f7acc09e240 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): my global_id is 5762 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ab0003520 con 0x7f7acc09e240 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345682386 0 0) 0x7f7acc139f00 con 0x7f7acc134870 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): my global_id is 64167 2024-09-16T04:10:23.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc09c990 con 0x7f7acc134870 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360081579 0 0) 0x7f7acc136210 con 0x7f7acc13ae80 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): my global_id is 5725 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc139f00 con 0x7f7acc13ae80 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3726827943 0 0) 0x7f7acc09c990 con 0x7f7acc134870 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7acc136210 con 0x7f7acc134870 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ac80017c0 con 0x7f7acc134870 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): dump: 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 204927414 0 0) 0x7f7ab0003520 con 0x7f7acc09e240 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7acc09c990 con 0x7f7acc09e240 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7abc001c00 con 0x7f7acc09e240 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.669+0000 7f7ac27fc640 10 monclient(hunting): dump: 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.693 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1395803259 0 0) 0x7f7acc139f00 con 0x7f7acc13ae80 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ab0003520 con 0x7f7acc13ae80 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ac4001af0 con 0x7f7acc13ae80 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient(hunting): dump: 2024-09-16T04:10:23.694 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3818137257 0 0) 0x7f7acc136210 con 0x7f7acc134870 2024-09-16T04:10:23.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.148:6790/0 conn(0x7f7acc09e240 legacy=0x7f7acc09bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7acc13ae80 legacy=0x7f7acc1d3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 monclient: found mon.a 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc09cfe0 con 0x7f7acc134870 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 5 monclient: authenticate success, global_id 64167 2024-09-16T04:10:23.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 10 monclient: _renew_subs 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc135f30 con 0x7f7acc134870 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 10 monclient: _renew_subs 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7acc134c50 con 0x7f7acc134870 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7ac80035f0 con 0x7f7acc134870 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7ac80037a0 con 0x7f7acc134870 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac3fff640 4 set_mon_vals no callback set 2024-09-16T04:10:23.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.670+0000 7f7ac27fc640 10 monclient: dump: 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7ac800ea30 con 0x7f7acc134870 2024-09-16T04:10:23.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad421e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ab40065c0 con 0x7f7ab4002860 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f7ac8046a20 con 0x7f7acc134870 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad27bb640 10 monclient: get_auth_request con 0x7f7ab0039200 auth_method 0 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854946608 0 0) 0x7f7ab40065c0 con 0x7f7ab4002860 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient(hunting): my global_id is 5728 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc136210 con 0x7f7ab4002860 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3247327709 0 0) 0x7f7acc136210 con 0x7f7ab4002860 2024-09-16T04:10:23.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ab40065c0 con 0x7f7ab4002860 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7abc004dd0 con 0x7f7ab4002860 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 10 monclient: dump: 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.700 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad1fba640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab4002860 legacy=0x7f7ab4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad1fba640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab4002860 legacy=0x7f7ab4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad1fba640 1 --1- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab4002860 0x7f7ab4004d00 :-1 s=OPENED pgs=612 cs=1 l=1).handle_message read tag failed 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ad1fba640 1 --1- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab4002860 0x7f7ab4004d00 :-1 s=OPENED pgs=612 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab4002860 legacy=0x7f7ab4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab4002860 legacy=0x7f7ab4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.671+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7acc136210 con 0x7f7ab0045d00 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486485327 0 0) 0x7f7acc136210 con 0x7f7ab0045d00 2024-09-16T04:10:23.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient(hunting): my global_id is 5731 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc09c990 con 0x7f7ab0045d00 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 5904609 0 0) 0x7f7acc09c990 con 0x7f7ab0045d00 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7acc136210 con 0x7f7ab0045d00 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7abc005220 con 0x7f7ab0045d00 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:23.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient: dump: 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.703 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4183696953 0 0) 0x7f7acc136210 con 0x7f7ab0045d00 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ab00467f0 con 0x7f7ab0045d00 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7abc007510 con 0x7f7ab0045d00 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab0045d00 legacy=0x7f7ab00460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.672+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab0045d00 legacy=0x7f7ab00460e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.678+0000 7f7ad421e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.678+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7ab400c4b0 con 0x7f7ab40090e0 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560787037 0 0) 0x7f7ab400c4b0 con 0x7f7ab40090e0 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient(hunting): my global_id is 5734 2024-09-16T04:10:23.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc136210 con 0x7f7ab40090e0 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2675092131 0 0) 0x7f7acc136210 con 0x7f7ab40090e0 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ab400c4b0 con 0x7f7ab40090e0 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7abc0057e0 con 0x7f7ab40090e0 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 10 monclient: dump: 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.707 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3442643494 0 0) 0x7f7ab400c4b0 con 0x7f7ab40090e0 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.679+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ab4009c60 con 0x7f7ab40090e0 2024-09-16T04:10:23.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 3543885922) 0x7f7abc0076c0 con 0x7f7ab40090e0 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ac27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ac27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab40090e0 legacy=0x7f7ab4009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ac27fc640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6789/0 conn(0x7f7ab40090e0 legacy=0x7f7ab4009550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad27bb640 1 -- 172.21.15.148:0/1528962068 reap_dead start 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:23.709 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 22, 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 247717, 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:23.710 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.711 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.712 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:23.713 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:23.714 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:23.715 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.716 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.717 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.718 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:23.719 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:23.720 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:23.721 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.722 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:23.723 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:23.724 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:23.724 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.171:6812/7327273 conn(0x7f7ab0039200 legacy=0x7f7ab003b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 10 monclient: shutdown 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 >> v1:172.21.15.148:6789/0 conn(0x7f7acc134870 legacy=0x7f7acc09c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 shutdown_connections 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 >> 172.21.15.148:0/1528962068 conn(0x7f7acc0a5640 msgr2=0x7f7acc0a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:23.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 shutdown_connections 2024-09-16T04:10:23.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:23.680+0000 7f7ad421e640 1 -- 172.21.15.148:0/1528962068 wait complete. 2024-09-16T04:10:23.726 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:23.726 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:10:23.726 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T04:10:23.726 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:10:23.726 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:10:23.727 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:10:23.727 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:23.739 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:10:23.739 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:10:23.740 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:10:23.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:10:25.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:25.053+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:25.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:25.053+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:25.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:25.053+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:25.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:25.053+0000 7f69f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:26.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:26.054+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:26.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:26.054+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:26.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:26.054+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:27.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:27.053+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:27.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:27.053+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:27.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:27.053+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:28.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:28.054+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:28.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:28.054+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:28.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:28.054+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:29.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:29.054+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:29.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:29.054+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:29.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:29.054+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:30.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:30.054+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:30.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:30.054+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:30.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:30.054+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:31.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:31.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:31.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:31.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:31.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:31.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:32.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:32.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:32.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:32.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:32.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:32.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:33.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:33.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:33.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:33.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:33.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:33.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:34.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:34.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:34.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:34.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:34.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:34.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:35.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:35.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:35.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:35.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:35.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:35.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:35.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:35.055+0000 7f69f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:36.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:36.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:36.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:36.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:36.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:36.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:37.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:37.055+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:37.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:37.055+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:37.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:37.055+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:38.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:38.056+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:38.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:38.056+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:38.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:38.056+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:38.741 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:10:38.741 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:10:38.741 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:10:38.742 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:38.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.848+0000 7f252c637640 10 monclient: get_monmap_and_config 2024-09-16T04:10:38.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.848+0000 7f252c637640 10 monclient: build_initial_monmap 2024-09-16T04:10:38.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.848+0000 7f252c637640 10 monclient: monmap: 2024-09-16T04:10:38.850 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:38.849434+0000 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:38.849434+0000 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:38.851 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:38.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 1 Processor -- start 2024-09-16T04:10:38.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 1 -- start start 2024-09-16T04:10:38.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient: init 2024-09-16T04:10:38.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient(hunting): picked mon.noname-b con 0x7f25240a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient(hunting): picked mon.noname-c con 0x7f252409b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient(hunting): picked mon.noname-a con 0x7f252409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252409e040 con 0x7f252409b5c0 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252409d660 con 0x7f252409b170 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252409c0d0 con 0x7f25240a0e90 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252c637640 10 monclient(hunting): authenticate will time out at 1059.474842s 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252a3d3640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f252409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252abd4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f252409b5c0 0x7f252413bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36846/0 (socket says 172.21.15.148:36846) 2024-09-16T04:10:38.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:38.849+0000 7f252abd4640 1 -- 172.21.15.148:0/3300523163 learned_addr learned my addr 172.21.15.148:0/3300523163 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:39.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:39.050+0000 7f252a3d3640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f252409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:39.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:39.056+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:39.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:39.056+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:39.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:39.056+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:39.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:39.451+0000 7f252a3d3640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f252409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:40.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:40.056+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:40.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:40.056+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:40.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:40.056+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:40.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:40.252+0000 7f252a3d3640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f252409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:41.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:41.056+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:41.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:41.056+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:41.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:41.056+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:41.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): tick 2024-09-16T04:10:41.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:10:41.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 monclient(hunting): continuing hunt 2024-09-16T04:10:41.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f252409aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.171:6789/0 conn(0x7f252409b170 legacy=0x7f25241383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6789/0 conn(0x7f252409b5c0 legacy=0x7f252413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f25080036c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f252409b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f25240a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 -- 172.21.15.148:0/3300523163 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252409d380 con 0x7f25080036c0 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 -- 172.21.15.148:0/3300523163 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252409c0d0 con 0x7f252409b170 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 1 -- 172.21.15.148:0/3300523163 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2508012630 con 0x7f25240a0e90 2024-09-16T04:10:41.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f251bfff640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:41.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:41.850+0000 7f252abd4640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f250800eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:42.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:42.051+0000 7f252abd4640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f250800eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:42.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:42.056+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:42.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:42.056+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:42.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:42.056+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:42.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:42.452+0000 7f252abd4640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f250800eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:43.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:43.056+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:43.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:43.056+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:43.056+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.253+0000 7f252abd4640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f250800eef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:43.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470485518 0 0) 0x7f252409d380 con 0x7f25080036c0 2024-09-16T04:10:43.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient(hunting): _negotiate 2024-09-16T04:10:43.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient(hunting): my global_id is 64170 2024-09-16T04:10:43.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2510003520 con 0x7f25080036c0 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3443122815 0 0) 0x7f2510003520 con 0x7f25080036c0 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f250800eef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.171:6789/0 conn(0x7f252409b170 legacy=0x7f250800b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 monclient: found mon.noname-a 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:43.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2508013870 con 0x7f25080036c0 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f25200041a0 con 0x7f25080036c0 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f252c637640 5 monclient: authenticate success, global_id 64170 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f252c637640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f25293d1640 10 monclient: dump: 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:43.761 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.751+0000 7f252c637640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2520004470 con 0x7f25080036c0 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f25293d1640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f25293d1640 1 -- 172.21.15.148:0/3300523163 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2520004880 con 0x7f25080036c0 2024-09-16T04:10:43.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f25293d1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f25293d1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f25293d1640 10 monclient: dump: 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:43.763 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f252c637640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f252c637640 4 set_mon_vals no callback set 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f252c637640 10 monclient: shutdown 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f252c637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.752+0000 7f252c637640 1 -- 172.21.15.148:0/3300523163 >> v1:172.21.15.148:6789/0 conn(0x7f25080036c0 legacy=0x7f2508003b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.753+0000 7f252c637640 1 -- 172.21.15.148:0/3300523163 shutdown_connections 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.753+0000 7f252c637640 1 -- 172.21.15.148:0/3300523163 >> 172.21.15.148:0/3300523163 conn(0x7f25240a5640 msgr2=0x7f25240a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.753+0000 7f252c637640 1 -- 172.21.15.148:0/3300523163 shutdown_connections 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.753+0000 7f252c637640 1 -- 172.21.15.148:0/3300523163 wait complete. 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.753+0000 7f252c637640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.753+0000 7f252c637640 10 monclient: monmap: 2024-09-16T04:10:43.764 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 1 Processor -- start 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 1 -- start start 2024-09-16T04:10:43.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 10 monclient: init 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 10 monclient(hunting): picked mon.noname-b con 0x7f25080036c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 10 monclient(hunting): picked mon.noname-a con 0x7f252409b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 10 monclient(hunting): picked mon.noname-c con 0x7f25240a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.754+0000 7f252c637640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25241c1a30 con 0x7f252409b170 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f252c637640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25241a6210 con 0x7f25080036c0 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f252c637640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25241a8c80 con 0x7f25240a0e90 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f252c637640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f252c637640 10 monclient(hunting): authenticate will time out at 1064.380049s 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f252abd4640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f25241cfd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f2529bd2640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f252409b170 0x7f25241cc650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42166/0 (socket says 172.21.15.148:42166) 2024-09-16T04:10:43.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f2529bd2640 1 -- 172.21.15.148:0/933579758 learned_addr learned my addr 172.21.15.148:0/933579758 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541419121 0 0) 0x7f25241c1a30 con 0x7f252409b170 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): my global_id is 64173 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2508003b00 con 0x7f252409b170 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1245872977 0 0) 0x7f2508003b00 con 0x7f252409b170 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25241c1a30 con 0x7f252409b170 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2514001820 con 0x7f252409b170 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:43.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): dump: 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:43.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.755+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2539554677 0 0) 0x7f25241c1a30 con 0x7f252409b170 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 >> v1:172.21.15.148:6790/0 conn(0x7f25240a0e90 legacy=0x7f25241cfd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 >> v1:172.21.15.171:6789/0 conn(0x7f25080036c0 legacy=0x7f25241c1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 1 monclient: found mon.a 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2524072ae0 con 0x7f252409b170 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 5 monclient: authenticate success, global_id 64173 2024-09-16T04:10:43.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 10 monclient: _renew_subs 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25241a69b0 con 0x7f252409b170 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 10 monclient: _renew_subs 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f25241a7730 con 0x7f252409b170 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2514001c70 con 0x7f252409b170 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2514003a40 con 0x7f252409b170 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f2528bd0640 4 set_mon_vals no callback set 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.756+0000 7f251affd640 10 monclient: dump: 2024-09-16T04:10:43.770 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.757+0000 7f252c637640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.757+0000 7f252c637640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.757+0000 7f252c637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:43.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.757+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f250c005080 con 0x7f252409b170 2024-09-16T04:10:43.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.758+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2514003c60 con 0x7f252409b170 2024-09-16T04:10:43.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.759+0000 7f252a3d3640 10 monclient: get_auth_request con 0x7f250803b9d0 auth_method 0 2024-09-16T04:10:43.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.759+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f2514044b90 con 0x7f252409b170 2024-09-16T04:10:43.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.765+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f252409d660 con 0x7f252409b170 2024-09-16T04:10:43.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.765+0000 7f251affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:43.765+0000 7f251affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f252c637640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f252c637640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f252c637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f250c005370 con 0x7f252409b170 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f251affd640 1 -- 172.21.15.148:0/933579758 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3585544621) 0x7f2514015cb0 con 0x7f252409b170 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f251affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:44.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.002+0000 7f251affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:44.005 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:44.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 >> v1:172.21.15.171:6812/7327273 conn(0x7f250803b9d0 legacy=0x7f250803de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 10 monclient: shutdown 2024-09-16T04:10:44.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 >> v1:172.21.15.148:6789/0 conn(0x7f252409b170 legacy=0x7f25241cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 shutdown_connections 2024-09-16T04:10:44.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 >> 172.21.15.148:0/933579758 conn(0x7f25240a5640 msgr2=0x7f25240a35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 shutdown_connections 2024-09-16T04:10:44.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.004+0000 7f252c637640 1 -- 172.21.15.148:0/933579758 wait complete. 2024-09-16T04:10:44.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:10:44.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:10:44.018 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:44.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.057+0000 7f69f4ff9640 10 monclient: tick 2024-09-16T04:10:44.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.057+0000 7f69f4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.057+0000 7f69f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.113+0000 7f8833ae6640 10 monclient: get_monmap_and_config 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.113+0000 7f8833ae6640 10 monclient: build_initial_monmap 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.113+0000 7f8833ae6640 10 monclient: monmap: 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:44.114582+0000 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:44.114582+0000 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:44.116 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:44.117 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:44.117 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 1 Processor -- start 2024-09-16T04:10:44.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 1 -- start start 2024-09-16T04:10:44.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient: init 2024-09-16T04:10:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f882c09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f882c09c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f882c09cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:44.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c0a1010 con 0x7f882c09bb70 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c09f030 con 0x7f882c09cc30 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c09dc60 con 0x7f882c09c660 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8833ae6640 10 monclient(hunting): authenticate will time out at 1064.739926s 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.114+0000 7f8831081640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f882c09c660 legacy=0x7f882c138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8831882640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f882c09bb70 0x7f882c09bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42176/0 (socket says 172.21.15.148:42176) 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8831882640 1 -- 172.21.15.148:0/1914885818 learned_addr learned my addr 172.21.15.148:0/1914885818 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246378968 0 0) 0x7f882c0a1010 con 0x7f882c09bb70 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): my global_id is 64176 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8818003520 con 0x7f882c09bb70 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621909916 0 0) 0x7f882c09f030 con 0x7f882c09cc30 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient(hunting): my global_id is 5737 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882c0a1010 con 0x7f882c09cc30 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982980389 0 0) 0x7f8818003520 con 0x7f882c09bb70 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 >> v1:172.21.15.148:6790/0 conn(0x7f882c09c660 legacy=0x7f882c138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 >> v1:172.21.15.171:6789/0 conn(0x7f882c09cc30 legacy=0x7f882c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: _finish_hunting 0 2024-09-16T04:10:44.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 monclient: found mon.noname-a 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c09df40 con 0x7f882c09bb70 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: _finish_auth 0 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8833ae6640 5 monclient: authenticate success, global_id 64176 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8828001610 con 0x7f882c09bb70 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: dump: 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.129 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8833ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88280018c0 con 0x7f882c09bb70 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 1 -- 172.21.15.148:0/1914885818 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8828001c90 con 0x7f882c09bb70 2024-09-16T04:10:44.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8833ae6640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8833ae6640 4 set_mon_vals no callback set 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.115+0000 7f8830880640 10 monclient: dump: 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.131 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 10 monclient: shutdown 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 1 -- 172.21.15.148:0/1914885818 >> v1:172.21.15.148:6789/0 conn(0x7f882c09bb70 legacy=0x7f882c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 1 -- 172.21.15.148:0/1914885818 shutdown_connections 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 1 -- 172.21.15.148:0/1914885818 >> 172.21.15.148:0/1914885818 conn(0x7f882c0a5640 msgr2=0x7f882c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 1 -- 172.21.15.148:0/1914885818 shutdown_connections 2024-09-16T04:10:44.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 1 -- 172.21.15.148:0/1914885818 wait complete. 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 10 monclient: build_initial_monmap 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.116+0000 7f8833ae6640 10 monclient: monmap: 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 1 Processor -- start 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 1 -- start start 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient: init 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f882c09bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f882c09c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f882c09cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c06e450 con 0x7f882c09cc30 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8831882640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f882c09bb70 legacy=0x7f882c13b4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c11c960 con 0x7f882c09c660 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c11cc40 con 0x7f882c09bb70 2024-09-16T04:10:44.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8833ae6640 10 monclient(hunting): authenticate will time out at 1064.742596s 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8832083640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f882c09cc30 0x7f882c1d1820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42186/0 (socket says 172.21.15.148:42186) 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f8832083640 1 -- 172.21.15.148:0/2501670744 learned_addr learned my addr 172.21.15.148:0/2501670744 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098511040 0 0) 0x7f882c0a1010 con 0x7f882c09cc30 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): my global_id is 64179 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8810003550 con 0x7f882c09cc30 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772658650 0 0) 0x7f882c11c960 con 0x7f882c09c660 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 10 monclient(hunting): my global_id is 5740 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.117+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882c0a1010 con 0x7f882c09c660 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2841783321 0 0) 0x7f8810003550 con 0x7f882c09cc30 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f882c11c960 con 0x7f882c09cc30 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8824001c50 con 0x7f882c09cc30 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient(hunting): dump: 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.136 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1399596160 0 0) 0x7f882c11c960 con 0x7f882c09cc30 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.148:6790/0 conn(0x7f882c09bb70 legacy=0x7f882c13b4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.171:6789/0 conn(0x7f882c09c660 legacy=0x7f882c13bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 monclient: found mon.a 2024-09-16T04:10:44.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c13c2f0 con 0x7f882c09cc30 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 5 monclient: authenticate success, global_id 64179 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 10 monclient: _renew_subs 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f882c11c3a0 con 0x7f882c09cc30 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 10 monclient: _renew_subs 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f882c09ad00 con 0x7f882c09cc30 2024-09-16T04:10:44.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8824002020 con 0x7f882c09cc30 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8824003de0 con 0x7f882c09cc30 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8823fff640 4 set_mon_vals no callback set 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 10 monclient: dump: 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.139 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8824003f80 con 0x7f882c09cc30 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.118+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88140065c0 con 0x7f8814002860 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f8824048330 con 0x7f882c09cc30 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998506889 0 0) 0x7f88140065c0 con 0x7f8814002860 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient(hunting): my global_id is 64182 2024-09-16T04:10:44.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f882c11c960 con 0x7f8814002860 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f8831081640 10 monclient: get_auth_request con 0x7f8810038f70 auth_method 0 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2406236339 0 0) 0x7f882c11c960 con 0x7f8814002860 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88140065c0 con 0x7f8814002860 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8828001870 con 0x7f8814002860 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient: dump: 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.141 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3064983544 0 0) 0x7f88140065c0 con 0x7f8814002860 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f881400a150 con 0x7f8814002860 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8828001820 con 0x7f8814002860 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.119+0000 7f88227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.120+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.148:6789/0 conn(0x7f8814002860 legacy=0x7f8814004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.120+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.148:6789/0 conn(0x7f8814002860 legacy=0x7f8814004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:44.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.125+0000 7f8833ae6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:44.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.125+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8814009e50 con 0x7f8814009110 2024-09-16T04:10:44.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886656090 0 0) 0x7f8814009e50 con 0x7f8814009110 2024-09-16T04:10:44.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient(hunting): my global_id is 64185 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88140065c0 con 0x7f8814009110 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2687023346 0 0) 0x7f88140065c0 con 0x7f8814009110 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8814009e50 con 0x7f8814009110 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8828004a40 con 0x7f8814009110 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient: dump: 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.145 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3987889082 0 0) 0x7f8814009e50 con 0x7f8814009110 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f881400c3d0 con 0x7f8814009110 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4134 (unknown 0 0 3640972201) 0x7f8828005ed0 con 0x7f8814009110 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.148:6789/0 conn(0x7f8814009110 legacy=0x7f88140095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.126+0000 7f88227fc640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.148:6789/0 conn(0x7f8814009110 legacy=0x7f88140095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 24, 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 23324, 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:10:44.147 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:44.148 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:44.149 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:44.150 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:44.151 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:44.152 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:44.153 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:44.155 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:44.156 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:10:44.157 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.158 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.159 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:44.159 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:44.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.171:6812/7327273 conn(0x7f8810038f70 legacy=0x7f881003b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 10 monclient: shutdown 2024-09-16T04:10:44.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 >> v1:172.21.15.148:6789/0 conn(0x7f882c09cc30 legacy=0x7f882c1d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8832083640 1 -- 172.21.15.148:0/2501670744 reap_dead start 2024-09-16T04:10:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8832083640 1 -- 172.21.15.148:0/2501670744 reap_dead start 2024-09-16T04:10:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 shutdown_connections 2024-09-16T04:10:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 >> 172.21.15.148:0/2501670744 conn(0x7f882c0a5640 msgr2=0x7f882c071a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 shutdown_connections 2024-09-16T04:10:44.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.127+0000 7f8833ae6640 1 -- 172.21.15.148:0/2501670744 wait complete. 2024-09-16T04:10:44.161 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:10:44.161 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:10:44.161 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.258+0000 7f881a9c4640 10 monclient: get_monmap_and_config 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.258+0000 7f881a9c4640 10 monclient: build_initial_monmap 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.258+0000 7f881a9c4640 10 monclient: monmap: 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:10:44.259200+0000 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:10:44.259200+0000 2024-09-16T04:10:44.260 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:44.261 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:44.261 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 1 Processor -- start 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 1 -- start start 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient: init 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f88140a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f88140a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient(hunting): picked mon.noname-c con 0x7f881413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:44.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88140a0e00 con 0x7f88140a2410 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88140a10e0 con 0x7f881413d060 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88140a4970 con 0x7f88140a1b10 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f881a9c4640 10 monclient(hunting): authenticate will time out at 1064.884808s 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f880bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f88140a1b10 legacy=0x7f881413b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f8813fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f88140a2410 0x7f88140a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42208/0 (socket says 172.21.15.148:42208) 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.259+0000 7f8813fff640 1 -- 172.21.15.148:0/2642002975 learned_addr learned my addr 172.21.15.148:0/2642002975 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290765139 0 0) 0x7f88140a0e00 con 0x7f88140a2410 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): my global_id is 64188 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8800003520 con 0x7f88140a2410 2024-09-16T04:10:44.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1938907662 0 0) 0x7f88140a10e0 con 0x7f881413d060 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient(hunting): my global_id is 5743 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88140a0e00 con 0x7f881413d060 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1158543646 0 0) 0x7f8800003520 con 0x7f88140a2410 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 >> v1:172.21.15.148:6790/0 conn(0x7f88140a1b10 legacy=0x7f881413b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 >> v1:172.21.15.171:6789/0 conn(0x7f881413d060 legacy=0x7f881413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 monclient: found mon.noname-a 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881406dfd0 con 0x7f88140a2410 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8804001610 con 0x7f88140a2410 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f881a9c4640 5 monclient: authenticate success, global_id 64188 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f881a9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: dump: 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.273 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f881a9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8804001870 con 0x7f88140a2410 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 1 -- 172.21.15.148:0/2642002975 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8804001c40 con 0x7f88140a2410 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f881a9c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f881a9c4640 4 set_mon_vals no callback set 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f88137fe640 10 monclient: dump: 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.275 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.260+0000 7f881a9c4640 10 monclient: shutdown 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 -- 172.21.15.148:0/2642002975 >> v1:172.21.15.148:6789/0 conn(0x7f88140a2410 legacy=0x7f88140a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 -- 172.21.15.148:0/2642002975 shutdown_connections 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 -- 172.21.15.148:0/2642002975 >> 172.21.15.148:0/2642002975 conn(0x7f88140a97b0 msgr2=0x7f88140a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 -- 172.21.15.148:0/2642002975 shutdown_connections 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 -- 172.21.15.148:0/2642002975 wait complete. 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 10 monclient: build_initial_monmap 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 10 monclient: monmap: 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:44.276 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 Processor -- start 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.261+0000 7f881a9c4640 1 -- start start 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient: init 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f88140a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient(hunting): picked mon.noname-c con 0x7f88140a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f881413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f881409e770 con 0x7f881413d060 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88141380f0 con 0x7f88140a1b10 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8814137cd0 con 0x7f88140a2410 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f880bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f88140a2410 legacy=0x7f881409d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f881a9c4640 10 monclient(hunting): authenticate will time out at 1064.887448s 2024-09-16T04:10:44.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f8818f61640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f881413d060 0x7f88141d3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42214/0 (socket says 172.21.15.148:42214) 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f8818f61640 1 -- 172.21.15.148:0/2931407029 learned_addr learned my addr 172.21.15.148:0/2931407029 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062658676 0 0) 0x7f881409e770 con 0x7f881413d060 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): my global_id is 64191 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f4003520 con 0x7f881413d060 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 572762055 0 0) 0x7f881406dfd0 con 0x7f88140a1b10 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): my global_id is 5746 2024-09-16T04:10:44.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f881409e770 con 0x7f88140a1b10 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1991374120 0 0) 0x7f87f4003520 con 0x7f881413d060 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f881406dfd0 con 0x7f881413d060 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f880c001d00 con 0x7f881413d060 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): dump: 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.280 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.262+0000 7f88117fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1061019441 0 0) 0x7f881409e770 con 0x7f88140a1b10 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87f4003520 con 0x7f88140a1b10 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8804001370 con 0x7f88140a1b10 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient(hunting): dump: 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.281 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1539401073 0 0) 0x7f881406dfd0 con 0x7f881413d060 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.148:6790/0 conn(0x7f88140a2410 legacy=0x7f881409d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.171:6789/0 conn(0x7f88140a1b10 legacy=0x7f881409cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 monclient: found mon.a 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881413cce0 con 0x7f881413d060 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 5 monclient: authenticate success, global_id 64191 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 10 monclient: _renew_subs 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88141379f0 con 0x7f881413d060 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 10 monclient: _renew_subs 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f881409f4f0 con 0x7f881413d060 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f880c003c40 con 0x7f881413d060 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f880c003f90 con 0x7f881413d060 2024-09-16T04:10:44.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f8812ffd640 4 set_mon_vals no callback set 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 10 monclient: dump: 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.284 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87f80065c0 con 0x7f87f8002860 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.263+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f880c004130 con 0x7f881413d060 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(144..144 src has 1..144) ==== 8239+0+0 (unknown 2990707811 0 0) 0x7f880c048360 con 0x7f881413d060 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313592813 0 0) 0x7f87f80065c0 con 0x7f87f8002860 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient(hunting): my global_id is 5749 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f881406dfd0 con 0x7f87f8002860 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f880bfff640 10 monclient: get_auth_request con 0x7f87f4038ef0 auth_method 0 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3501802575 0 0) 0x7f881406dfd0 con 0x7f87f8002860 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87f80065c0 con 0x7f87f8002860 2024-09-16T04:10:44.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8804004b30 con 0x7f87f8002860 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 10 monclient: dump: 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.286 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4181496217 0 0) 0x7f87f80065c0 con 0x7f87f8002860 2024-09-16T04:10:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.264+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87f800a150 con 0x7f87f8002860 2024-09-16T04:10:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.265+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8804006090 con 0x7f87f8002860 2024-09-16T04:10:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.265+0000 7f88117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.265+0000 7f88117fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:44.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.265+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.171:6789/0 conn(0x7f87f8002860 legacy=0x7f87f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.265+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.171:6789/0 conn(0x7f87f8002860 legacy=0x7f87f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f881a9c4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87f8009e40 con 0x7f87f8009100 2024-09-16T04:10:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302058240 0 0) 0x7f87f8009e40 con 0x7f87f8009100 2024-09-16T04:10:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f88117fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f88117fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f88117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f88117fa640 10 monclient(hunting): my global_id is 5752 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.271+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f80065c0 con 0x7f87f8009100 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 181893381 0 0) 0x7f87f80065c0 con 0x7f87f8009100 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87f8009e40 con 0x7f87f8009100 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88040064e0 con 0x7f87f8009100 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 10 monclient: dump: 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.290 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2166852874 0 0) 0x7f87f8009e40 con 0x7f87f8009100 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87f800c3d0 con 0x7f87f8009100 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4300 (unknown 0 0 1962308240) 0x7f8804006420 con 0x7f87f8009100 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:44.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.171:6789/0 conn(0x7f87f8009100 legacy=0x7f87f8009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f88117fa640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.171:6789/0 conn(0x7f87f8009100 legacy=0x7f87f8009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 24, 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 268310, 2024-09-16T04:10:44.292 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:44.293 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:44.294 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:10:44.295 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:10:44.296 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:10:44.297 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:44.298 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.299 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:10:44.300 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:10:44.301 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.302 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.303 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:44.304 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:10:44.305 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:10:44.305 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:10:44.305 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:10:44.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:10:44.305 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:10:44.305 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:10:44.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.171:6812/7327273 conn(0x7f87f4038ef0 legacy=0x7f87f403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.272+0000 7f881a9c4640 10 monclient: shutdown 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f881a9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 >> v1:172.21.15.148:6789/0 conn(0x7f881413d060 legacy=0x7f88141d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f8818f61640 1 -- 172.21.15.148:0/2931407029 reap_dead start 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f8818f61640 1 -- 172.21.15.148:0/2931407029 reap_dead start 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 shutdown_connections 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 >> 172.21.15.148:0/2931407029 conn(0x7f88140a97b0 msgr2=0x7f88141359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 shutdown_connections 2024-09-16T04:10:44.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:10:44.273+0000 7f881a9c4640 1 -- 172.21.15.148:0/2931407029 wait complete. 2024-09-16T04:10:44.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:10:44.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:10:44.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.756+0000 7f69f5ffb640 1 -- 172.21.15.171:0/309680098 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v145) ==== 149+0+0 (unknown 2008214431 0 0) 0x7f69f80197d0 con 0x7f6a000a0130 2024-09-16T04:10:44.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.756+0000 7f69f5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:44.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.756+0000 7f69f5ffb640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T04:10:44.758 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 >> v1:172.21.15.171:6812/7327273 conn(0x7f69e4039210 legacy=0x7f69e403b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 10 monclient: shutdown 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 >> v1:172.21.15.171:6789/0 conn(0x7f6a000a0130 legacy=0x7f6a001cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 shutdown_connections 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 >> 172.21.15.171:0/309680098 conn(0x7f6a000a52a0 msgr2=0x7f6a000a2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 shutdown_connections 2024-09-16T04:10:44.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.759+0000 7f6a073fd640 1 -- 172.21.15.171:0/309680098 wait complete. 2024-09-16T04:10:44.772 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-16T04:10:44.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.858+0000 7f3388fe4640 10 monclient: get_monmap_and_config 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.858+0000 7f3388fe4640 10 monclient: build_initial_monmap 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.858+0000 7f3388fe4640 10 monclient: monmap: 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:44.858819+0000 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:44.858819+0000 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:44.860 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.858+0000 7f3388fe4640 1 Processor -- start 2024-09-16T04:10:44.868 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.858+0000 7f3388fe4640 1 -- start start 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.858+0000 7f3388fe4640 10 monclient: init 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient(hunting): picked mon.noname-a con 0x7f3384136980 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient(hunting): picked mon.noname-b con 0x7f338409c1e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient(hunting): picked mon.noname-c con 0x7f3384137470 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f338409aa20 con 0x7f3384136980 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f338409af40 con 0x7f3384137470 2024-09-16T04:10:44.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3384073920 con 0x7f338409c1e0 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3388fe4640 10 monclient(hunting): authenticate will time out at 1020.364361s 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3382d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3384137470 0x7f338413f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58058/0 (socket says 172.21.15.171:58058) 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3382d9d640 1 -- 172.21.15.171:0/2043725851 learned_addr learned my addr 172.21.15.171:0/2043725851 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f3381d9b640 1 -- 172.21.15.171:0/2043725851 >> v1:172.21.15.148:6790/0 conn(0x7f338409c1e0 legacy=0x7f338413beb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378751441 0 0) 0x7f338409af40 con 0x7f3384137470 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): my global_id is 5755 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f336c003520 con 0x7f3384137470 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1634499826 0 0) 0x7f338409aa20 con 0x7f3384136980 2024-09-16T04:10:44.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient(hunting): my global_id is 64194 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f338409af40 con 0x7f3384136980 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538926482 0 0) 0x7f336c003520 con 0x7f3384137470 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 >> v1:172.21.15.148:6790/0 conn(0x7f338409c1e0 legacy=0x7f338413beb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 >> v1:172.21.15.148:6789/0 conn(0x7f3384136980 legacy=0x7f3384136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient: _finish_hunting 0 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 monclient: found mon.noname-c 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33840a5300 con 0x7f3384137470 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient: _finish_auth 0 2024-09-16T04:10:44.871 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.859+0000 7f338159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3374001990 con 0x7f3384137470 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 5 monclient: authenticate success, global_id 5755 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: dump: 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.872 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3374001cb0 con 0x7f3384137470 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 1 -- 172.21.15.171:0/2043725851 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3374002080 con 0x7f3384137470 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 4 set_mon_vals no callback set 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:44.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f338159a640 10 monclient: dump: 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 10 monclient: shutdown 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 1 -- 172.21.15.171:0/2043725851 >> v1:172.21.15.171:6789/0 conn(0x7f3384137470 legacy=0x7f338413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 1 -- 172.21.15.171:0/2043725851 shutdown_connections 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 1 -- 172.21.15.171:0/2043725851 >> 172.21.15.171:0/2043725851 conn(0x7f33840a9970 msgr2=0x7f33840a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 1 -- 172.21.15.171:0/2043725851 shutdown_connections 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.860+0000 7f3388fe4640 1 -- 172.21.15.171:0/2043725851 wait complete. 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.861+0000 7f3388fe4640 10 monclient: build_initial_monmap 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.861+0000 7f3388fe4640 10 monclient: monmap: 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:44.875 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.861+0000 7f3388fe4640 1 Processor -- start 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.861+0000 7f3388fe4640 1 -- start start 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.861+0000 7f3388fe4640 10 monclient: init 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient(hunting): picked mon.noname-c con 0x7f338409c1e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient(hunting): picked mon.noname-a con 0x7f3384136980 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient(hunting): picked mon.noname-b con 0x7f3384137470 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3384135b90 con 0x7f3384136980 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3384132d80 con 0x7f3384137470 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3384135e70 con 0x7f338409c1e0 2024-09-16T04:10:44.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3388fe4640 10 monclient(hunting): authenticate will time out at 1020.367350s 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f338259c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f338409c1e0 legacy=0x7f3384131f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3382d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3384137470 0x7f33841d3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58060/0 (socket says 172.21.15.171:58060) 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3382d9d640 1 -- 172.21.15.171:0/2872113795 learned_addr learned my addr 172.21.15.171:0/2872113795 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247467369 0 0) 0x7f3384132d80 con 0x7f3384137470 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): my global_id is 5758 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3364003520 con 0x7f3384137470 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 237181468 0 0) 0x7f3384135b90 con 0x7f3384136980 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:44.877 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): my global_id is 64197 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3384132d80 con 0x7f3384136980 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 993888974 0 0) 0x7f3364003520 con 0x7f3384137470 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3384135b90 con 0x7f3384137470 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3374001950 con 0x7f3384137470 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.878 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 378170758 0 0) 0x7f3384132d80 con 0x7f3384136980 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3364003520 con 0x7f3384136980 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33780040f0 con 0x7f3384136980 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.862+0000 7f3372ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:44.879 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2239652563 0 0) 0x7f3384135b90 con 0x7f3384137470 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 >> v1:172.21.15.148:6790/0 conn(0x7f338409c1e0 legacy=0x7f3384131f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 >> v1:172.21.15.148:6789/0 conn(0x7f3384136980 legacy=0x7f3384132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:44.880 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 monclient: found mon.b 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33840a83f0 con 0x7f3384137470 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 5 monclient: authenticate success, global_id 5758 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 10 monclient: _renew_subs 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f33840a0340 con 0x7f3384137470 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 10 monclient: _renew_subs 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3384134a20 con 0x7f3384137470 2024-09-16T04:10:44.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3374005040 con 0x7f3384137470 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3374005390 con 0x7f3384137470 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3380d99640 4 set_mon_vals no callback set 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 10 monclient: dump: 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:44.882 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.863+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f33740107c0 con 0x7f3384137470 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.864+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(145..145 src has 1..145) ==== 8239+0+0 (unknown 3233546243 0 0) 0x7f33740483e0 con 0x7f3384137470 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.864+0000 7f338259c640 10 monclient: get_auth_request con 0x7f3364038ef0 auth_method 0 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.864+0000 7f3388fe4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.864+0000 7f3388fe4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.864+0000 7f3388fe4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:44.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.864+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3368005080 con 0x7f3384137470 2024-09-16T04:10:44.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.869+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3374016190 con 0x7f3384137470 2024-09-16T04:10:44.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.869+0000 7f3372ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:44.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:44.869+0000 7f3372ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:45.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.043+0000 7f3388fe4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T04:10:45.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.043+0000 7f3388fe4640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T04:10:45.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.043+0000 7f3388fe4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:45.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.043+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) -- 0x7f3368005370 con 0x7f3384137470 2024-09-16T04:10:45.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.758+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v146) ==== 119+0+0 (unknown 4230661651 0 0) 0x7f3374015a90 con 0x7f3384137470 2024-09-16T04:10:45.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.758+0000 7f3372ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T04:10:45.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.758+0000 7f3372ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:45.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.881+0000 7f3388fe4640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T04:10:45.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.881+0000 7f3388fe4640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T04:10:45.883 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.881+0000 7f3388fe4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:45.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:45.881+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) -- 0x7f3368005bd0 con 0x7f3384137470 2024-09-16T04:10:46.825 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.823+0000 7f3372ffd640 1 -- 172.21.15.171:0/2872113795 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v148) ==== 119+0+0 (unknown 3924686237 0 0) 0x7f3374045ff0 con 0x7f3384137470 2024-09-16T04:10:46.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.823+0000 7f3372ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-16T04:10:46.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.823+0000 7f3372ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:46.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.827+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 >> v1:172.21.15.171:6812/7327273 conn(0x7f3364038ef0 legacy=0x7f336403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.829 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.827+0000 7f3388fe4640 10 monclient: shutdown 2024-09-16T04:10:46.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.827+0000 7f3388fe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:46.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.827+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 >> v1:172.21.15.171:6789/0 conn(0x7f3384137470 legacy=0x7f33841d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.827+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 shutdown_connections 2024-09-16T04:10:46.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.827+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 >> 172.21.15.171:0/2872113795 conn(0x7f33840a9970 msgr2=0x7f3384140210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:46.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.828+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 shutdown_connections 2024-09-16T04:10:46.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.828+0000 7f3388fe4640 1 -- 172.21.15.171:0/2872113795 wait complete. 2024-09-16T04:10:46.841 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-16T04:10:46.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.927+0000 7fcf231fd640 10 monclient: get_monmap_and_config 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.928+0000 7fcf231fd640 10 monclient: build_initial_monmap 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.928+0000 7fcf231fd640 10 monclient: monmap: 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:46.928798+0000 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:46.928798+0000 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:46.930 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.928+0000 7fcf231fd640 1 Processor -- start 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.928+0000 7fcf231fd640 1 -- start start 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.928+0000 7fcf231fd640 10 monclient: init 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c1341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c13a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c09da90 con 0x7fcf1c133700 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c09dd70 con 0x7fcf1c1341f0 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c09e050 con 0x7fcf1c13a350 2024-09-16T04:10:46.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf231fd640 10 monclient(hunting): authenticate will time out at 1022.434452s 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcf1c1341f0 0x7fcf1c138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58076/0 (socket says 172.21.15.171:58076) 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1bfff640 1 -- 172.21.15.171:0/2305355143 learned_addr learned my addr 172.21.15.171:0/2305355143 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf2179a640 1 -- 172.21.15.171:0/2305355143 >> v1:172.21.15.148:6790/0 conn(0x7fcf1c13a350 legacy=0x7fcf1c13c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737655202 0 0) 0x7fcf1c09dd70 con 0x7fcf1c1341f0 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 10 monclient(hunting): my global_id is 5761 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf08003520 con 0x7fcf1c1341f0 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2620863121 0 0) 0x7fcf08003520 con 0x7fcf1c1341f0 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 >> v1:172.21.15.148:6790/0 conn(0x7fcf1c13a350 legacy=0x7fcf1c13c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:46.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 >> v1:172.21.15.148:6789/0 conn(0x7fcf1c133700 legacy=0x7fcf1c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.929+0000 7fcf1b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c09d4f0 con 0x7fcf1c1341f0 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:46.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 5 monclient: authenticate success, global_id 5761 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf04001610 con 0x7fcf1c1341f0 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: dump: 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:46.942 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf04001970 con 0x7fcf1c1341f0 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 1 -- 172.21.15.171:0/2305355143 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf04001d40 con 0x7fcf1c1341f0 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf1b7fe640 10 monclient: dump: 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:46.943 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 4 set_mon_vals no callback set 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 10 monclient: shutdown 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 1 -- 172.21.15.171:0/2305355143 >> v1:172.21.15.171:6789/0 conn(0x7fcf1c1341f0 legacy=0x7fcf1c138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 1 -- 172.21.15.171:0/2305355143 shutdown_connections 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 1 -- 172.21.15.171:0/2305355143 >> 172.21.15.171:0/2305355143 conn(0x7fcf1c0a5450 msgr2=0x7fcf1c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 1 -- 172.21.15.171:0/2305355143 shutdown_connections 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.930+0000 7fcf231fd640 1 -- 172.21.15.171:0/2305355143 wait complete. 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.931+0000 7fcf231fd640 10 monclient: build_initial_monmap 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.931+0000 7fcf231fd640 10 monclient: monmap: 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:46.945 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.931+0000 7fcf231fd640 1 Processor -- start 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.931+0000 7fcf231fd640 1 -- start start 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.931+0000 7fcf231fd640 10 monclient: init 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c1341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c13a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c0a0140 con 0x7fcf1c1341f0 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c1392e0 con 0x7fcf1c133700 2024-09-16T04:10:46.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c11b410 con 0x7fcf1c13a350 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf231fd640 10 monclient(hunting): authenticate will time out at 1022.437363s 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf2179a640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fcf1c13a350 legacy=0x7fcf1c1d3550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf20f99640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcf1c133700 0x7fcf1c1384c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58092/0 (socket says 172.21.15.171:58092) 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf20f99640 1 -- 172.21.15.171:0/143580686 learned_addr learned my addr 172.21.15.171:0/143580686 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846636320 0 0) 0x7fcf1c1392e0 con 0x7fcf1c133700 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): my global_id is 5764 2024-09-16T04:10:46.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcefc003520 con 0x7fcf1c133700 2024-09-16T04:10:46.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485970618 0 0) 0x7fcf1c09d4f0 con 0x7fcf1c1341f0 2024-09-16T04:10:46.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:46.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): my global_id is 64203 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1c1392e0 con 0x7fcf1c1341f0 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 440574805 0 0) 0x7fcefc003520 con 0x7fcf1c133700 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf1c09d4f0 con 0x7fcf1c133700 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf0c0019e0 con 0x7fcf1c133700 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): dump: 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.949 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 482162853 0 0) 0x7fcf1c1392e0 con 0x7fcf1c1341f0 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcefc003520 con 0x7fcf1c1341f0 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf04001430 con 0x7fcf1c1341f0 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:46.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 10 monclient(hunting): dump: 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4110916303 0 0) 0x7fcf1c09d4f0 con 0x7fcf1c133700 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 >> v1:172.21.15.148:6790/0 conn(0x7fcf1c13a350 legacy=0x7fcf1c1d3550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.932+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 >> v1:172.21.15.148:6789/0 conn(0x7fcf1c1341f0 legacy=0x7fcf1c138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 1 monclient: found mon.b 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c11bfe0 con 0x7fcf1c133700 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 5 monclient: authenticate success, global_id 5764 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 10 monclient: _renew_subs 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c11c5a0 con 0x7fcf1c133700 2024-09-16T04:10:46.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 10 monclient: _renew_subs 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf1c11d010 con 0x7fcf1c133700 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf0c0038c0 con 0x7fcf1c133700 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcf0c003c10 con 0x7fcf1c133700 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 10 monclient: dump: 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:46.953 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf1affd640 4 set_mon_vals no callback set 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.933+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcf0c003db0 con 0x7fcf1c133700 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.934+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(148..148 src has 1..148) ==== 8396+0+0 (unknown 2353097959 0 0) 0x7fcf0c0480c0 con 0x7fcf1c133700 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.934+0000 7fcf1bfff640 10 monclient: get_auth_request con 0x7fcefc038f10 auth_method 0 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.934+0000 7fcf231fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.934+0000 7fcf231fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:46.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.934+0000 7fcf231fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:46.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.934+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf00005080 con 0x7fcf1c133700 2024-09-16T04:10:46.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.938+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fcf0c015c50 con 0x7fcf1c133700 2024-09-16T04:10:46.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.938+0000 7fcf197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:46.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:46.938+0000 7fcf197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:47.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.115+0000 7fcf231fd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T04:10:47.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.115+0000 7fcf231fd640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T04:10:47.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.116+0000 7fcf231fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:47.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.116+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) -- 0x7fcf00005370 con 0x7fcf1c133700 2024-09-16T04:10:47.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.835+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v149) ==== 128+0+0 (unknown 2733990529 0 0) 0x7fcf0c019900 con 0x7fcf1c133700 2024-09-16T04:10:47.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.835+0000 7fcf197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T04:10:47.838 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.835+0000 7fcf197fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:47.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.945+0000 7fcf231fd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T04:10:47.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.945+0000 7fcf231fd640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T04:10:47.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.945+0000 7fcf231fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:47.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:47.945+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) -- 0x7fcf00005ab0 con 0x7fcf1c133700 2024-09-16T04:10:48.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.837+0000 7fcf197fa640 1 -- 172.21.15.171:0/143580686 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v150) ==== 128+0+0 (unknown 406991207 0 0) 0x7fcf0c01e1c0 con 0x7fcf1c133700 2024-09-16T04:10:48.839 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.837+0000 7fcf197fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-16T04:10:48.840 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.837+0000 7fcf197fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.839+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 >> v1:172.21.15.171:6812/7327273 conn(0x7fcefc038f10 legacy=0x7fcefc03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.839+0000 7fcf231fd640 10 monclient: shutdown 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.839+0000 7fcf231fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.839+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 >> v1:172.21.15.171:6789/0 conn(0x7fcf1c133700 legacy=0x7fcf1c1384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.840+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 shutdown_connections 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.840+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 >> 172.21.15.171:0/143580686 conn(0x7fcf1c0a5450 msgr2=0x7fcf1c0a4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:48.842 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.840+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 shutdown_connections 2024-09-16T04:10:48.843 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.840+0000 7fcf231fd640 1 -- 172.21.15.171:0/143580686 wait complete. 2024-09-16T04:10:48.852 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.936+0000 7fe4e93fd640 10 monclient: get_monmap_and_config 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.936+0000 7fe4e93fd640 10 monclient: build_initial_monmap 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.936+0000 7fe4e93fd640 10 monclient: monmap: 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:48.937125+0000 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:48.937125+0000 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:48.938 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.939 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:48.939 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:48.939 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:48.939 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.936+0000 7fe4e93fd640 1 Processor -- start 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.936+0000 7fe4e93fd640 1 -- start start 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.936+0000 7fe4e93fd640 10 monclient: init 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient(hunting): picked mon.noname-b con 0x7fe4e409eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e4134b20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient(hunting): picked mon.noname-c con 0x7fe4e4134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e409d060 con 0x7fe4e4134b20 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e409d340 con 0x7fe4e4134f30 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e4068840 con 0x7fe4e409eaa0 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:48.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e93fd640 10 monclient(hunting): authenticate will time out at 1024.442474s 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e2ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe4e409eaa0 legacy=0x7fe4e409ee80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e37fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe4e4134f30 0x7fe4e413d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45214/0 (socket says 172.21.15.171:45214) 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e37fe640 1 -- 172.21.15.171:0/2954595319 learned_addr learned my addr 172.21.15.171:0/2954595319 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428289591 0 0) 0x7fe4e409d340 con 0x7fe4e4134f30 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): my global_id is 5767 2024-09-16T04:10:48.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4cc003520 con 0x7fe4e4134f30 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 821797665 0 0) 0x7fe4e409d060 con 0x7fe4e4134b20 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 10 monclient(hunting): my global_id is 64206 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4e409d340 con 0x7fe4e4134b20 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.937+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3120327189 0 0) 0x7fe4cc003520 con 0x7fe4e4134f30 2024-09-16T04:10:48.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 >> v1:172.21.15.148:6790/0 conn(0x7fe4e409eaa0 legacy=0x7fe4e409ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 >> v1:172.21.15.148:6789/0 conn(0x7fe4e4134b20 legacy=0x7fe4e4139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 monclient: found mon.noname-c 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e4141800 con 0x7fe4e4134f30 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe4dc001990 con 0x7fe4e4134f30 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:48.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: dump: 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:48.952 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 5 monclient: authenticate success, global_id 5767 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4dc001c40 con 0x7fe4e4134f30 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 1 -- 172.21.15.171:0/2954595319 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe4dc002010 con 0x7fe4e4134f30 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e1ffb640 10 monclient: dump: 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.953 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 4 set_mon_vals no callback set 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 10 monclient: shutdown 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2954595319 >> v1:172.21.15.171:6789/0 conn(0x7fe4e4134f30 legacy=0x7fe4e413d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2954595319 shutdown_connections 2024-09-16T04:10:48.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2954595319 >> 172.21.15.171:0/2954595319 conn(0x7fe4e40a1e90 msgr2=0x7fe4e40a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2954595319 shutdown_connections 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.938+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2954595319 wait complete. 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.939+0000 7fe4e93fd640 10 monclient: build_initial_monmap 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.939+0000 7fe4e93fd640 10 monclient: monmap: 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:48.955 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.939+0000 7fe4e93fd640 1 Processor -- start 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.939+0000 7fe4e93fd640 1 -- start start 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.939+0000 7fe4e93fd640 10 monclient: init 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient(hunting): picked mon.noname-c con 0x7fe4e409eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e4134b20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient(hunting): picked mon.noname-b con 0x7fe4e4134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e409aa10 con 0x7fe4e4134b20 2024-09-16T04:10:48.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e4141560 con 0x7fe4e4134f30 2024-09-16T04:10:48.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e41411c0 con 0x7fe4e409eaa0 2024-09-16T04:10:48.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:48.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient(hunting): authenticate will time out at 1024.445259s 2024-09-16T04:10:48.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e2ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe4e409eaa0 legacy=0x7fe4e4098370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:48.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e37fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe4e4134f30 0x7fe4e41d42b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45230/0 (socket says 172.21.15.171:45230) 2024-09-16T04:10:48.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e37fe640 1 -- 172.21.15.171:0/2525988689 learned_addr learned my addr 172.21.15.171:0/2525988689 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:48.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4192591745 0 0) 0x7fe4e4141800 con 0x7fe4e4134f30 2024-09-16T04:10:48.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:48.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:48.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:48.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): my global_id is 5770 2024-09-16T04:10:48.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4d0003520 con 0x7fe4e4134f30 2024-09-16T04:10:48.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3355397708 0 0) 0x7fe4e409aa10 con 0x7fe4e4134b20 2024-09-16T04:10:48.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:48.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:48.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:48.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): my global_id is 64209 2024-09-16T04:10:48.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4e4141800 con 0x7fe4e4134b20 2024-09-16T04:10:48.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3005686307 0 0) 0x7fe4d0003520 con 0x7fe4e4134f30 2024-09-16T04:10:48.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4e409aa10 con 0x7fe4e4134f30 2024-09-16T04:10:48.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe4dc001f90 con 0x7fe4e4134f30 2024-09-16T04:10:48.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:48.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:48.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.961 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4242952122 0 0) 0x7fe4e4141800 con 0x7fe4e4134b20 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4d0003520 con 0x7fe4e4134b20 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe4d80018b0 con 0x7fe4e4134b20 2024-09-16T04:10:48.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:48.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:48.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient(hunting): dump: 2024-09-16T04:10:48.963 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:48.963 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:48.963 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.964 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.964 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:48.964 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.964 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:48.964 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:48.964 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:48.965 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 207532959 0 0) 0x7fe4e409aa10 con 0x7fe4e4134f30 2024-09-16T04:10:48.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 >> v1:172.21.15.148:6790/0 conn(0x7fe4e409eaa0 legacy=0x7fe4e4098370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:48.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 >> v1:172.21.15.148:6789/0 conn(0x7fe4e4134b20 legacy=0x7fe4e41d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:48.965 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 monclient: found mon.b 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e409c580 con 0x7fe4e4134f30 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:48.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 5 monclient: authenticate success, global_id 5770 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient: _renew_subs 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.940+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e413cd00 con 0x7fe4e4134f30 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4e93fd640 10 monclient: _renew_subs 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4e93fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:48.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4e413ab20 con 0x7fe4e4134f30 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4dc001c40 con 0x7fe4e4134f30 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe4dc004fb0 con 0x7fe4e4134f30 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4e17fa640 4 set_mon_vals no callback set 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 10 monclient: dump: 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:48.968 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.941+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe4dc010460 con 0x7fe4e4134f30 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.942+0000 7fe4e93fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.942+0000 7fe4e93fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.942+0000 7fe4e93fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:48.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.942+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c4005080 con 0x7fe4e4134f30 2024-09-16T04:10:48.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.942+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(150..150 src has 1..150) ==== 8396+0+0 (unknown 1287840509 0 0) 0x7fe4dc048430 con 0x7fe4e4134f30 2024-09-16T04:10:48.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.942+0000 7fe4e2ffd640 10 monclient: get_auth_request con 0x7fe4d0038ef0 auth_method 0 2024-09-16T04:10:48.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.946+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe4e4141560 con 0x7fe4e4134f30 2024-09-16T04:10:48.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.946+0000 7fe4cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:48.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:48.946+0000 7fe4cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:49.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.120+0000 7fe4e93fd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T04:10:49.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.120+0000 7fe4e93fd640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T04:10:49.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.120+0000 7fe4e93fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:49.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.120+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) -- 0x7fe4c4005370 con 0x7fe4e4134f30 2024-09-16T04:10:49.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.850+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v151) ==== 134+0+0 (unknown 3734060805 0 0) 0x7fe4dc016140 con 0x7fe4e4134f30 2024-09-16T04:10:49.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.850+0000 7fe4cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T04:10:49.852 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.850+0000 7fe4cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:49.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.961+0000 7fe4e93fd640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T04:10:49.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.961+0000 7fe4e93fd640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T04:10:49.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.961+0000 7fe4e93fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:49.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:49.961+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) -- 0x7fe4c4005c60 con 0x7fe4e4134f30 2024-09-16T04:10:50.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.857+0000 7fe4cb7fe640 1 -- 172.21.15.171:0/2525988689 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v152) ==== 134+0+0 (unknown 2469453746 0 0) 0x7fe4dc015a40 con 0x7fe4e4134f30 2024-09-16T04:10:50.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.857+0000 7fe4cb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-16T04:10:50.859 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.857+0000 7fe4cb7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:50.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 >> v1:172.21.15.171:6812/7327273 conn(0x7fe4d0038ef0 legacy=0x7fe4d003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:50.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 10 monclient: shutdown 2024-09-16T04:10:50.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:50.862 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 >> v1:172.21.15.171:6789/0 conn(0x7fe4e4134f30 legacy=0x7fe4e41d42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:50.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 shutdown_connections 2024-09-16T04:10:50.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 >> 172.21.15.171:0/2525988689 conn(0x7fe4e40a1e90 msgr2=0x7fe4e4140630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:50.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 shutdown_connections 2024-09-16T04:10:50.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.860+0000 7fe4e93fd640 1 -- 172.21.15.171:0/2525988689 wait complete. 2024-09-16T04:10:50.874 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.956+0000 7f1c4c3a6640 10 monclient: get_monmap_and_config 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.956+0000 7f1c4c3a6640 10 monclient: build_initial_monmap 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.956+0000 7f1c4c3a6640 10 monclient: monmap: 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:50.956996+0000 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:50.956996+0000 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:50.958 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:50.959 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.956+0000 7f1c4c3a6640 1 Processor -- start 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.956+0000 7f1c4c3a6640 1 -- start start 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.956+0000 7f1c4c3a6640 10 monclient: init 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c441327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4409eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c44139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409ef30 con 0x7f1c441327c0 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409d880 con 0x7f1c4409eb20 2024-09-16T04:10:50.966 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409db60 con 0x7f1c44139ea0 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4c3a6640 10 monclient(hunting): authenticate will time out at 1026.462596s 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49941640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1c4409eb20 0x7f1c44138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45246/0 (socket says 172.21.15.171:45246) 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c4a943640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f1c44139ea0 legacy=0x7f1c4413c290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49941640 1 -- 172.21.15.171:0/341175491 learned_addr learned my addr 172.21.15.171:0/341175491 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689288983 0 0) 0x7f1c4409d880 con 0x7f1c4409eb20 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49140640 10 monclient(hunting): _negotiate 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49140640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49140640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49140640 10 monclient(hunting): my global_id is 5773 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.957+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c30003520 con 0x7f1c4409eb20 2024-09-16T04:10:50.967 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1499634499 0 0) 0x7f1c4409ef30 con 0x7f1c441327c0 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient(hunting): _negotiate 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient(hunting): my global_id is 64212 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4409d880 con 0x7f1c441327c0 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3029858982 0 0) 0x7f1c30003520 con 0x7f1c4409eb20 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 >> v1:172.21.15.148:6790/0 conn(0x7f1c44139ea0 legacy=0x7f1c4413c290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 >> v1:172.21.15.148:6789/0 conn(0x7f1c441327c0 legacy=0x7f1c44132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: _finish_hunting 0 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 monclient: found mon.noname-c 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:50.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4409de40 con 0x7f1c4409eb20 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: _finish_auth 0 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: _check_auth_tickets 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 5 monclient: authenticate success, global_id 5773 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1c34001610 con 0x7f1c4409eb20 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: dump: 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:50.969 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c340018c0 con 0x7f1c4409eb20 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 1 -- 172.21.15.171:0/341175491 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1c34001c90 con 0x7f1c4409eb20 2024-09-16T04:10:50.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c49140640 10 monclient: dump: 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:50.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 4 set_mon_vals no callback set 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 10 monclient: shutdown 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/341175491 >> v1:172.21.15.171:6789/0 conn(0x7f1c4409eb20 legacy=0x7f1c44138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/341175491 shutdown_connections 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.958+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/341175491 >> 172.21.15.171:0/341175491 conn(0x7f1c440a5370 msgr2=0x7f1c440a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/341175491 shutdown_connections 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/341175491 wait complete. 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 10 monclient: build_initial_monmap 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 10 monclient: monmap: 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:50.972 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 1 Processor -- start 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 1 -- start start 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.959+0000 7f1c4c3a6640 10 monclient: init 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4409eb20 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:50.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c441327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c44139ea0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c44137740 con 0x7f1c441327c0 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c44071ef0 con 0x7f1c44139ea0 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4413d620 con 0x7f1c4409eb20 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4c3a6640 10 monclient(hunting): authenticate will time out at 1026.465403s 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4a142640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f1c4409eb20 legacy=0x7f1c440710d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4a943640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1c44139ea0 0x7f1c441d1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45256/0 (socket says 172.21.15.171:45256) 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c4a943640 1 -- 172.21.15.171:0/3752246490 learned_addr learned my addr 172.21.15.171:0/3752246490 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 210814725 0 0) 0x7f1c44071ef0 con 0x7f1c44139ea0 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:50.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): my global_id is 5776 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c28003520 con 0x7f1c44139ea0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 214348010 0 0) 0x7f1c44137740 con 0x7f1c441327c0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): my global_id is 64215 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c44071ef0 con 0x7f1c441327c0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1943418410 0 0) 0x7f1c28003520 con 0x7f1c44139ea0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c44137740 con 0x7f1c44139ea0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1c3c004440 con 0x7f1c44139ea0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:50.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): dump: 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.960+0000 7f1c3affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:50.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 259274201 0 0) 0x7f1c44137740 con 0x7f1c44139ea0 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 >> v1:172.21.15.148:6790/0 conn(0x7f1c4409eb20 legacy=0x7f1c440710d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 >> v1:172.21.15.148:6789/0 conn(0x7f1c441327c0 legacy=0x7f1c440717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 monclient: found mon.b 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4406ce80 con 0x7f1c44139ea0 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 5 monclient: authenticate success, global_id 5776 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 monclient: _renew_subs 2024-09-16T04:10:50.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c440727e0 con 0x7f1c44139ea0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 monclient: _renew_subs 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c44072a20 con 0x7f1c44139ea0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c3c0046a0 con 0x7f1c44139ea0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1c3c0049f0 con 0x7f1c44139ea0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:50.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4893f640 4 set_mon_vals no callback set 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 10 monclient: dump: 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:50.979 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1c3c004c10 con 0x7f1c44139ea0 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.961+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c2c005080 con 0x7f1c44139ea0 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.962+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(152..152 src has 1..152) ==== 8396+0+0 (unknown 76588377 0 0) 0x7f1c3c048320 con 0x7f1c44139ea0 2024-09-16T04:10:50.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.962+0000 7f1c4a142640 10 monclient: get_auth_request con 0x7f1c28038f60 auth_method 0 2024-09-16T04:10:50.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.966+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1c3c015f10 con 0x7f1c44139ea0 2024-09-16T04:10:50.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.966+0000 7f1c3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:50.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:50.966+0000 7f1c3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:51.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.135+0000 7f1c4c3a6640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T04:10:51.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.135+0000 7f1c4c3a6640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T04:10:51.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.135+0000 7f1c4c3a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:51.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.135+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) -- 0x7f1c2c005370 con 0x7f1c44139ea0 2024-09-16T04:10:51.863 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.862+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v153) ==== 137+0+0 (unknown 2156202667 0 0) 0x7f1c3c015810 con 0x7f1c44139ea0 2024-09-16T04:10:51.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.862+0000 7f1c3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T04:10:51.864 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.862+0000 7f1c3affd640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-16T04:10:51.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.968+0000 7f1c4c3a6640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T04:10:51.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.968+0000 7f1c4c3a6640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T04:10:51.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.968+0000 7f1c4c3a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:51.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:51.968+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) -- 0x7f1c2c000bf0 con 0x7f1c44139ea0 2024-09-16T04:10:52.866 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.864+0000 7f1c3affd640 1 -- 172.21.15.171:0/3752246490 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v154) ==== 241+0+0 (unknown 316544665 0 0) 0x7f1c3c019b60 con 0x7f1c44139ea0 2024-09-16T04:10:52.867 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.864+0000 7f1c3affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-16T04:10:52.867 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.864+0000 7f1c3affd640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T04:10:52.867 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-16T04:10:52.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 >> v1:172.21.15.171:6812/7327273 conn(0x7f1c28038f60 legacy=0x7f1c2803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:52.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 10 monclient: shutdown 2024-09-16T04:10:52.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:52.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 >> v1:172.21.15.171:6789/0 conn(0x7f1c44139ea0 legacy=0x7f1c441d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:52.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 shutdown_connections 2024-09-16T04:10:52.869 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 >> 172.21.15.171:0/3752246490 conn(0x7f1c440a5370 msgr2=0x7f1c440a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:52.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 shutdown_connections 2024-09-16T04:10:52.870 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.867+0000 7f1c4c3a6640 1 -- 172.21.15.171:0/3752246490 wait complete. 2024-09-16T04:10:52.880 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:10:52.880 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:10:52.880 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep fish 2024-09-16T04:10:52.880 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '10\.' 2024-09-16T04:10:52.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.960+0000 7fb3ab59e640 10 monclient: get_monmap_and_config 2024-09-16T04:10:52.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.960+0000 7fb3ab59e640 10 monclient: build_initial_monmap 2024-09-16T04:10:52.962 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.960+0000 7fb3ab59e640 10 monclient: monmap: 2024-09-16T04:10:52.962 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:52.962 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:52.962 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:52.960881+0000 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:52.960881+0000 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.960+0000 7fb3ab59e640 1 Processor -- start 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.960+0000 7fb3ab59e640 1 -- start start 2024-09-16T04:10:52.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.960+0000 7fb3ab59e640 10 monclient: init 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ac0a22f0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ac0a19f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ac13cfb0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac0a4c20 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac0a0740 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac0a0a20 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.971 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient(hunting): authenticate will time out at 1028.466203s 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3aad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb3ac13cfb0 0x7fb3ac13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45272/0 (socket says 172.21.15.171:45272) 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3aad9d640 1 -- 172.21.15.171:0/646136912 learned_addr learned my addr 172.21.15.171:0/646136912 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3aa59c640 1 -- 172.21.15.171:0/646136912 >> v1:172.21.15.148:6790/0 conn(0x7fb3ac0a22f0 legacy=0x7fb3ac0a26d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200830904 0 0) 0x7fb3ac0a0740 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient(hunting): my global_id is 5779 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb398003520 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2646600684 0 0) 0x7fb398003520 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.972 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 >> v1:172.21.15.148:6790/0 conn(0x7fb3ac0a22f0 legacy=0x7fb3ac0a26d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 >> v1:172.21.15.148:6789/0 conn(0x7fb3ac0a19f0 legacy=0x7fb3ac13b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac0a47a0 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: _finish_auth 0 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 5 monclient: authenticate success, global_id 5779 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3a4001990 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: dump: 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:52.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3a4001bf0 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 1 -- 172.21.15.171:0/646136912 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3a4001fc0 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3ab59e640 4 set_mon_vals no callback set 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.961+0000 7fb3a9d9b640 10 monclient: dump: 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:52.975 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 10 monclient: shutdown 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 1 -- 172.21.15.171:0/646136912 >> v1:172.21.15.171:6789/0 conn(0x7fb3ac13cfb0 legacy=0x7fb3ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 1 -- 172.21.15.171:0/646136912 shutdown_connections 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 1 -- 172.21.15.171:0/646136912 >> 172.21.15.171:0/646136912 conn(0x7fb3ac0a9440 msgr2=0x7fb3ac0a5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:52.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 1 -- 172.21.15.171:0/646136912 shutdown_connections 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.962+0000 7fb3ab59e640 1 -- 172.21.15.171:0/646136912 wait complete. 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.963+0000 7fb3ab59e640 10 monclient: build_initial_monmap 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.963+0000 7fb3ab59e640 10 monclient: monmap: 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.963+0000 7fb3ab59e640 1 Processor -- start 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.963+0000 7fb3ab59e640 1 -- start start 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.963+0000 7fb3ab59e640 10 monclient: init 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ac0a19f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ac0a22f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ac13cfb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac1407a0 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac123ae0 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac120f30 con 0x7fb3ac13cfb0 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient(hunting): authenticate will time out at 1028.469193s 2024-09-16T04:10:52.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3aad9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb3ac13cfb0 legacy=0x7fb3ac1d39a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3aa59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb3ac0a19f0 0x7fb3ac120110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45276/0 (socket says 172.21.15.171:45276) 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3aa59c640 1 -- 172.21.15.171:0/2282030356 learned_addr learned my addr 172.21.15.171:0/2282030356 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131681111 0 0) 0x7fb3ac123ae0 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): my global_id is 5782 2024-09-16T04:10:52.979 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb38c003520 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2693680659 0 0) 0x7fb3ac0a4c20 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:52.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:52.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:52.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): my global_id is 64221 2024-09-16T04:10:52.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3ac123ae0 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1339988548 0 0) 0x7fb38c003520 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3ac0a4c20 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb39c004180 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): dump: 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.981 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2268940433 0 0) 0x7fb3ac123ae0 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb38c003520 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3940017d0 con 0x7fb3ac0a22f0 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient(hunting): dump: 2024-09-16T04:10:52.982 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3602938401 0 0) 0x7fb3ac0a4c20 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 >> v1:172.21.15.148:6790/0 conn(0x7fb3ac13cfb0 legacy=0x7fb3ac1d39a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 >> v1:172.21.15.148:6789/0 conn(0x7fb3ac0a22f0 legacy=0x7fb3ac120820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:52.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 monclient: found mon.b 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac1240a0 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 5 monclient: authenticate success, global_id 5782 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient: _renew_subs 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac13bdf0 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient: _renew_subs 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:52.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.964+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3ac123dc0 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb39c0043e0 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb39c004730 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a959a640 4 set_mon_vals no callback set 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 10 monclient: dump: 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:52.985 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb39c004950 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3ab59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3ab59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.965+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb390005080 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.966+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(154..154 src has 1..154) ==== 9061+0+0 (unknown 3139070324 0 0) 0x7fb39c048300 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.966+0000 7fb3a1d9b640 10 monclient: get_auth_request con 0x7fb38c038f10 auth_method 0 2024-09-16T04:10:52.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.970+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb39c015c60 con 0x7fb3ac0a19f0 2024-09-16T04:10:52.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.970+0000 7fb3a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:52.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:52.970+0000 7fb3a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:53.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3ab59e640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3ab59e640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fb390005370 con 0x7fb3ac0a19f0 2024-09-16T04:10:53.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) ==== 83+0+2175 (unknown 2736296030 0 2286913804) 0x7fb39c015560 con 0x7fb3ac0a19f0 2024-09-16T04:10:53.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.143+0000 7fb3a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:53.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.248+0000 7fb3ab59e640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.248+0000 7fb3ab59e640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.248+0000 7fb3ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.248+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fb390001090 con 0x7fb3ac0a19f0 2024-09-16T04:10:53.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.248+0000 7fb3a37fe640 1 -- 172.21.15.171:0/2282030356 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) ==== 83+0+2175 (unknown 2736296030 0 2286913804) 0x7fb39c0198b0 con 0x7fb3ac0a19f0 2024-09-16T04:10:53.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.249+0000 7fb3a37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.249+0000 7fb3a37fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 >> v1:172.21.15.171:6812/7327273 conn(0x7fb38c038f10 legacy=0x7fb38c03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 10 monclient: shutdown 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 >> v1:172.21.15.171:6789/0 conn(0x7fb3ac0a19f0 legacy=0x7fb3ac120110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 shutdown_connections 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 >> 172.21.15.171:0/2282030356 conn(0x7fb3ac0a9440 msgr2=0x7fb3ac0a3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 shutdown_connections 2024-09-16T04:10:53.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.251+0000 7fb3ab59e640 1 -- 172.21.15.171:0/2282030356 wait complete. 2024-09-16T04:10:53.262 INFO:tasks.workunit.client.0.smithi171.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:53.262 INFO:tasks.workunit.client.0.smithi171.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:53.263 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:10:53.263 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:10:53.263 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep fish 2024-09-16T04:10:53.263 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '11\.' 2024-09-16T04:10:53.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.340+0000 7f2188f91640 10 monclient: get_monmap_and_config 2024-09-16T04:10:53.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.340+0000 7f2188f91640 10 monclient: build_initial_monmap 2024-09-16T04:10:53.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.340+0000 7f2188f91640 10 monclient: monmap: 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:53.341228+0000 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:53.341228+0000 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.340+0000 7f2188f91640 1 Processor -- start 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 1 -- start start 2024-09-16T04:10:53.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient: init 2024-09-16T04:10:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient(hunting): picked mon.noname-a con 0x7f218409f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient(hunting): picked mon.noname-c con 0x7f21840a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:53.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient(hunting): picked mon.noname-b con 0x7f218409e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21840a1260 con 0x7f218409f640 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f218409d0c0 con 0x7f21840a0130 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f218409d3a0 con 0x7f218409e1d0 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2188f91640 10 monclient(hunting): authenticate will time out at 1028.846694s 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2181d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f21840a0130 0x7f2184138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45280/0 (socket says 172.21.15.171:45280) 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2181d9b640 1 -- 172.21.15.171:0/2012101704 learned_addr learned my addr 172.21.15.171:0/2012101704 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.341+0000 7f2182d9d640 1 -- 172.21.15.171:0/2012101704 >> v1:172.21.15.148:6790/0 conn(0x7f218409e1d0 legacy=0x7f218413bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4016298488 0 0) 0x7f218409d0c0 con 0x7f21840a0130 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): my global_id is 5785 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f216c003520 con 0x7f21840a0130 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3820740781 0 0) 0x7f21840a1260 con 0x7f218409f640 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient(hunting): my global_id is 64224 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f218409d0c0 con 0x7f218409f640 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2815356469 0 0) 0x7f216c003520 con 0x7f21840a0130 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 >> v1:172.21.15.148:6790/0 conn(0x7f218409e1d0 legacy=0x7f218413bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 >> v1:172.21.15.148:6789/0 conn(0x7f218409f640 legacy=0x7f218409fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: _finish_hunting 0 2024-09-16T04:10:53.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 monclient: found mon.noname-c 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218409d680 con 0x7f21840a0130 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: _finish_auth 0 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2178001610 con 0x7f21840a0130 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 5 monclient: authenticate success, global_id 5785 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: dump: 2024-09-16T04:10:53.353 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f21780018c0 con 0x7f21840a0130 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:53.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 1 -- 172.21.15.171:0/2012101704 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2178001c90 con 0x7f21840a0130 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f218159a640 10 monclient: dump: 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.355 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 4 set_mon_vals no callback set 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 10 monclient: shutdown 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 1 -- 172.21.15.171:0/2012101704 >> v1:172.21.15.171:6789/0 conn(0x7f21840a0130 legacy=0x7f2184138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.342+0000 7f2188f91640 1 -- 172.21.15.171:0/2012101704 shutdown_connections 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.343+0000 7f2188f91640 1 -- 172.21.15.171:0/2012101704 >> 172.21.15.171:0/2012101704 conn(0x7f21840a52a0 msgr2=0x7f21840a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.343+0000 7f2188f91640 1 -- 172.21.15.171:0/2012101704 shutdown_connections 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.343+0000 7f2188f91640 1 -- 172.21.15.171:0/2012101704 wait complete. 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.343+0000 7f2188f91640 10 monclient: build_initial_monmap 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.343+0000 7f2188f91640 10 monclient: monmap: 2024-09-16T04:10:53.356 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 1 Processor -- start 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 1 -- start start 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient: init 2024-09-16T04:10:53.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient(hunting): picked mon.noname-b con 0x7f218409e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient(hunting): picked mon.noname-c con 0x7f218409f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient(hunting): picked mon.noname-a con 0x7f21840a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21841c13a0 con 0x7f21840a0130 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21841a5b30 con 0x7f218409e1d0 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21841a85f0 con 0x7f218409f640 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2188f91640 10 monclient(hunting): authenticate will time out at 1028.849558s 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f218259c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f218409e1d0 0x7f21841c0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45284/0 (socket says 172.21.15.171:45284) 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f218259c640 1 -- 172.21.15.171:0/3783052014 learned_addr learned my addr 172.21.15.171:0/3783052014 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:53.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2181d9b640 1 -- 172.21.15.171:0/3783052014 >> v1:172.21.15.148:6790/0 conn(0x7f218409f640 legacy=0x7f21841cbfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058986033 0 0) 0x7f21841a5b30 con 0x7f218409e1d0 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): my global_id is 5788 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2164003520 con 0x7f218409e1d0 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 984756171 0 0) 0x7f21841c13a0 con 0x7f21840a0130 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 10 monclient(hunting): my global_id is 64227 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21841a5b30 con 0x7f21840a0130 2024-09-16T04:10:53.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.344+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2864468372 0 0) 0x7f2164003520 con 0x7f218409e1d0 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21841c13a0 con 0x7f218409e1d0 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2174001910 con 0x7f218409e1d0 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.360 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2686166900 0 0) 0x7f21841a5b30 con 0x7f21840a0130 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2164003520 con 0x7f21840a0130 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f217c0043b0 con 0x7f21840a0130 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient(hunting): dump: 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.361 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2479674610 0 0) 0x7f21841c13a0 con 0x7f218409e1d0 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 >> v1:172.21.15.148:6790/0 conn(0x7f218409f640 legacy=0x7f21841cbfc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 >> v1:172.21.15.148:6789/0 conn(0x7f21840a0130 legacy=0x7f21841cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 monclient: found mon.b 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:53.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2184072140 con 0x7f218409e1d0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 5 monclient: authenticate success, global_id 5788 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 10 monclient: _renew_subs 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21841a6320 con 0x7f218409e1d0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 10 monclient: _renew_subs 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f21841a70a0 con 0x7f218409e1d0 2024-09-16T04:10:53.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2174001ce0 con 0x7f218409e1d0 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2174003a60 con 0x7f218409e1d0 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2180d99640 4 set_mon_vals no callback set 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.345+0000 7f2172ffd640 10 monclient: dump: 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.364 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2174003c80 con 0x7f218409e1d0 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2188f91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2188f91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2188f91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2168005080 con 0x7f218409e1d0 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(154..154 src has 1..154) ==== 9061+0+0 (unknown 3139070324 0 0) 0x7f2174048390 con 0x7f218409e1d0 2024-09-16T04:10:53.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.346+0000 7f2181d9b640 10 monclient: get_auth_request con 0x7f2164038ea0 auth_method 0 2024-09-16T04:10:53.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.350+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2174015d50 con 0x7f218409e1d0 2024-09-16T04:10:53.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.350+0000 7f2172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:53.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.350+0000 7f2172ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:53.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2188f91640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2188f91640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2188f91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f2168005370 con 0x7f218409e1d0 2024-09-16T04:10:53.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) ==== 83+0+2175 (unknown 2736296030 0 2286913804) 0x7f2174015650 con 0x7f218409e1d0 2024-09-16T04:10:53.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2172ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.521+0000 7f2172ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:53.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2188f91640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2188f91640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2188f91640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f21680065c0 con 0x7f218409e1d0 2024-09-16T04:10:53.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2172ffd640 1 -- 172.21.15.171:0/3783052014 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) ==== 83+0+2175 (unknown 2736296030 0 2286913804) 0x7f217401e200 con 0x7f218409e1d0 2024-09-16T04:10:53.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2172ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.625+0000 7f2172ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.627+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 >> v1:172.21.15.171:6812/7327273 conn(0x7f2164038ea0 legacy=0x7f216403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.627+0000 7f2188f91640 10 monclient: shutdown 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.627+0000 7f2188f91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.627+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 >> v1:172.21.15.171:6789/0 conn(0x7f218409e1d0 legacy=0x7f21841c0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.628+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 shutdown_connections 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.628+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 >> 172.21.15.171:0/3783052014 conn(0x7f21840a52a0 msgr2=0x7f2184070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.628+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 shutdown_connections 2024-09-16T04:10:53.630 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.628+0000 7f2188f91640 1 -- 172.21.15.171:0/3783052014 wait complete. 2024-09-16T04:10:53.638 INFO:tasks.workunit.client.0.smithi171.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:53.639 INFO:tasks.workunit.client.0.smithi171.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:53.639 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:10:53.640 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:10:53.640 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep fish 2024-09-16T04:10:53.640 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '12\.' 2024-09-16T04:10:53.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.715+0000 7f7159dc6640 10 monclient: get_monmap_and_config 2024-09-16T04:10:53.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.715+0000 7f7159dc6640 10 monclient: build_initial_monmap 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.715+0000 7f7159dc6640 10 monclient: monmap: 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:53.716705+0000 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:53.716705+0000 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:53.718 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.716+0000 7f7159dc6640 1 Processor -- start 2024-09-16T04:10:53.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.716+0000 7f7159dc6640 1 -- start start 2024-09-16T04:10:53.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.716+0000 7f7159dc6640 10 monclient: init 2024-09-16T04:10:53.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:53.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:53.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f7154132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:53.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f715409ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:53.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f7154132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:53.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71541332b0 con 0x7f715409ec20 2024-09-16T04:10:53.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7154134310 con 0x7f7154132e60 2024-09-16T04:10:53.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715409d980 con 0x7f7154132a80 2024-09-16T04:10:53.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:53.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient(hunting): authenticate will time out at 1029.222207s 2024-09-16T04:10:53.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7153fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7154132e60 0x7f715413c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45300/0 (socket says 172.21.15.171:45300) 2024-09-16T04:10:53.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7153fff640 1 -- 172.21.15.171:0/2188046884 learned_addr learned my addr 172.21.15.171:0/2188046884 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:53.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71537fe640 1 -- 172.21.15.171:0/2188046884 >> v1:172.21.15.148:6790/0 conn(0x7f7154132a80 legacy=0x7f71541334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:53.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409995620 0 0) 0x7f7154134310 con 0x7f7154132e60 2024-09-16T04:10:53.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient(hunting): my global_id is 5791 2024-09-16T04:10:53.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f713c003520 con 0x7f7154132e60 2024-09-16T04:10:53.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3641776750 0 0) 0x7f713c003520 con 0x7f7154132e60 2024-09-16T04:10:53.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 >> v1:172.21.15.148:6790/0 conn(0x7f7154132a80 legacy=0x7f71541334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:53.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 >> v1:172.21.15.148:6789/0 conn(0x7f715409ec20 legacy=0x7f7154133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:53.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:53.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:53.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715409dc60 con 0x7f7154132e60 2024-09-16T04:10:53.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:53.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:53.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:53.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f714c001610 con 0x7f7154132e60 2024-09-16T04:10:53.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 5 monclient: authenticate success, global_id 5791 2024-09-16T04:10:53.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:53.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:53.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:53.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: dump: 2024-09-16T04:10:53.743 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.743 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.743 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.743 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.744 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.744 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.744 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.744 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.744 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.744 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:53.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f714c001900 con 0x7f7154132e60 2024-09-16T04:10:53.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:53.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 1 -- 172.21.15.171:0/2188046884 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f714c001cd0 con 0x7f7154132e60 2024-09-16T04:10:53.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:53.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f7159dc6640 4 set_mon_vals no callback set 2024-09-16T04:10:53.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:53.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:53.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.717+0000 7f71527fc640 10 monclient: dump: 2024-09-16T04:10:53.746 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.747 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.747 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.747 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.747 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.747 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.748 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.748 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.748 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.748 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.748 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 10 monclient: shutdown 2024-09-16T04:10:53.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:53.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 1 -- 172.21.15.171:0/2188046884 >> v1:172.21.15.171:6789/0 conn(0x7f7154132e60 legacy=0x7f715413c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 1 -- 172.21.15.171:0/2188046884 shutdown_connections 2024-09-16T04:10:53.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 1 -- 172.21.15.171:0/2188046884 >> 172.21.15.171:0/2188046884 conn(0x7f71540a52a0 msgr2=0x7f71540a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:53.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 1 -- 172.21.15.171:0/2188046884 shutdown_connections 2024-09-16T04:10:53.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 1 -- 172.21.15.171:0/2188046884 wait complete. 2024-09-16T04:10:53.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 10 monclient: build_initial_monmap 2024-09-16T04:10:53.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.718+0000 7f7159dc6640 10 monclient: monmap: 2024-09-16T04:10:53.750 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:53.750 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:53.750 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:53.750 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:53.751 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:53.751 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.751 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:53.751 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:53.751 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:53.752 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 1 Processor -- start 2024-09-16T04:10:53.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 1 -- start start 2024-09-16T04:10:53.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient: init 2024-09-16T04:10:53.752 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:53.753 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:53.753 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f715409ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:53.753 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f7154132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:53.753 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f7154132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:53.753 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715406ceb0 con 0x7f7154132a80 2024-09-16T04:10:53.754 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715411f000 con 0x7f7154132e60 2024-09-16T04:10:53.754 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715411f2e0 con 0x7f715409ec20 2024-09-16T04:10:53.754 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:53.754 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f7159dc6640 10 monclient(hunting): authenticate will time out at 1029.224956s 2024-09-16T04:10:53.754 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.719+0000 7f71537fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f715409ec20 legacy=0x7f71541c6bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:53.754 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7153fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7154132e60 0x7f71541cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45308/0 (socket says 172.21.15.171:45308) 2024-09-16T04:10:53.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7153fff640 1 -- 172.21.15.171:0/443681450 learned_addr learned my addr 172.21.15.171:0/443681450 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:53.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741494587 0 0) 0x7f715411f000 con 0x7f7154132e60 2024-09-16T04:10:53.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.755 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): my global_id is 5794 2024-09-16T04:10:53.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7140003520 con 0x7f7154132e60 2024-09-16T04:10:53.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591053943 0 0) 0x7f7148001230 con 0x7f7154132a80 2024-09-16T04:10:53.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:53.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:53.757 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:53.757 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): my global_id is 64233 2024-09-16T04:10:53.757 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f715411f000 con 0x7f7154132a80 2024-09-16T04:10:53.757 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3186245577 0 0) 0x7f7140003520 con 0x7f7154132e60 2024-09-16T04:10:53.757 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7148001230 con 0x7f7154132e60 2024-09-16T04:10:53.757 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f714c001430 con 0x7f7154132e60 2024-09-16T04:10:53.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:53.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:53.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): dump: 2024-09-16T04:10:53.758 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.758 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.759 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.759 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.759 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.759 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.759 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.759 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.760 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.760 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:53.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435293151 0 0) 0x7f715411f000 con 0x7f7154132a80 2024-09-16T04:10:53.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7140003520 con 0x7f7154132a80 2024-09-16T04:10:53.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7148001a00 con 0x7f7154132a80 2024-09-16T04:10:53.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:53.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:53.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient(hunting): dump: 2024-09-16T04:10:53.761 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.762 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.762 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.762 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.762 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.762 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.762 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.763 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.763 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.763 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2184759694 0 0) 0x7f7148001230 con 0x7f7154132e60 2024-09-16T04:10:53.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 >> v1:172.21.15.148:6790/0 conn(0x7f715409ec20 legacy=0x7f71541c6bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:53.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 >> v1:172.21.15.148:6789/0 conn(0x7f7154132a80 legacy=0x7f7154131c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:53.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:53.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 monclient: found mon.b 2024-09-16T04:10:53.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:53.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71540a3b90 con 0x7f7154132e60 2024-09-16T04:10:53.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:53.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:53.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:53.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 5 monclient: authenticate success, global_id 5794 2024-09-16T04:10:53.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 10 monclient: _renew_subs 2024-09-16T04:10:53.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f715409c690 con 0x7f7154132e60 2024-09-16T04:10:53.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 10 monclient: _renew_subs 2024-09-16T04:10:53.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7154132310 con 0x7f7154132e60 2024-09-16T04:10:53.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f714c004cc0 con 0x7f7154132e60 2024-09-16T04:10:53.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:53.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f714c001900 con 0x7f7154132e60 2024-09-16T04:10:53.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:53.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f7151ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:53.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:53.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.720+0000 7f713bfff640 10 monclient: dump: 2024-09-16T04:10:53.768 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:53.768 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:53.768 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.769 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:53.769 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:53.769 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:53.769 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:53.769 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:53.770 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:53.770 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:53.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f714c0102a0 con 0x7f7154132e60 2024-09-16T04:10:53.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f7159dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:53.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f7159dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:53.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f7159dc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7134005080 con 0x7f7154132e60 2024-09-16T04:10:53.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(154..154 src has 1..154) ==== 9061+0+0 (unknown 3139070324 0 0) 0x7f714c048390 con 0x7f7154132e60 2024-09-16T04:10:53.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.721+0000 7f71537fe640 10 monclient: get_auth_request con 0x7f7140039090 auth_method 0 2024-09-16T04:10:53.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.726+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f714c015db0 con 0x7f7154132e60 2024-09-16T04:10:53.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.726+0000 7f713bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:53.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.726+0000 7f713bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:53.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f7159dc6640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f7159dc6640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f7159dc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:53.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f7134005370 con 0x7f7154132e60 2024-09-16T04:10:53.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) ==== 83+0+2175 (unknown 2736296030 0 2286913804) 0x7f714c0156b0 con 0x7f7154132e60 2024-09-16T04:10:53.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f713bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:53.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:53.899+0000 7f713bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:54.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f7159dc6640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:54.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f7159dc6640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:54.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f7159dc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f7134005ab0 con 0x7f7154132e60 2024-09-16T04:10:54.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f713bfff640 1 -- 172.21.15.171:0/443681450 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v154) ==== 83+0+2175 (unknown 2736296030 0 2286913804) 0x7f714c01e200 con 0x7f7154132e60 2024-09-16T04:10:54.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f713bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:54.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.002+0000 7f713bfff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:54.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.004+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 >> v1:172.21.15.171:6812/7327273 conn(0x7f7140039090 legacy=0x7f714003b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:54.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.004+0000 7f7159dc6640 10 monclient: shutdown 2024-09-16T04:10:54.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.004+0000 7f7159dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:54.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.004+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 >> v1:172.21.15.171:6789/0 conn(0x7f7154132e60 legacy=0x7f71541cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:54.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.004+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 shutdown_connections 2024-09-16T04:10:54.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.004+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 >> 172.21.15.171:0/443681450 conn(0x7f71540a52a0 msgr2=0x7f71540a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:54.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.005+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 shutdown_connections 2024-09-16T04:10:54.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.005+0000 7f7159dc6640 1 -- 172.21.15.171:0/443681450 wait complete. 2024-09-16T04:10:54.015 INFO:tasks.workunit.client.0.smithi171.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:54.016 INFO:tasks.workunit.client.0.smithi171.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:54.016 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T04:10:54.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.093+0000 7fed444e6640 10 monclient: get_monmap_and_config 2024-09-16T04:10:54.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.093+0000 7fed444e6640 10 monclient: build_initial_monmap 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.093+0000 7fed444e6640 10 monclient: monmap: 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:54.094051+0000 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:54.094051+0000 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.096 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:54.097 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:54.097 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:54.097 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.093+0000 7fed444e6640 1 Processor -- start 2024-09-16T04:10:54.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.093+0000 7fed444e6640 1 -- start start 2024-09-16T04:10:54.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.093+0000 7fed444e6640 10 monclient: init 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient(hunting): picked mon.noname-b con 0x7fed3c09ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient(hunting): picked mon.noname-a con 0x7fed3c0a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient(hunting): picked mon.noname-c con 0x7fed3c09e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c09fbf0 con 0x7fed3c0a0750 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c0a0e10 con 0x7fed3c09e570 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c09f640 con 0x7fed3c09ba30 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:54.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed444e6640 10 monclient(hunting): authenticate will time out at 1029.599539s 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed42a83640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fed3c09e570 0x7fed3c13d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45318/0 (socket says 172.21.15.171:45318) 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed42282640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fed3c09ba30 legacy=0x7fed3c0a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed42a83640 1 -- 172.21.15.171:0/1802892149 learned_addr learned my addr 172.21.15.171:0/1802892149 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351562497 0 0) 0x7fed3c0a0e10 con 0x7fed3c09e570 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed41280640 10 monclient(hunting): _negotiate 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed41280640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed41280640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed41280640 10 monclient(hunting): my global_id is 5797 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.094+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed28003520 con 0x7fed3c09e570 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269996256 0 0) 0x7fed3c09fbf0 con 0x7fed3c0a0750 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient(hunting): _negotiate 2024-09-16T04:10:54.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient(hunting): my global_id is 64236 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed3c0a0e10 con 0x7fed3c0a0750 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2410457091 0 0) 0x7fed28003520 con 0x7fed3c09e570 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 >> v1:172.21.15.148:6790/0 conn(0x7fed3c09ba30 legacy=0x7fed3c0a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 >> v1:172.21.15.148:6789/0 conn(0x7fed3c0a0750 legacy=0x7fed3c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: _finish_hunting 0 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 monclient: found mon.noname-c 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c09f810 con 0x7fed3c09e570 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: _finish_auth 0 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: _check_auth_tickets 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed34001990 con 0x7fed3c09e570 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: dump: 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.110 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 5 monclient: authenticate success, global_id 5797 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed34001cb0 con 0x7fed3c09e570 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 1 -- 172.21.15.171:0/1802892149 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed34002080 con 0x7fed3c09e570 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:54.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 4 set_mon_vals no callback set 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed41280640 10 monclient: dump: 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:54.112 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 10 monclient: shutdown 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 1 -- 172.21.15.171:0/1802892149 >> v1:172.21.15.171:6789/0 conn(0x7fed3c09e570 legacy=0x7fed3c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 1 -- 172.21.15.171:0/1802892149 shutdown_connections 2024-09-16T04:10:54.113 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 1 -- 172.21.15.171:0/1802892149 >> 172.21.15.171:0/1802892149 conn(0x7fed3c0a52a0 msgr2=0x7fed3c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 1 -- 172.21.15.171:0/1802892149 shutdown_connections 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.095+0000 7fed444e6640 1 -- 172.21.15.171:0/1802892149 wait complete. 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.096+0000 7fed444e6640 10 monclient: build_initial_monmap 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.096+0000 7fed444e6640 10 monclient: monmap: 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:54.114 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.096+0000 7fed444e6640 1 Processor -- start 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.096+0000 7fed444e6640 1 -- start start 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.096+0000 7fed444e6640 10 monclient: init 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient(hunting): picked mon.noname-b con 0x7fed3c09ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient(hunting): picked mon.noname-c con 0x7fed3c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient(hunting): picked mon.noname-a con 0x7fed3c0a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c12e730 con 0x7fed3c0a0750 2024-09-16T04:10:54.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c12ea10 con 0x7fed3c09ba30 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed3c12c500 con 0x7fed3c09e570 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient(hunting): authenticate will time out at 1029.602262s 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed41a81640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fed3c09e570 legacy=0x7fed3c12bdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed42282640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fed3c09ba30 0x7fed3c12b6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45320/0 (socket says 172.21.15.171:45320) 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed42282640 1 -- 172.21.15.171:0/3472116442 learned_addr learned my addr 172.21.15.171:0/3472116442 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1854687190 0 0) 0x7fed3c12ea10 con 0x7fed3c09ba30 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:54.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): my global_id is 5800 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed20003520 con 0x7fed3c09ba30 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124735251 0 0) 0x7fed3c12e730 con 0x7fed3c0a0750 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): my global_id is 64239 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed3c12ea10 con 0x7fed3c0a0750 2024-09-16T04:10:54.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3707005120 0 0) 0x7fed20003520 con 0x7fed3c09ba30 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed3c12e730 con 0x7fed3c09ba30 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed30004180 con 0x7fed3c09ba30 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): dump: 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:54.118 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1472964837 0 0) 0x7fed3c12ea10 con 0x7fed3c0a0750 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fed20003520 con 0x7fed3c0a0750 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed340017b0 con 0x7fed3c0a0750 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient(hunting): dump: 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.119 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3430439529 0 0) 0x7fed3c12e730 con 0x7fed3c09ba30 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 >> v1:172.21.15.148:6790/0 conn(0x7fed3c09e570 legacy=0x7fed3c12bdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 >> v1:172.21.15.148:6789/0 conn(0x7fed3c0a0750 legacy=0x7fed3c1d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:54.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 monclient: found mon.b 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c12d4f0 con 0x7fed3c09ba30 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient: _finish_auth 0 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 5 monclient: authenticate success, global_id 5800 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient: _renew_subs 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.097+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c132810 con 0x7fed3c09ba30 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 10 monclient: _renew_subs 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fed3c12d0a0 con 0x7fed3c09ba30 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fed30004430 con 0x7fed3c09ba30 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fed30004780 con 0x7fed3c09ba30 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed40a7f640 4 set_mon_vals no callback set 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 10 monclient: dump: 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:54.122 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fed30004920 con 0x7fed3c09ba30 2024-09-16T04:10:54.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.098+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed24005080 con 0x7fed3c09ba30 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.099+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(154..154 src has 1..154) ==== 9061+0+0 (unknown 3139070324 0 0) 0x7fed30048280 con 0x7fed3c09ba30 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.099+0000 7fed41a81640 10 monclient: get_auth_request con 0x7fed20038ef0 auth_method 0 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.103+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fed30015be0 con 0x7fed3c09ba30 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.103+0000 7fed2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:54.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.103+0000 7fed2effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:54.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.275+0000 7fed444e6640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:54.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.275+0000 7fed444e6640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:54.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.275+0000 7fed444e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.275+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7fed24005b80 con 0x7fed3c09ba30 2024-09-16T04:10:54.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.879+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v155) ==== 123+0+0 (unknown 3197670327 0 0) 0x7fed300154e0 con 0x7fed3c09ba30 2024-09-16T04:10:54.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.879+0000 7fed2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:54.881 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.879+0000 7fed2effd640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T04:10:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.983+0000 7fed444e6640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.983+0000 7fed444e6640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.983+0000 7fed444e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:54.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:54.983+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7fed24001090 con 0x7fed3c09ba30 2024-09-16T04:10:55.888 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.886+0000 7fed2effd640 1 -- 172.21.15.171:0/3472116442 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v156) ==== 149+0+0 (unknown 2716580459 0 0) 0x7fed30019830 con 0x7fed3c09ba30 2024-09-16T04:10:55.888 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.886+0000 7fed2effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:10:55.888 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.886+0000 7fed2effd640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T04:10:55.888 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T04:10:55.890 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 >> v1:172.21.15.171:6812/7327273 conn(0x7fed20038ef0 legacy=0x7fed2003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 10 monclient: shutdown 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 >> v1:172.21.15.171:6789/0 conn(0x7fed3c09ba30 legacy=0x7fed3c12b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 shutdown_connections 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 >> 172.21.15.171:0/3472116442 conn(0x7fed3c0a52a0 msgr2=0x7fed3c131bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 shutdown_connections 2024-09-16T04:10:55.891 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.888+0000 7fed444e6640 1 -- 172.21.15.171:0/3472116442 wait complete. 2024-09-16T04:10:55.900 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.979+0000 7fa41f9c4640 10 monclient: get_monmap_and_config 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.979+0000 7fa41f9c4640 10 monclient: build_initial_monmap 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.980+0000 7fa41f9c4640 10 monclient: monmap: 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:55.980768+0000 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:55.980768+0000 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:55.991 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.980+0000 7fa41f9c4640 1 Processor -- start 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.980+0000 7fa41f9c4640 1 -- start start 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.980+0000 7fa41f9c4640 10 monclient: init 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa418132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa41809ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa418132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4181332b0 con 0x7fa41809ec20 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa418134310 con 0x7fa418132e60 2024-09-16T04:10:55.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41809d980 con 0x7fa418132a80 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41f9c4640 10 monclient(hunting): authenticate will time out at 1031.486421s 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41d760640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa418132a80 legacy=0x7fa4181334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41df61640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa418132e60 0x7fa41813c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45324/0 (socket says 172.21.15.171:45324) 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa41df61640 1 -- 172.21.15.171:0/465060265 learned_addr learned my addr 172.21.15.171:0/465060265 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862055299 0 0) 0x7fa418134310 con 0x7fa418132e60 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 10 monclient(hunting): my global_id is 5803 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa404003520 con 0x7fa418132e60 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3779092929 0 0) 0x7fa404003520 con 0x7fa418132e60 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 >> v1:172.21.15.148:6790/0 conn(0x7fa418132a80 legacy=0x7fa4181334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:55.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 >> v1:172.21.15.148:6789/0 conn(0x7fa41809ec20 legacy=0x7fa418133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 1 monclient: found mon.noname-c 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.981+0000 7fa40ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41809dc60 con 0x7fa418132e60 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: _finish_auth 0 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 5 monclient: authenticate success, global_id 5803 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa410001990 con 0x7fa418132e60 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:55.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: dump: 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:55.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa410001c80 con 0x7fa418132e60 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 1 -- 172.21.15.171:0/465060265 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa410002090 con 0x7fa418132e60 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 4 set_mon_vals no callback set 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa40ffff640 10 monclient: dump: 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:55.996 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 10 monclient: shutdown 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 1 -- 172.21.15.171:0/465060265 >> v1:172.21.15.171:6789/0 conn(0x7fa418132e60 legacy=0x7fa41813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 1 -- 172.21.15.171:0/465060265 shutdown_connections 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 1 -- 172.21.15.171:0/465060265 >> 172.21.15.171:0/465060265 conn(0x7fa4180a52a0 msgr2=0x7fa4180a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:55.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 1 -- 172.21.15.171:0/465060265 shutdown_connections 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.982+0000 7fa41f9c4640 1 -- 172.21.15.171:0/465060265 wait complete. 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 10 monclient: build_initial_monmap 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 10 monclient: monmap: 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:55.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 1 Processor -- start 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 1 -- start start 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 10 monclient: init 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.983+0000 7fa41f9c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa41809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa418132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa418132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa418122cf0 con 0x7fa418132e60 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4180a3d20 con 0x7fa418132a80 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41d760640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa41809ec20 legacy=0x7fa4181c9660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41809c940 con 0x7fa41809ec20 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient(hunting): authenticate will time out at 1031.489106s 2024-09-16T04:10:55.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41cf5f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa418132a80 0x7fa4181ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:45334/0 (socket says 172.21.15.171:45334) 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41cf5f640 1 -- 172.21.15.171:0/1331068817 learned_addr learned my addr 172.21.15.171:0/1331068817 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299466183 0 0) 0x7fa4180a3d20 con 0x7fa418132a80 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): my global_id is 5806 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3fc003520 con 0x7fa418132a80 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3825464551 0 0) 0x7fa418122cf0 con 0x7fa418132e60 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): my global_id is 64245 2024-09-16T04:10:56.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4180a3d20 con 0x7fa418132e60 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1833807340 0 0) 0x7fa3fc003520 con 0x7fa418132a80 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa418122cf0 con 0x7fa418132a80 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa408001930 con 0x7fa418132a80 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): dump: 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:56.001 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1398968850 0 0) 0x7fa418122cf0 con 0x7fa418132a80 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 >> v1:172.21.15.148:6790/0 conn(0x7fa41809ec20 legacy=0x7fa4181c9660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 >> v1:172.21.15.148:6789/0 conn(0x7fa418132e60 legacy=0x7fa4181d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 monclient: found mon.b 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa418122730 con 0x7fa418132a80 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:56.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 5 monclient: authenticate success, global_id 5806 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient: _renew_subs 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa418122a10 con 0x7fa418132a80 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa408003610 con 0x7fa418132a80 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa408003960 con 0x7fa418132a80 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:56.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40dffb640 10 monclient: dump: 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa40f7fe640 4 set_mon_vals no callback set 2024-09-16T04:10:56.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient: _renew_subs 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.984+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa41811fe90 con 0x7fa418132a80 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa408003b00 con 0x7fa418132a80 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa41f9c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(156..156 src has 1..156) ==== 8396+0+0 (unknown 2155597724 0 0) 0x7fa408047fe0 con 0x7fa418132a80 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa41f9c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa41f9c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa418122530 con 0x7fa418132a80 2024-09-16T04:10:56.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.985+0000 7fa41d760640 10 monclient: get_auth_request con 0x7fa3fc039030 auth_method 0 2024-09-16T04:10:56.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.990+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa408015bd0 con 0x7fa418132a80 2024-09-16T04:10:56.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.990+0000 7fa40dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:56.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:55.990+0000 7fa40dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:56.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.157+0000 7fa41f9c4640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T04:10:56.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.157+0000 7fa41f9c4640 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T04:10:56.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.157+0000 7fa41f9c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:56.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.157+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) -- 0x7fa41806bcf0 con 0x7fa418132a80 2024-09-16T04:10:56.888 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.887+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v157) ==== 145+0+0 (unknown 3109629405 0 0) 0x7fa408019880 con 0x7fa418132a80 2024-09-16T04:10:56.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.887+0000 7fa40dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T04:10:56.889 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.887+0000 7fa40dffb640 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-16T04:10:56.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.990+0000 7fa41f9c4640 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T04:10:56.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.990+0000 7fa41f9c4640 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T04:10:56.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.990+0000 7fa41f9c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:56.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:56.990+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) -- 0x7fa418120120 con 0x7fa418132a80 2024-09-16T04:10:57.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.895+0000 7fa40dffb640 1 -- 172.21.15.171:0/1331068817 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v158) ==== 257+0+0 (unknown 2513524880 0 0) 0x7fa408045b30 con 0x7fa418132a80 2024-09-16T04:10:57.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.895+0000 7fa40dffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-16T04:10:57.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.895+0000 7fa40dffb640 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T04:10:57.898 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-16T04:10:57.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 >> v1:172.21.15.171:6812/7327273 conn(0x7fa3fc039030 legacy=0x7fa3fc03b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:57.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 10 monclient: shutdown 2024-09-16T04:10:57.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:57.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 >> v1:172.21.15.171:6789/0 conn(0x7fa418132a80 legacy=0x7fa4181ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:57.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 shutdown_connections 2024-09-16T04:10:57.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 >> 172.21.15.171:0/1331068817 conn(0x7fa4180a52a0 msgr2=0x7fa41809be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:57.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 shutdown_connections 2024-09-16T04:10:57.902 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.898+0000 7fa41f9c4640 1 -- 172.21.15.171:0/1331068817 wait complete. 2024-09-16T04:10:57.912 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:10:57.912 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:10:57.913 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep globster 2024-09-16T04:10:57.913 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '10\.' 2024-09-16T04:10:57.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.993+0000 7f6b048e6640 10 monclient: get_monmap_and_config 2024-09-16T04:10:57.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.993+0000 7f6b048e6640 10 monclient: build_initial_monmap 2024-09-16T04:10:57.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.993+0000 7f6b048e6640 10 monclient: monmap: 2024-09-16T04:10:57.995 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:57.995 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:57.995 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:57.994424+0000 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:57.994424+0000 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:57.996 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 1 Processor -- start 2024-09-16T04:10:58.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 1 -- start start 2024-09-16T04:10:58.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient: init 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6b0009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6b000a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6b0009e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b000a1260 con 0x7f6b0009f640 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0009d0c0 con 0x7f6b0009e1d0 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0009d3a0 con 0x7f6b000a0130 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6b048e6640 10 monclient(hunting): authenticate will time out at 1033.499825s 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6af5d9b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6b000a0130 legacy=0x7f6b00138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6afed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6b0009e1d0 0x7f6b0013bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42930/0 (socket says 172.21.15.171:42930) 2024-09-16T04:10:58.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.994+0000 7f6afed9d640 1 -- 172.21.15.171:0/3021171146 learned_addr learned my addr 172.21.15.171:0/3021171146 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255834998 0 0) 0x7f6b0009d0c0 con 0x7f6b0009e1d0 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): my global_id is 5809 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aec003520 con 0x7f6b0009e1d0 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047213795 0 0) 0x7f6b000a1260 con 0x7f6b0009f640 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient(hunting): my global_id is 64248 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b0009d0c0 con 0x7f6b0009f640 2024-09-16T04:10:58.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2829028888 0 0) 0x7f6aec003520 con 0x7f6b0009e1d0 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 >> v1:172.21.15.148:6790/0 conn(0x7f6b000a0130 legacy=0x7f6b00138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 >> v1:172.21.15.148:6789/0 conn(0x7f6b0009f640 legacy=0x7f6b0009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0009d680 con 0x7f6b0009e1d0 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: _finish_auth 0 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6af8001990 con 0x7f6b0009e1d0 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 5 monclient: authenticate success, global_id 5809 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:58.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: dump: 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6af8001c40 con 0x7f6b0009e1d0 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 1 -- 172.21.15.171:0/3021171146 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6af8002010 con 0x7f6b0009e1d0 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 4 set_mon_vals no callback set 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6afdd9b640 10 monclient: dump: 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.008 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 10 monclient: shutdown 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 1 -- 172.21.15.171:0/3021171146 >> v1:172.21.15.171:6789/0 conn(0x7f6b0009e1d0 legacy=0x7f6b0013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.995+0000 7f6b048e6640 1 -- 172.21.15.171:0/3021171146 shutdown_connections 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.996+0000 7f6b048e6640 1 -- 172.21.15.171:0/3021171146 >> 172.21.15.171:0/3021171146 conn(0x7f6b000a52a0 msgr2=0x7f6b000a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.996+0000 7f6b048e6640 1 -- 172.21.15.171:0/3021171146 shutdown_connections 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.996+0000 7f6b048e6640 1 -- 172.21.15.171:0/3021171146 wait complete. 2024-09-16T04:10:58.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.996+0000 7f6b048e6640 10 monclient: build_initial_monmap 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.996+0000 7f6b048e6640 10 monclient: monmap: 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 1 Processor -- start 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 1 -- start start 2024-09-16T04:10:58.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient: init 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6b0009e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6b0009f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6b000a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b001a4a00 con 0x7f6b0009f640 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b001a4fc0 con 0x7f6b000a0130 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b001a4720 con 0x7f6b0009e1d0 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6b048e6640 10 monclient(hunting): authenticate will time out at 1033.502784s 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6afe59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6b0009e1d0 legacy=0x7f6b001b65f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6afed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6b000a0130 0x7f6b001cf5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42932/0 (socket says 172.21.15.171:42932) 2024-09-16T04:10:58.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.997+0000 7f6afed9d640 1 -- 172.21.15.171:0/1960334332 learned_addr learned my addr 172.21.15.171:0/1960334332 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749503291 0 0) 0x7f6b001a4fc0 con 0x7f6b000a0130 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): my global_id is 5812 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ae0003520 con 0x7f6b000a0130 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 391456034 0 0) 0x7f6b001a4a00 con 0x7f6b0009f640 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): my global_id is 64251 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b001a4fc0 con 0x7f6b0009f640 2024-09-16T04:10:58.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3257635222 0 0) 0x7f6ae0003520 con 0x7f6b000a0130 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b001a4a00 con 0x7f6b000a0130 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6af8001f90 con 0x7f6b000a0130 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): dump: 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.013 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2115359654 0 0) 0x7f6b001a4a00 con 0x7f6b000a0130 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 >> v1:172.21.15.148:6790/0 conn(0x7f6b0009e1d0 legacy=0x7f6b001b65f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 >> v1:172.21.15.148:6789/0 conn(0x7f6b0009f640 legacy=0x7f6b001cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 monclient: found mon.b 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b001a4ce0 con 0x7f6b000a0130 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 5 monclient: authenticate success, global_id 5812 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 10 monclient: _renew_subs 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b001b6d00 con 0x7f6b000a0130 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 10 monclient: _renew_subs 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b001c6d40 con 0x7f6b000a0130 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6af8004fc0 con 0x7f6b000a0130 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:58.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6af80014f0 con 0x7f6b000a0130 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6afd59a640 4 set_mon_vals no callback set 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.998+0000 7f6af77fe640 10 monclient: dump: 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.016 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6af8005170 con 0x7f6b000a0130 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6b048e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6b048e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6b048e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae4005080 con 0x7f6b000a0130 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9085+0+0 (unknown 2148242990 0 0) 0x7f6af8048790 con 0x7f6b000a0130 2024-09-16T04:10:58.017 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:57.999+0000 7f6afe59c640 10 monclient: get_auth_request con 0x7f6ae0038d30 auth_method 0 2024-09-16T04:10:58.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.004+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6af8019d00 con 0x7f6b000a0130 2024-09-16T04:10:58.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.004+0000 7f6af77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.004+0000 7f6af77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:58.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6b048e6640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6b048e6640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6b048e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f6ae4005370 con 0x7f6b000a0130 2024-09-16T04:10:58.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2378 (unknown 807768366 0 113133268) 0x7f6af801e5c0 con 0x7f6b000a0130 2024-09-16T04:10:58.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6af77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.174+0000 7f6af77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:58.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.278+0000 7f6b048e6640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.278+0000 7f6b048e6640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.278+0000 7f6b048e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.278+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f6ae40065c0 con 0x7f6b000a0130 2024-09-16T04:10:58.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.279+0000 7f6af77fe640 1 -- 172.21.15.171:0/1960334332 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2378 (unknown 807768366 0 113133268) 0x7f6af8013df0 con 0x7f6b000a0130 2024-09-16T04:10:58.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.279+0000 7f6af77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.279+0000 7f6af77fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:58.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 >> v1:172.21.15.171:6812/7327273 conn(0x7f6ae0038d30 legacy=0x7f6ae003b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 10 monclient: shutdown 2024-09-16T04:10:58.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:58.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 >> v1:172.21.15.171:6789/0 conn(0x7f6b000a0130 legacy=0x7f6b001cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 shutdown_connections 2024-09-16T04:10:58.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 >> 172.21.15.171:0/1960334332 conn(0x7f6b000a52a0 msgr2=0x7f6b000a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:58.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 shutdown_connections 2024-09-16T04:10:58.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.281+0000 7f6b048e6640 1 -- 172.21.15.171:0/1960334332 wait complete. 2024-09-16T04:10:58.292 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:58.292 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:58.293 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:10:58.293 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:10:58.293 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep globster 2024-09-16T04:10:58.293 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '11\.' 2024-09-16T04:10:58.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.369+0000 7fda91e77640 10 monclient: get_monmap_and_config 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.369+0000 7fda91e77640 10 monclient: build_initial_monmap 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.369+0000 7fda91e77640 10 monclient: monmap: 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:58.369925+0000 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:58.369925+0000 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:58.371 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.372 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:58.372 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:58.372 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:58.372 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.369+0000 7fda91e77640 1 Processor -- start 2024-09-16T04:10:58.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.369+0000 7fda91e77640 1 -- start start 2024-09-16T04:10:58.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.369+0000 7fda91e77640 10 monclient: init 2024-09-16T04:10:58.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient(hunting): picked mon.noname-c con 0x7fda8c136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient(hunting): picked mon.noname-b con 0x7fda8c139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient(hunting): picked mon.noname-a con 0x7fda8c13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c0a1340 con 0x7fda8c13d620 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c137650 con 0x7fda8c136b60 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c0a0aa0 con 0x7fda8c139b10 2024-09-16T04:10:58.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 10 monclient(hunting): authenticate will time out at 1033.875401s 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fda8c136b60 0x7fda8c136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42940/0 (socket says 172.21.15.171:42940) 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8b7fe640 1 -- 172.21.15.171:0/2055455501 learned_addr learned my addr 172.21.15.171:0/2055455501 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8affd640 1 -- 172.21.15.171:0/2055455501 >> v1:172.21.15.148:6790/0 conn(0x7fda8c139b10 legacy=0x7fda8c13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435959011 0 0) 0x7fda8c137650 con 0x7fda8c136b60 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient(hunting): my global_id is 5815 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda78003520 con 0x7fda8c136b60 2024-09-16T04:10:58.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3686654232 0 0) 0x7fda78003520 con 0x7fda8c136b60 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 >> v1:172.21.15.148:6790/0 conn(0x7fda8c139b10 legacy=0x7fda8c13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 >> v1:172.21.15.148:6789/0 conn(0x7fda8c13d620 legacy=0x7fda8c13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c0a0550 con 0x7fda8c136b60 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda7c001990 con 0x7fda8c136b60 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 5 monclient: authenticate success, global_id 5815 2024-09-16T04:10:58.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda8a7fc640 10 monclient: dump: 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.382 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.370+0000 7fda91e77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda7c001cf0 con 0x7fda8c136b60 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda8a7fc640 1 -- 172.21.15.171:0/2055455501 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda7c0020c0 con 0x7fda8c136b60 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 4 set_mon_vals no callback set 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda8a7fc640 10 monclient: dump: 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.383 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 10 monclient: shutdown 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 1 -- 172.21.15.171:0/2055455501 >> v1:172.21.15.171:6789/0 conn(0x7fda8c136b60 legacy=0x7fda8c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 1 -- 172.21.15.171:0/2055455501 shutdown_connections 2024-09-16T04:10:58.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 1 -- 172.21.15.171:0/2055455501 >> 172.21.15.171:0/2055455501 conn(0x7fda8c0a98b0 msgr2=0x7fda8c0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 1 -- 172.21.15.171:0/2055455501 shutdown_connections 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.371+0000 7fda91e77640 1 -- 172.21.15.171:0/2055455501 wait complete. 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.372+0000 7fda91e77640 10 monclient: build_initial_monmap 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.372+0000 7fda91e77640 10 monclient: monmap: 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:58.385 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.372+0000 7fda91e77640 1 Processor -- start 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.372+0000 7fda91e77640 1 -- start start 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.372+0000 7fda91e77640 10 monclient: init 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.372+0000 7fda91e77640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient(hunting): picked mon.noname-b con 0x7fda8c136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient(hunting): picked mon.noname-a con 0x7fda8c139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient(hunting): picked mon.noname-c con 0x7fda8c13d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c0a0240 con 0x7fda8c139b10 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c120c70 con 0x7fda8c136b60 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c09f590 con 0x7fda8c13d620 2024-09-16T04:10:58.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient(hunting): authenticate will time out at 1033.878105s 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda8b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fda8c136b60 0x7fda8c1cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42952/0 (socket says 172.21.15.171:42952) 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda8b7fe640 1 -- 172.21.15.171:0/629458645 learned_addr learned my addr 172.21.15.171:0/629458645 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda8bfff640 1 -- 172.21.15.171:0/629458645 >> v1:172.21.15.148:6790/0 conn(0x7fda8c13d620 legacy=0x7fda8c1d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3597527372 0 0) 0x7fda8c120c70 con 0x7fda8c136b60 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): my global_id is 5818 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda6c003550 con 0x7fda8c136b60 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2341850692 0 0) 0x7fda6c003550 con 0x7fda8c136b60 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda8c120c70 con 0x7fda8c136b60 2024-09-16T04:10:58.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda7c001950 con 0x7fda8c136b60 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): dump: 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.388 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189316367 0 0) 0x7fda740039f0 con 0x7fda8c139b10 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient(hunting): my global_id is 64257 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda6c003550 con 0x7fda8c139b10 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1435879541 0 0) 0x7fda8c120c70 con 0x7fda8c136b60 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 >> v1:172.21.15.148:6790/0 conn(0x7fda8c13d620 legacy=0x7fda8c1d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 >> v1:172.21.15.148:6789/0 conn(0x7fda8c139b10 legacy=0x7fda8c1d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:58.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 monclient: found mon.b 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c0a2b80 con 0x7fda8c136b60 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 5 monclient: authenticate success, global_id 5818 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient: _renew_subs 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c0a70a0 con 0x7fda8c136b60 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient: _renew_subs 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda8c121370 con 0x7fda8c136b60 2024-09-16T04:10:58.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda7c005130 con 0x7fda8c136b60 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fda7c005500 con 0x7fda8c136b60 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda89ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.373+0000 7fda83fff640 10 monclient: dump: 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.391 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fda7c005720 con 0x7fda8c136b60 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda91e77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda91e77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda91e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda70005080 con 0x7fda8c136b60 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9085+0+0 (unknown 2148242990 0 0) 0x7fda7c048840 con 0x7fda8c136b60 2024-09-16T04:10:58.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.374+0000 7fda8affd640 10 monclient: get_auth_request con 0x7fda6c038fa0 auth_method 0 2024-09-16T04:10:58.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.379+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fda7c016310 con 0x7fda8c136b60 2024-09-16T04:10:58.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.379+0000 7fda83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.379+0000 7fda83fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:58.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda91e77640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda91e77640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda91e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fda70005370 con 0x7fda8c136b60 2024-09-16T04:10:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2378 (unknown 807768366 0 113133268) 0x7fda7c015c10 con 0x7fda8c136b60 2024-09-16T04:10:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.551+0000 7fda83fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:58.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.656+0000 7fda91e77640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.656+0000 7fda91e77640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.656+0000 7fda91e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.656+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fda70005ab0 con 0x7fda8c136b60 2024-09-16T04:10:58.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.657+0000 7fda83fff640 1 -- 172.21.15.171:0/629458645 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2378 (unknown 807768366 0 113133268) 0x7fda7c045180 con 0x7fda8c136b60 2024-09-16T04:10:58.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.657+0000 7fda83fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.657+0000 7fda83fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 >> v1:172.21.15.171:6812/7327273 conn(0x7fda6c038fa0 legacy=0x7fda6c03b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 10 monclient: shutdown 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 >> v1:172.21.15.171:6789/0 conn(0x7fda8c136b60 legacy=0x7fda8c1cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 shutdown_connections 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 >> 172.21.15.171:0/629458645 conn(0x7fda8c0a98b0 msgr2=0x7fda8c1391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 shutdown_connections 2024-09-16T04:10:58.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.659+0000 7fda91e77640 1 -- 172.21.15.171:0/629458645 wait complete. 2024-09-16T04:10:58.670 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:58.670 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:58.670 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:10:58.671 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:10:58.671 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep globster 2024-09-16T04:10:58.671 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '12\.' 2024-09-16T04:10:58.749 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.748+0000 7fdd8c391640 10 monclient: get_monmap_and_config 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.748+0000 7fdd8c391640 10 monclient: build_initial_monmap 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.748+0000 7fdd8c391640 10 monclient: monmap: 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:58.748913+0000 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:58.748913+0000 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:58.750 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.748+0000 7fdd8c391640 1 Processor -- start 2024-09-16T04:10:58.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.748+0000 7fdd8c391640 1 -- start start 2024-09-16T04:10:58.758 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.748+0000 7fdd8c391640 10 monclient: init 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient(hunting): picked mon.noname-c con 0x7fdd84136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient(hunting): picked mon.noname-b con 0x7fdd84139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient(hunting): picked mon.noname-a con 0x7fdd8413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd840a1340 con 0x7fdd8413d620 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd84137650 con 0x7fdd84136b60 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd840a0aa0 con 0x7fdd84139b10 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 10 monclient(hunting): authenticate will time out at 1034.254487s 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8992c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fdd84139b10 legacy=0x7fdd8413bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8a12d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdd84136b60 0x7fdd84136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42960/0 (socket says 172.21.15.171:42960) 2024-09-16T04:10:58.759 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8a12d640 1 -- 172.21.15.171:0/4251138445 learned_addr learned my addr 172.21.15.171:0/4251138445 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142431790 0 0) 0x7fdd84137650 con 0x7fdd84136b60 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): my global_id is 5821 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd70003520 con 0x7fdd84136b60 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4079879883 0 0) 0x7fdd840a1340 con 0x7fdd8413d620 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.760 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient(hunting): my global_id is 64260 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd84137650 con 0x7fdd8413d620 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 942321459 0 0) 0x7fdd70003520 con 0x7fdd84136b60 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 >> v1:172.21.15.148:6790/0 conn(0x7fdd84139b10 legacy=0x7fdd8413bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 >> v1:172.21.15.148:6789/0 conn(0x7fdd8413d620 legacy=0x7fdd8413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 monclient: found mon.noname-c 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd840a0550 con 0x7fdd84136b60 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient: _finish_auth 0 2024-09-16T04:10:58.761 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 10 monclient: _check_auth_tickets 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd80001610 con 0x7fdd84136b60 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.749+0000 7fdd8c391640 5 monclient: authenticate success, global_id 5821 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: dump: 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.762 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd80001930 con 0x7fdd84136b60 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 1 -- 172.21.15.171:0/4251138445 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd80001d00 con 0x7fdd84136b60 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 4 set_mon_vals no callback set 2024-09-16T04:10:58.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8912b640 10 monclient: dump: 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 10 monclient: shutdown 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 1 -- 172.21.15.171:0/4251138445 >> v1:172.21.15.171:6789/0 conn(0x7fdd84136b60 legacy=0x7fdd84136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 1 -- 172.21.15.171:0/4251138445 shutdown_connections 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 1 -- 172.21.15.171:0/4251138445 >> 172.21.15.171:0/4251138445 conn(0x7fdd840a98b0 msgr2=0x7fdd840a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 1 -- 172.21.15.171:0/4251138445 shutdown_connections 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.750+0000 7fdd8c391640 1 -- 172.21.15.171:0/4251138445 wait complete. 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.751+0000 7fdd8c391640 10 monclient: build_initial_monmap 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.751+0000 7fdd8c391640 10 monclient: monmap: 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:58.765 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.751+0000 7fdd8c391640 1 Processor -- start 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.751+0000 7fdd8c391640 1 -- start start 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.751+0000 7fdd8c391640 10 monclient: init 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient(hunting): picked mon.noname-a con 0x7fdd84136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient(hunting): picked mon.noname-c con 0x7fdd84139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:58.766 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient(hunting): picked mon.noname-b con 0x7fdd8413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8413bc00 con 0x7fdd84136b60 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8409ad30 con 0x7fdd8413d620 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8409b2f0 con 0x7fdd84139b10 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient(hunting): authenticate will time out at 1034.257214s 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8992c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fdd84139b10 legacy=0x7fdd841d09e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8a92e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdd8413d620 0x7fdd841d4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42968/0 (socket says 172.21.15.171:42968) 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8a92e640 1 -- 172.21.15.171:0/2353293571 learned_addr learned my addr 172.21.15.171:0/2353293571 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405492722 0 0) 0x7fdd8409ad30 con 0x7fdd8413d620 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): my global_id is 5824 2024-09-16T04:10:58.767 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd68003550 con 0x7fdd8413d620 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583031530 0 0) 0x7fdd8413bc00 con 0x7fdd84136b60 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _negotiate 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): my global_id is 64263 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd8409ad30 con 0x7fdd84136b60 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1912993435 0 0) 0x7fdd68003550 con 0x7fdd8413d620 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd8413bc00 con 0x7fdd8413d620 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd7c001c50 con 0x7fdd8413d620 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): dump: 2024-09-16T04:10:58.768 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2208850774 0 0) 0x7fdd8409ad30 con 0x7fdd84136b60 2024-09-16T04:10:58.769 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd68003550 con 0x7fdd84136b60 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd80001430 con 0x7fdd84136b60 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient(hunting): dump: 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.770 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3094412908 0 0) 0x7fdd8413bc00 con 0x7fdd8413d620 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 >> v1:172.21.15.148:6790/0 conn(0x7fdd84139b10 legacy=0x7fdd841d09e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 >> v1:172.21.15.148:6789/0 conn(0x7fdd84136b60 legacy=0x7fdd841cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 monclient: found mon.b 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8413c0b0 con 0x7fdd8413d620 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:58.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 5 monclient: authenticate success, global_id 5824 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient: _renew_subs 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8413c760 con 0x7fdd8413d620 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient: _renew_subs 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd8413afb0 con 0x7fdd8413d620 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd7c002020 con 0x7fdd8413d620 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.752+0000 7fdd7affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd7c003de0 con 0x7fdd8413d620 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd8892a640 4 set_mon_vals no callback set 2024-09-16T04:10:58.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd7affd640 10 monclient: dump: 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdd7c004000 con 0x7fdd8413d620 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd8c391640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.773 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd8c391640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd8c391640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd6c005080 con 0x7fdd8413d620 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.753+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9085+0+0 (unknown 2148242990 0 0) 0x7fdd7c048760 con 0x7fdd8413d620 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.754+0000 7fdd8a12d640 10 monclient: get_auth_request con 0x7fdd68038f40 auth_method 0 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.758+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdd7c016120 con 0x7fdd8413d620 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.758+0000 7fdd7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:58.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.758+0000 7fdd7affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd8c391640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd8c391640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd8c391640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fdd6c005370 con 0x7fdd8413d620 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2378 (unknown 807768366 0 113133268) 0x7fdd7c015a20 con 0x7fdd8413d620 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:58.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:58.930+0000 7fdd7affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:59.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd8c391640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:59.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd8c391640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:59.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd8c391640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fdd6c005ab0 con 0x7fdd8413d620 2024-09-16T04:10:59.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd7affd640 1 -- 172.21.15.171:0/2353293571 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v158) ==== 83+0+2378 (unknown 807768366 0 113133268) 0x7fdd7c01e5b0 con 0x7fdd8413d620 2024-09-16T04:10:59.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd7affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:10:59.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.033+0000 7fdd7affd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:10:59.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 >> v1:172.21.15.171:6812/7327273 conn(0x7fdd68038f40 legacy=0x7fdd6803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 10 monclient: shutdown 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 >> v1:172.21.15.171:6789/0 conn(0x7fdd8413d620 legacy=0x7fdd841d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 shutdown_connections 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 >> 172.21.15.171:0/2353293571 conn(0x7fdd840a98b0 msgr2=0x7fdd84132c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 shutdown_connections 2024-09-16T04:10:59.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.035+0000 7fdd8c391640 1 -- 172.21.15.171:0/2353293571 wait complete. 2024-09-16T04:10:59.046 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:59.047 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-16T04:10:59.047 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-16T04:10:59.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.127+0000 7fb336c65640 10 monclient: get_monmap_and_config 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.127+0000 7fb336c65640 10 monclient: build_initial_monmap 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.127+0000 7fb336c65640 10 monclient: monmap: 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:10:59.128653+0000 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:10:59.128653+0000 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:10:59.130 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.128+0000 7fb336c65640 1 Processor -- start 2024-09-16T04:10:59.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.128+0000 7fb336c65640 1 -- start start 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.128+0000 7fb336c65640 10 monclient: init 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient(hunting): picked mon.noname-a con 0x7fb33009c1b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient(hunting): picked mon.noname-b con 0x7fb3300a0f30 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient(hunting): picked mon.noname-c con 0x7fb33009cca0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb33009d9f0 con 0x7fb33009c1b0 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb33009dcd0 con 0x7fb33009cca0 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb33009dfb0 con 0x7fb3300a0f30 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 10 monclient(hunting): authenticate will time out at 1034.634220s 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32ffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb3300a0f30 legacy=0x7fb330138bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:59.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb335202640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb33009cca0 0x7fb33013c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42972/0 (socket says 172.21.15.171:42972) 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb334a01640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb33009c1b0 0x7fb33009c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.171:49060/0 (socket says 172.21.15.171:49060) 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb335202640 1 -- 172.21.15.171:0/3723807709 learned_addr learned my addr 172.21.15.171:0/3723807709 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859950316 0 0) 0x7fb33009dcd0 con 0x7fb33009cca0 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): my global_id is 5827 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb31c003520 con 0x7fb33009cca0 2024-09-16T04:10:59.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1459975895 0 0) 0x7fb33009d9f0 con 0x7fb33009c1b0 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient(hunting): my global_id is 64266 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb33009dcd0 con 0x7fb33009c1b0 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1481964253 0 0) 0x7fb31c003520 con 0x7fb33009cca0 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 >> v1:172.21.15.148:6790/0 conn(0x7fb3300a0f30 legacy=0x7fb330138bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 >> v1:172.21.15.148:6789/0 conn(0x7fb33009c1b0 legacy=0x7fb33009c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:59.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33009ec90 con 0x7fb33009cca0 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb318001610 con 0x7fb33009cca0 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 5 monclient: authenticate success, global_id 5827 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb336c65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:59.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.129+0000 7fb32f7fe640 10 monclient: dump: 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb318001930 con 0x7fb33009cca0 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb32f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:59.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb32f7fe640 1 -- 172.21.15.171:0/3723807709 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb318001d00 con 0x7fb33009cca0 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 4 set_mon_vals no callback set 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb32f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb32f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb32f7fe640 10 monclient: dump: 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:59.143 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 10 monclient: shutdown 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 1 -- 172.21.15.171:0/3723807709 >> v1:172.21.15.171:6789/0 conn(0x7fb33009cca0 legacy=0x7fb33013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 1 -- 172.21.15.171:0/3723807709 shutdown_connections 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 1 -- 172.21.15.171:0/3723807709 >> 172.21.15.171:0/3723807709 conn(0x7fb3300a53f0 msgr2=0x7fb3300a17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 1 -- 172.21.15.171:0/3723807709 shutdown_connections 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.130+0000 7fb336c65640 1 -- 172.21.15.171:0/3723807709 wait complete. 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.131+0000 7fb336c65640 10 monclient: build_initial_monmap 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.131+0000 7fb336c65640 10 monclient: monmap: 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:10:59.144 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.131+0000 7fb336c65640 1 Processor -- start 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.131+0000 7fb336c65640 1 -- start start 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.131+0000 7fb336c65640 10 monclient: init 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient(hunting): picked mon.noname-a con 0x7fb33009c1b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:10:59.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient(hunting): picked mon.noname-b con 0x7fb33009cca0 addr v1:172.21.15.171:6789/0 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient(hunting): picked mon.noname-c con 0x7fb3300a0f30 addr v1:172.21.15.148:6790/0 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb330120330 con 0x7fb33009c1b0 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb33011e7c0 con 0x7fb33009cca0 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb330120610 con 0x7fb3300a0f30 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient(hunting): authenticate will time out at 1034.637216s 2024-09-16T04:10:59.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb335202640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb3300a0f30 legacy=0x7fb33011d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb33009cca0 0x7fb33011cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42982/0 (socket says 172.21.15.171:42982) 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32ffff640 1 -- 172.21.15.171:0/1530156144 learned_addr learned my addr 172.21.15.171:0/1530156144 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1140688925 0 0) 0x7fb33011e7c0 con 0x7fb33009cca0 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): my global_id is 5830 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb310003520 con 0x7fb33009cca0 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942546525 0 0) 0x7fb330120330 con 0x7fb33009c1b0 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:59.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): my global_id is 64269 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb33011e7c0 con 0x7fb33009c1b0 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2834592168 0 0) 0x7fb310003520 con 0x7fb33009cca0 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb330120330 con 0x7fb33009cca0 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb320004500 con 0x7fb33009cca0 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): dump: 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.148 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4276242439 0 0) 0x7fb33011e7c0 con 0x7fb33009c1b0 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb310003520 con 0x7fb33009c1b0 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3280018b0 con 0x7fb33009c1b0 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:10:59.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient(hunting): dump: 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:59.150 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 761844571 0 0) 0x7fb330120330 con 0x7fb33009cca0 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 >> v1:172.21.15.148:6790/0 conn(0x7fb3300a0f30 legacy=0x7fb33011d5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 >> v1:172.21.15.148:6789/0 conn(0x7fb33009c1b0 legacy=0x7fb33011c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 monclient: found mon.b 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3301339a0 con 0x7fb33009cca0 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:59.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 5 monclient: authenticate success, global_id 5830 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient: _renew_subs 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb330120050 con 0x7fb33009cca0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient: _renew_subs 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3301d0bd0 con 0x7fb33009cca0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb3200047b0 con 0x7fb33009cca0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb320004b00 con 0x7fb33009cca0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:59.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32d7fa640 10 monclient: dump: 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.132+0000 7fb32effd640 4 set_mon_vals no callback set 2024-09-16T04:10:59.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.133+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb320004da0 con 0x7fb33009cca0 2024-09-16T04:10:59.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.133+0000 7fb336c65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:59.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.133+0000 7fb336c65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:59.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.133+0000 7fb336c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.133+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3300008d0 con 0x7fb33009cca0 2024-09-16T04:10:59.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.133+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(158..158 src has 1..158) ==== 9085+0+0 (unknown 2148242990 0 0) 0x7fb320048740 con 0x7fb33009cca0 2024-09-16T04:10:59.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.134+0000 7fb334a01640 10 monclient: get_auth_request con 0x7fb310038ea0 auth_method 0 2024-09-16T04:10:59.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.138+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb320016060 con 0x7fb33009cca0 2024-09-16T04:10:59.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.138+0000 7fb32d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:59.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.138+0000 7fb32d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.313+0000 7fb336c65640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T04:10:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.313+0000 7fb336c65640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T04:10:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.313+0000 7fb336c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:10:59.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.313+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) -- 0x7fb3300a2130 con 0x7fb33009cca0 2024-09-16T04:10:59.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.901+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v159) ==== 118+0+0 (unknown 4059000833 0 0) 0x7fb320015960 con 0x7fb33009cca0 2024-09-16T04:10:59.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.901+0000 7fb32d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T04:10:59.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:10:59.901+0000 7fb32d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:00.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.011+0000 7fb336c65640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T04:11:00.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.011+0000 7fb336c65640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T04:11:00.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.011+0000 7fb336c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:00.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.011+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) -- 0x7fb3301d0da0 con 0x7fb33009cca0 2024-09-16T04:11:00.817 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.815+0000 7fb32d7fa640 1 -- 172.21.15.171:0/1530156144 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v160) ==== 118+0+0 (unknown 1854454076 0 0) 0x7fb32001e570 con 0x7fb33009cca0 2024-09-16T04:11:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.815+0000 7fb32d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-16T04:11:00.818 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.815+0000 7fb32d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.819+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 >> v1:172.21.15.171:6812/7327273 conn(0x7fb310038ea0 legacy=0x7fb31003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.819+0000 7fb336c65640 10 monclient: shutdown 2024-09-16T04:11:00.822 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.819+0000 7fb336c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.819+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 >> v1:172.21.15.171:6789/0 conn(0x7fb33009cca0 legacy=0x7fb33011cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.819+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 shutdown_connections 2024-09-16T04:11:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.819+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 >> 172.21.15.171:0/1530156144 conn(0x7fb3300a53f0 msgr2=0x7fb3301324a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.820+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 shutdown_connections 2024-09-16T04:11:00.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.820+0000 7fb336c65640 1 -- 172.21.15.171:0/1530156144 wait complete. 2024-09-16T04:11:00.835 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-16T04:11:00.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.916+0000 7fd47da77640 10 monclient: get_monmap_and_config 2024-09-16T04:11:00.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.916+0000 7fd47da77640 10 monclient: build_initial_monmap 2024-09-16T04:11:00.918 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.916+0000 7fd47da77640 10 monclient: monmap: 2024-09-16T04:11:00.918 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:00.917282+0000 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:00.917282+0000 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:00.919 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 1 Processor -- start 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 1 -- start start 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient: init 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient(hunting): picked mon.noname-c con 0x7fd478133950 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:00.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient(hunting): picked mon.noname-a con 0x7fd478137920 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient(hunting): picked mon.noname-b con 0x7fd47813b430 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd478076200 con 0x7fd478137920 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4780a0260 con 0x7fd478133950 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd478075880 con 0x7fd47813b430 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd47da77640 10 monclient(hunting): authenticate will time out at 1036.422764s 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd4777fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd478133950 0x7fd478133d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42998/0 (socket says 172.21.15.171:42998) 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd4777fe640 1 -- 172.21.15.171:0/2565652739 learned_addr learned my addr 172.21.15.171:0/2565652739 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.917+0000 7fd477fff640 1 -- 172.21.15.171:0/2565652739 >> v1:172.21.15.148:6790/0 conn(0x7fd47813b430 legacy=0x7fd47813d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114246904 0 0) 0x7fd4780a0260 con 0x7fd478133950 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:00.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): my global_id is 5833 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd460003520 con 0x7fd478133950 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164454936 0 0) 0x7fd478076200 con 0x7fd478137920 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:00.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient(hunting): my global_id is 64272 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4780a0260 con 0x7fd478137920 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 257102477 0 0) 0x7fd460003520 con 0x7fd478133950 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 >> v1:172.21.15.148:6790/0 conn(0x7fd47813b430 legacy=0x7fd47813d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 >> v1:172.21.15.148:6789/0 conn(0x7fd478137920 legacy=0x7fd478139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47809da90 con 0x7fd478133950 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:00.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd468001610 con 0x7fd478133950 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 5 monclient: authenticate success, global_id 5833 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: dump: 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.931 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd4680018c0 con 0x7fd478133950 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 1 -- 172.21.15.171:0/2565652739 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd468001c90 con 0x7fd478133950 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 4 set_mon_vals no callback set 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd4767fc640 10 monclient: dump: 2024-09-16T04:11:00.932 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 10 monclient: shutdown 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 1 -- 172.21.15.171:0/2565652739 >> v1:172.21.15.171:6789/0 conn(0x7fd478133950 legacy=0x7fd478133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:00.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 1 -- 172.21.15.171:0/2565652739 shutdown_connections 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 1 -- 172.21.15.171:0/2565652739 >> 172.21.15.171:0/2565652739 conn(0x7fd4780a6460 msgr2=0x7fd4780a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 1 -- 172.21.15.171:0/2565652739 shutdown_connections 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.918+0000 7fd47da77640 1 -- 172.21.15.171:0/2565652739 wait complete. 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.919+0000 7fd47da77640 10 monclient: build_initial_monmap 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.919+0000 7fd47da77640 10 monclient: monmap: 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:00.934 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.919+0000 7fd47da77640 1 Processor -- start 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.919+0000 7fd47da77640 1 -- start start 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.919+0000 7fd47da77640 10 monclient: init 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient(hunting): picked mon.noname-c con 0x7fd478133950 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient(hunting): picked mon.noname-b con 0x7fd478137920 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient(hunting): picked mon.noname-a con 0x7fd47813b430 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:00.935 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd47811d740 con 0x7fd47813b430 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd478072840 con 0x7fd478137920 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd47811da20 con 0x7fd478133950 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd47da77640 10 monclient(hunting): authenticate will time out at 1036.425428s 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd4777fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fd478133950 legacy=0x7fd4781c9820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd476ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd478137920 0x7fd4781ce7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43002/0 (socket says 172.21.15.171:43002) 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd476ffd640 1 -- 172.21.15.171:0/2536190969 learned_addr learned my addr 172.21.15.171:0/2536190969 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911112361 0 0) 0x7fd478072840 con 0x7fd478137920 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:00.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): my global_id is 5836 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd464003520 con 0x7fd478137920 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522420831 0 0) 0x7fd47811d740 con 0x7fd47813b430 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): my global_id is 64275 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd478072840 con 0x7fd47813b430 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 51810491 0 0) 0x7fd464003520 con 0x7fd478137920 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd47811d740 con 0x7fd478137920 2024-09-16T04:11:00.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd46c001910 con 0x7fd478137920 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): dump: 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:00.938 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.920+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3937492601 0 0) 0x7fd478072840 con 0x7fd47813b430 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd464003520 con 0x7fd47813b430 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd470004380 con 0x7fd47813b430 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient(hunting): dump: 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:00.939 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1418477791 0 0) 0x7fd47811d740 con 0x7fd478137920 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 >> v1:172.21.15.148:6790/0 conn(0x7fd478133950 legacy=0x7fd4781c9820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 >> v1:172.21.15.148:6789/0 conn(0x7fd47813b430 legacy=0x7fd4781d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 monclient: found mon.b 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47813b080 con 0x7fd478137920 2024-09-16T04:11:00.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: _finish_auth 0 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 5 monclient: authenticate success, global_id 5836 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 10 monclient: _renew_subs 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd47813ad30 con 0x7fd478137920 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 10 monclient: _renew_subs 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd478071710 con 0x7fd478137920 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd46c001ce0 con 0x7fd478137920 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:00.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd46c003a60 con 0x7fd478137920 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 10 monclient: dump: 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:00.942 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd475ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.921+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd46c003d00 con 0x7fd478137920 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.922+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(160..160 src has 1..160) ==== 9023+0+0 (unknown 2374161094 0 0) 0x7fd46c0482e0 con 0x7fd478137920 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.922+0000 7fd47da77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.922+0000 7fd47da77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.922+0000 7fd47da77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.922+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4780008d0 con 0x7fd478137920 2024-09-16T04:11:00.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.922+0000 7fd4777fe640 10 monclient: get_auth_request con 0x7fd46403d2b0 auth_method 0 2024-09-16T04:11:00.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.926+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd46c015d60 con 0x7fd478137920 2024-09-16T04:11:00.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.926+0000 7fd45ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:00.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:00.926+0000 7fd45ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.099+0000 7fd47da77640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T04:11:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.099+0000 7fd47da77640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T04:11:01.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.099+0000 7fd47da77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:01.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.099+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) -- 0x7fd4780a3250 con 0x7fd478137920 2024-09-16T04:11:01.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.824+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v161) ==== 127+0+0 (unknown 327323727 0 0) 0x7fd46c015660 con 0x7fd478137920 2024-09-16T04:11:01.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.824+0000 7fd45ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T04:11:01.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.824+0000 7fd45ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:01.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.938+0000 7fd47da77640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T04:11:01.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.938+0000 7fd47da77640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T04:11:01.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.938+0000 7fd47da77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:01.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:01.938+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) -- 0x7fd4780719a0 con 0x7fd478137920 2024-09-16T04:11:02.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.825+0000 7fd45ffff640 1 -- 172.21.15.171:0/2536190969 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v162) ==== 127+0+0 (unknown 2663385207 0 0) 0x7fd46c01e210 con 0x7fd478137920 2024-09-16T04:11:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.826+0000 7fd45ffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-16T04:11:02.828 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.826+0000 7fd45ffff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.828+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 >> v1:172.21.15.171:6812/7327273 conn(0x7fd46403d2b0 legacy=0x7fd46403f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.828+0000 7fd47da77640 10 monclient: shutdown 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.828+0000 7fd47da77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.828+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 >> v1:172.21.15.171:6789/0 conn(0x7fd478137920 legacy=0x7fd4781ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.829+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 shutdown_connections 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.829+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 >> 172.21.15.171:0/2536190969 conn(0x7fd4780a6460 msgr2=0x7fd47813a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.829+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 shutdown_connections 2024-09-16T04:11:02.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.829+0000 7fd47da77640 1 -- 172.21.15.171:0/2536190969 wait complete. 2024-09-16T04:11:02.842 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-16T04:11:02.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.929+0000 7f8969168640 10 monclient: get_monmap_and_config 2024-09-16T04:11:02.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.929+0000 7f8969168640 10 monclient: build_initial_monmap 2024-09-16T04:11:02.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.929+0000 7f8969168640 10 monclient: monmap: 2024-09-16T04:11:02.932 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:02.932 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:02.932 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:02.930278+0000 2024-09-16T04:11:02.932 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:02.930278+0000 2024-09-16T04:11:02.932 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:02.932 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:02.933 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:02.933 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:02.933 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:02.933 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:02.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 1 Processor -- start 2024-09-16T04:11:02.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 1 -- start start 2024-09-16T04:11:02.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient: init 2024-09-16T04:11:02.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:02.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:02.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient(hunting): picked mon.noname-b con 0x7f896409b660 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:02.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient(hunting): picked mon.noname-c con 0x7f89640734b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:02.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient(hunting): picked mon.noname-a con 0x7f8964136d70 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:02.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f896409c150 con 0x7f8964136d70 2024-09-16T04:11:02.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89640998b0 con 0x7f89640734b0 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8964099b90 con 0x7f896409b660 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8969168640 10 monclient(hunting): authenticate will time out at 1038.435766s 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f895bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f89640734b0 0x7f896413bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43004/0 (socket says 172.21.15.171:43004) 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f895bfff640 1 -- 172.21.15.171:0/2431574523 learned_addr learned my addr 172.21.15.171:0/2431574523 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.930+0000 7f8962d9d640 1 -- 172.21.15.171:0/2431574523 >> v1:172.21.15.148:6790/0 conn(0x7f896409b660 legacy=0x7f896409ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234938515 0 0) 0x7f89640998b0 con 0x7f89640734b0 2024-09-16T04:11:02.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient(hunting): _negotiate 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient(hunting): my global_id is 5839 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8950003520 con 0x7f89640734b0 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3617904258 0 0) 0x7f8950003520 con 0x7f89640734b0 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 >> v1:172.21.15.148:6790/0 conn(0x7f896409b660 legacy=0x7f896409ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:02.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 >> v1:172.21.15.148:6789/0 conn(0x7f8964136d70 legacy=0x7f896413f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: _finish_hunting 0 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 monclient: found mon.noname-c 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8964099e70 con 0x7f89640734b0 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: _finish_auth 0 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: _check_auth_tickets 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:02.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f894c001610 con 0x7f89640734b0 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: dump: 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:02.945 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 5 monclient: authenticate success, global_id 5839 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f894c001900 con 0x7f89640734b0 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 1 -- 172.21.15.171:0/2431574523 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f894c001d10 con 0x7f89640734b0 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 4 set_mon_vals no callback set 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f896259c640 10 monclient: dump: 2024-09-16T04:11:02.946 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 10 monclient: shutdown 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:02.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 1 -- 172.21.15.171:0/2431574523 >> v1:172.21.15.171:6789/0 conn(0x7f89640734b0 legacy=0x7f896413bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 1 -- 172.21.15.171:0/2431574523 shutdown_connections 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 1 -- 172.21.15.171:0/2431574523 >> 172.21.15.171:0/2431574523 conn(0x7f89640a1e90 msgr2=0x7f89640a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.931+0000 7f8969168640 1 -- 172.21.15.171:0/2431574523 shutdown_connections 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.932+0000 7f8969168640 1 -- 172.21.15.171:0/2431574523 wait complete. 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.932+0000 7f8969168640 10 monclient: build_initial_monmap 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.932+0000 7f8969168640 10 monclient: monmap: 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:02.948 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.932+0000 7f8969168640 1 Processor -- start 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 1 -- start start 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient: init 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:02.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient(hunting): picked mon.noname-b con 0x7f89640734b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient(hunting): picked mon.noname-a con 0x7f896409b660 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient(hunting): picked mon.noname-c con 0x7f8964136d70 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f896408c6a0 con 0x7f896409b660 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f896408c980 con 0x7f89640734b0 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f896408c3c0 con 0x7f8964136d70 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8969168640 10 monclient(hunting): authenticate will time out at 1038.438480s 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8962d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f89640734b0 0x7f8964095420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43020/0 (socket says 172.21.15.171:43020) 2024-09-16T04:11:02.950 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f896359e640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f8964136d70 legacy=0x7f89641d3a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f8962d9d640 1 -- 172.21.15.171:0/722834437 learned_addr learned my addr 172.21.15.171:0/722834437 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1684335665 0 0) 0x7f896408c980 con 0x7f89640734b0 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): my global_id is 5842 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8944003520 con 0x7f89640734b0 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393973258 0 0) 0x7f896408c6a0 con 0x7f896409b660 2024-09-16T04:11:02.951 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): my global_id is 64281 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f896408c980 con 0x7f896409b660 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3596689804 0 0) 0x7f8944003520 con 0x7f89640734b0 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f896408c6a0 con 0x7f89640734b0 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89540041d0 con 0x7f89640734b0 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:02.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:02.953 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.933+0000 7f895b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2241525385 0 0) 0x7f896408c6a0 con 0x7f89640734b0 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 >> v1:172.21.15.148:6790/0 conn(0x7f8964136d70 legacy=0x7f89641d3a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 >> v1:172.21.15.148:6789/0 conn(0x7f896409b660 legacy=0x7f8964095b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:02.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 monclient: found mon.b 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8964097240 con 0x7f89640734b0 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 5 monclient: authenticate success, global_id 5842 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 10 monclient: _renew_subs 2024-09-16T04:11:02.955 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8964096240 con 0x7f89640734b0 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 10 monclient: _renew_subs 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89641d5180 con 0x7f89640734b0 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8954004480 con 0x7f89640734b0 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:02.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8954004850 con 0x7f89640734b0 2024-09-16T04:11:02.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:02.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:02.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f8961d9b640 4 set_mon_vals no callback set 2024-09-16T04:11:02.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 10 monclient: dump: 2024-09-16T04:11:02.957 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:02.957 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:02.958 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.958 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:02.958 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:02.958 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:02.958 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:02.959 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:02.959 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:02.959 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:02.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.934+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f89540049f0 con 0x7f89640734b0 2024-09-16T04:11:02.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.935+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(162..162 src has 1..162) ==== 9023+0+0 (unknown 3411099224 0 0) 0x7f89540482d0 con 0x7f89640734b0 2024-09-16T04:11:02.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.935+0000 7f895bfff640 10 monclient: get_auth_request con 0x7f8944038fb0 auth_method 0 2024-09-16T04:11:02.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.935+0000 7f8969168640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:02.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.935+0000 7f8969168640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:02.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.935+0000 7f8969168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:02.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.935+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8948005080 con 0x7f89640734b0 2024-09-16T04:11:02.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.939+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8954015c50 con 0x7f89640734b0 2024-09-16T04:11:02.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.939+0000 7f895b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:02.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:02.939+0000 7f895b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:03.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.114+0000 7f8969168640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T04:11:03.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.114+0000 7f8969168640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T04:11:03.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.114+0000 7f8969168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:03.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.114+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) -- 0x7f8948005370 con 0x7f89640734b0 2024-09-16T04:11:03.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.913+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v163) ==== 132+0+0 (unknown 3253193396 0 0) 0x7f8954015550 con 0x7f89640734b0 2024-09-16T04:11:03.915 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.913+0000 7f895b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T04:11:03.916 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:03.913+0000 7f895b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:04.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.023+0000 7f8969168640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T04:11:04.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.023+0000 7f8969168640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T04:11:04.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.024+0000 7f8969168640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:04.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.024+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) -- 0x7f8948005ab0 con 0x7f89640734b0 2024-09-16T04:11:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.915+0000 7f895b7fe640 1 -- 172.21.15.171:0/722834437 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v164) ==== 132+0+0 (unknown 4266501022 0 0) 0x7f89540198a0 con 0x7f89640734b0 2024-09-16T04:11:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.915+0000 7f895b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-16T04:11:04.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.915+0000 7f895b7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 >> v1:172.21.15.171:6812/7327273 conn(0x7f8944038fb0 legacy=0x7f894403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 10 monclient: shutdown 2024-09-16T04:11:04.920 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 >> v1:172.21.15.171:6789/0 conn(0x7f89640734b0 legacy=0x7f8964095420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 shutdown_connections 2024-09-16T04:11:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 >> 172.21.15.171:0/722834437 conn(0x7f89640a1e90 msgr2=0x7f8964098720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 shutdown_connections 2024-09-16T04:11:04.921 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:04.918+0000 7f8969168640 1 -- 172.21.15.171:0/722834437 wait complete. 2024-09-16T04:11:04.933 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:11:04.933 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:11:04.933 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep globster 2024-09-16T04:11:04.933 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '7\.' 2024-09-16T04:11:05.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.013+0000 7f10ee54b640 10 monclient: get_monmap_and_config 2024-09-16T04:11:05.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.013+0000 7f10ee54b640 10 monclient: build_initial_monmap 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.013+0000 7f10ee54b640 10 monclient: monmap: 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:05.014516+0000 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:05.014516+0000 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.016 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:05.017 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:05.017 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:05.017 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.014+0000 7f10ee54b640 1 Processor -- start 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.014+0000 7f10ee54b640 1 -- start start 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.014+0000 7f10ee54b640 10 monclient: init 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.014+0000 7f10ee54b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.014+0000 7f10ee54b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 10 monclient(hunting): picked mon.noname-c con 0x7f10e809b660 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:05.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 10 monclient(hunting): picked mon.noname-a con 0x7f10e80734b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 10 monclient(hunting): picked mon.noname-b con 0x7f10e8136d70 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e809c150 con 0x7f10e80734b0 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e80998b0 con 0x7f10e809b660 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e8099b90 con 0x7f10e8136d70 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 10 monclient(hunting): authenticate will time out at 1040.520121s 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e7fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f10e809b660 0x7f10e809ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43024/0 (socket says 172.21.15.171:43024) 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e7fff640 1 -- 172.21.15.171:0/1951837819 learned_addr learned my addr 172.21.15.171:0/1951837819 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ecae8640 1 -- 172.21.15.171:0/1951837819 >> v1:172.21.15.148:6790/0 conn(0x7f10e8136d70 legacy=0x7f10e813f2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 279641510 0 0) 0x7f10e80998b0 con 0x7f10e809b660 2024-09-16T04:11:05.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): my global_id is 5845 2024-09-16T04:11:05.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10d0003520 con 0x7f10e809b660 2024-09-16T04:11:05.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1446136822 0 0) 0x7f10e809c150 con 0x7f10e80734b0 2024-09-16T04:11:05.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient(hunting): my global_id is 64284 2024-09-16T04:11:05.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e80998b0 con 0x7f10e80734b0 2024-09-16T04:11:05.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3611524238 0 0) 0x7f10d0003520 con 0x7f10e809b660 2024-09-16T04:11:05.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 >> v1:172.21.15.148:6790/0 conn(0x7f10e8136d70 legacy=0x7f10e813f2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:05.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 >> v1:172.21.15.148:6789/0 conn(0x7f10e80734b0 legacy=0x7f10e813bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 monclient: found mon.noname-c 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8099e70 con 0x7f10e809b660 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10d8001610 con 0x7f10e809b660 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 5 monclient: authenticate success, global_id 5845 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: dump: 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.032 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10d80018c0 con 0x7f10e809b660 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 1 -- 172.21.15.171:0/1951837819 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10d8001c90 con 0x7f10e809b660 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10ee54b640 4 set_mon_vals no callback set 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.015+0000 7f10e6ffd640 10 monclient: dump: 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.033 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 10 monclient: shutdown 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 1 -- 172.21.15.171:0/1951837819 >> v1:172.21.15.171:6789/0 conn(0x7f10e809b660 legacy=0x7f10e809ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 1 -- 172.21.15.171:0/1951837819 shutdown_connections 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 1 -- 172.21.15.171:0/1951837819 >> 172.21.15.171:0/1951837819 conn(0x7f10e80a1e90 msgr2=0x7f10e80a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:05.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 1 -- 172.21.15.171:0/1951837819 shutdown_connections 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 1 -- 172.21.15.171:0/1951837819 wait complete. 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 10 monclient: build_initial_monmap 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.016+0000 7f10ee54b640 10 monclient: monmap: 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:05.035 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 1 Processor -- start 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 1 -- start start 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient: init 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient(hunting): picked mon.noname-c con 0x7f10e80734b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient(hunting): picked mon.noname-a con 0x7f10e809b660 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient(hunting): picked mon.noname-b con 0x7f10e8136d70 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e813bf60 con 0x7f10e809b660 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e8139460 con 0x7f10e8136d70 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e813ad00 con 0x7f10e80734b0 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:05.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ee54b640 10 monclient(hunting): authenticate will time out at 1040.522830s 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10e7fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f10e80734b0 legacy=0x7f10e8137bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ecae8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f10e8136d70 0x7f10e81d39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43026/0 (socket says 172.21.15.171:43026) 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.017+0000 7f10ecae8640 1 -- 172.21.15.171:0/3961643403 learned_addr learned my addr 172.21.15.171:0/3961643403 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265342314 0 0) 0x7f10e8139460 con 0x7f10e8136d70 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): my global_id is 5848 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10d4003520 con 0x7f10e8136d70 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519445117 0 0) 0x7f10e80998b0 con 0x7f10e809b660 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): my global_id is 64287 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e8139460 con 0x7f10e809b660 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2518461282 0 0) 0x7f10d4003520 con 0x7f10e8136d70 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10e80998b0 con 0x7f10e8136d70 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10e0004440 con 0x7f10e8136d70 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): dump: 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.038 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3506420509 0 0) 0x7f10e8139460 con 0x7f10e809b660 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10d4003520 con 0x7f10e809b660 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10dc0018b0 con 0x7f10e809b660 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient(hunting): dump: 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.039 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2635867090 0 0) 0x7f10e80998b0 con 0x7f10e8136d70 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 >> v1:172.21.15.148:6790/0 conn(0x7f10e80734b0 legacy=0x7f10e8137bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 >> v1:172.21.15.148:6789/0 conn(0x7f10e809b660 legacy=0x7f10e81382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 monclient: found mon.b 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e813afe0 con 0x7f10e8136d70 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 5 monclient: authenticate success, global_id 5848 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 10 monclient: _renew_subs 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10e813b2c0 con 0x7f10e8136d70 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 10 monclient: _renew_subs 2024-09-16T04:11:05.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f10e813aa10 con 0x7f10e8136d70 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10e00046f0 con 0x7f10e8136d70 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10e0004a40 con 0x7f10e8136d70 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e67fc640 4 set_mon_vals no callback set 2024-09-16T04:11:05.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.018+0000 7f10e4ff9640 10 monclient: dump: 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10ee54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f10e0004be0 con 0x7f10e8136d70 2024-09-16T04:11:05.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10ee54b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10ee54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c8005080 con 0x7f10e8136d70 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(164..164 src has 1..164) ==== 9023+0+0 (unknown 2427898 0 0) 0x7f10e0048550 con 0x7f10e8136d70 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.019+0000 7f10e7fff640 10 monclient: get_auth_request con 0x7f10d4038f40 auth_method 0 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.024+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f10e0019b00 con 0x7f10e8136d70 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.024+0000 7f10e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.024+0000 7f10e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:05.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10ee54b640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10ee54b640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10ee54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f10c8005370 con 0x7f10e8136d70 2024-09-16T04:11:05.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) ==== 83+0+2320 (unknown 2472041953 0 832910183) 0x7f10e0013b30 con 0x7f10e8136d70 2024-09-16T04:11:05.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.197+0000 7f10e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10ee54b640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10ee54b640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10ee54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f10c80065c0 con 0x7f10e8136d70 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10e4ff9640 1 -- 172.21.15.171:0/3961643403 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) ==== 83+0+2320 (unknown 2472041953 0 832910183) 0x7f10e0044df0 con 0x7f10e8136d70 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10e4ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.303+0000 7f10e4ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 >> v1:172.21.15.171:6812/7327273 conn(0x7f10d4038f40 legacy=0x7f10d403b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 10 monclient: shutdown 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 >> v1:172.21.15.171:6789/0 conn(0x7f10e8136d70 legacy=0x7f10e81d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 shutdown_connections 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 >> 172.21.15.171:0/3961643403 conn(0x7f10e80a1e90 msgr2=0x7f10e8098950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 shutdown_connections 2024-09-16T04:11:05.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.305+0000 7f10ee54b640 1 -- 172.21.15.171:0/3961643403 wait complete. 2024-09-16T04:11:05.316 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T04:11:05.316 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T04:11:05.317 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:11:05.317 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:11:05.317 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep globster 2024-09-16T04:11:05.317 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '8\.' 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.394+0000 7f17f702e640 10 monclient: get_monmap_and_config 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.394+0000 7f17f702e640 10 monclient: build_initial_monmap 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.394+0000 7f17f702e640 10 monclient: monmap: 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:05.395022+0000 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:05.395022+0000 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:05.396 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.397 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:05.397 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:05.397 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:05.397 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.394+0000 7f17f702e640 1 Processor -- start 2024-09-16T04:11:05.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.394+0000 7f17f702e640 1 -- start start 2024-09-16T04:11:05.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.394+0000 7f17f702e640 10 monclient: init 2024-09-16T04:11:05.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:05.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:05.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient(hunting): picked mon.noname-b con 0x7f17f009ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient(hunting): picked mon.noname-a con 0x7f17f00a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient(hunting): picked mon.noname-c con 0x7f17f009e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f009fbf0 con 0x7f17f00a0750 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f00a0e10 con 0x7f17f009e570 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f009f640 con 0x7f17f009ba30 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f702e640 10 monclient(hunting): authenticate will time out at 1040.900541s 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f55cb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f17f009e570 0x7f17f013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43030/0 (socket says 172.21.15.171:43030) 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f55cb640 1 -- 172.21.15.171:0/179395563 learned_addr learned my addr 172.21.15.171:0/179395563 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17f4dca640 1 -- 172.21.15.171:0/179395563 >> v1:172.21.15.148:6790/0 conn(0x7f17f009ba30 legacy=0x7f17f00a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:05.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1327419349 0 0) 0x7f17f00a0e10 con 0x7f17f009e570 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17ef7fe640 10 monclient(hunting): my global_id is 5851 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.395+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e0003520 con 0x7f17f009e570 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1436818354 0 0) 0x7f17e0003520 con 0x7f17f009e570 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 >> v1:172.21.15.148:6790/0 conn(0x7f17f009ba30 legacy=0x7f17f00a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 >> v1:172.21.15.148:6789/0 conn(0x7f17f00a0750 legacy=0x7f17f01399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 monclient: found mon.noname-c 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f009f810 con 0x7f17f009e570 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17e8001990 con 0x7f17f009e570 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 5 monclient: authenticate success, global_id 5851 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: dump: 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.407 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17e8001cf0 con 0x7f17f009e570 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 1 -- 172.21.15.171:0/179395563 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17e8004c20 con 0x7f17f009e570 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:05.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17ef7fe640 10 monclient: dump: 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 4 set_mon_vals no callback set 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 10 monclient: shutdown 2024-09-16T04:11:05.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 1 -- 172.21.15.171:0/179395563 >> v1:172.21.15.171:6789/0 conn(0x7f17f009e570 legacy=0x7f17f013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 1 -- 172.21.15.171:0/179395563 shutdown_connections 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 1 -- 172.21.15.171:0/179395563 >> 172.21.15.171:0/179395563 conn(0x7f17f00a52a0 msgr2=0x7f17f00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 1 -- 172.21.15.171:0/179395563 shutdown_connections 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.396+0000 7f17f702e640 1 -- 172.21.15.171:0/179395563 wait complete. 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.397+0000 7f17f702e640 10 monclient: build_initial_monmap 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.397+0000 7f17f702e640 10 monclient: monmap: 2024-09-16T04:11:05.410 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:05.412 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.397+0000 7f17f702e640 1 Processor -- start 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.397+0000 7f17f702e640 1 -- start start 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.397+0000 7f17f702e640 10 monclient: init 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient(hunting): picked mon.noname-b con 0x7f17f009ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient(hunting): picked mon.noname-c con 0x7f17f009e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:05.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient(hunting): picked mon.noname-a con 0x7f17f00a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f0136040 con 0x7f17f00a0750 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f01326a0 con 0x7f17f009ba30 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f006ceb0 con 0x7f17f009e570 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient(hunting): authenticate will time out at 1040.903391s 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f4dca640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f17f009ba30 0x7f17f012df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43038/0 (socket says 172.21.15.171:43038) 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f4dca640 1 -- 172.21.15.171:0/2916508705 learned_addr learned my addr 172.21.15.171:0/2916508705 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17effff640 1 -- 172.21.15.171:0/2916508705 >> v1:172.21.15.148:6790/0 conn(0x7f17f009e570 legacy=0x7f17f012e6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128412415 0 0) 0x7f17f01326a0 con 0x7f17f009ba30 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): my global_id is 5854 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d4003520 con 0x7f17f009ba30 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1084887693 0 0) 0x7f17d4003520 con 0x7f17f009ba30 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17f01326a0 con 0x7f17f009ba30 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17e40041d0 con 0x7f17f009ba30 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): dump: 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.415 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3521869466 0 0) 0x7f17f009f810 con 0x7f17f00a0750 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient(hunting): my global_id is 64293 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17d4003520 con 0x7f17f00a0750 2024-09-16T04:11:05.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3238666783 0 0) 0x7f17f01326a0 con 0x7f17f009ba30 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 >> v1:172.21.15.148:6790/0 conn(0x7f17f009e570 legacy=0x7f17f012e6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 >> v1:172.21.15.148:6789/0 conn(0x7f17f00a0750 legacy=0x7f17f012edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 monclient: found mon.b 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f0135940 con 0x7f17f009ba30 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 5 monclient: authenticate success, global_id 5854 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient: _renew_subs 2024-09-16T04:11:05.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.398+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17f0135c20 con 0x7f17f009ba30 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 10 monclient: _renew_subs 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f17f01d2d10 con 0x7f17f009ba30 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17e40044c0 con 0x7f17f009ba30 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f17e4004890 con 0x7f17f009ba30 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17eeffd640 4 set_mon_vals no callback set 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 10 monclient: dump: 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.418 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f17e4004ab0 con 0x7f17f009ba30 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.399+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d0005080 con 0x7f17f009ba30 2024-09-16T04:11:05.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.400+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(164..164 src has 1..164) ==== 9023+0+0 (unknown 2427898 0 0) 0x7f17e40484b0 con 0x7f17f009ba30 2024-09-16T04:11:05.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.400+0000 7f17effff640 10 monclient: get_auth_request con 0x7f17d4038f70 auth_method 0 2024-09-16T04:11:05.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.404+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f17e4015db0 con 0x7f17f009ba30 2024-09-16T04:11:05.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.404+0000 7f17ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.404+0000 7f17ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:05.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.574+0000 7f17f702e640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.574+0000 7f17f702e640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.574+0000 7f17f702e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.574+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f17d0005370 con 0x7f17f009ba30 2024-09-16T04:11:05.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.575+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) ==== 83+0+2320 (unknown 2472041953 0 832910183) 0x7f17e4019a60 con 0x7f17f009ba30 2024-09-16T04:11:05.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.575+0000 7f17ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.575+0000 7f17ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:05.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.679+0000 7f17f702e640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.679+0000 7f17f702e640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.679+0000 7f17f702e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.679+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7f17d00065c0 con 0x7f17f009ba30 2024-09-16T04:11:05.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.680+0000 7f17ed7fa640 1 -- 172.21.15.171:0/2916508705 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) ==== 83+0+2320 (unknown 2472041953 0 832910183) 0x7f17e401e320 con 0x7f17f009ba30 2024-09-16T04:11:05.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.680+0000 7f17ed7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.682 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.680+0000 7f17ed7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 >> v1:172.21.15.171:6812/7327273 conn(0x7f17d4038f70 legacy=0x7f17d403b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 10 monclient: shutdown 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 >> v1:172.21.15.171:6789/0 conn(0x7f17f009ba30 legacy=0x7f17f012df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 shutdown_connections 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 >> 172.21.15.171:0/2916508705 conn(0x7f17f00a52a0 msgr2=0x7f17f0131bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 shutdown_connections 2024-09-16T04:11:05.685 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.682+0000 7f17f702e640 1 -- 172.21.15.171:0/2916508705 wait complete. 2024-09-16T04:11:05.693 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T04:11:05.693 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T04:11:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush tree --show-shadow 2024-09-16T04:11:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep 'osd\.0' 2024-09-16T04:11:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep globster 2024-09-16T04:11:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep '6\.' 2024-09-16T04:11:05.771 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.770+0000 7fe1820a6640 10 monclient: get_monmap_and_config 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.770+0000 7fe1820a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.770+0000 7fe1820a6640 10 monclient: monmap: 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:05.770844+0000 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:05.770844+0000 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:05.772 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:05.773 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.780 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.770+0000 7fe1820a6640 1 Processor -- start 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 1 -- start start 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient: init 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient(hunting): picked mon.noname-a con 0x7fe17c132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c1332b0 con 0x7fe17c132a80 2024-09-16T04:11:05.781 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c134310 con 0x7fe17c09ec20 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09d980 con 0x7fe17c132e60 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe1820a6640 10 monclient(hunting): authenticate will time out at 1041.276548s 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe173fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe17c09ec20 0x7fe17c133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43042/0 (socket says 172.21.15.171:43042) 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe17c132e60 legacy=0x7fe17c13c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe173fff640 1 -- 172.21.15.171:0/3447809606 learned_addr learned my addr 172.21.15.171:0/3447809606 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2571490052 0 0) 0x7fe17c134310 con 0x7fe17c09ec20 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17affd640 10 monclient(hunting): my global_id is 5857 2024-09-16T04:11:05.782 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.771+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe168003520 con 0x7fe17c09ec20 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4012830427 0 0) 0x7fe168003520 con 0x7fe17c09ec20 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 >> v1:172.21.15.148:6790/0 conn(0x7fe17c132e60 legacy=0x7fe17c13c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 >> v1:172.21.15.148:6789/0 conn(0x7fe17c132a80 legacy=0x7fe17c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 monclient: found mon.noname-c 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c09dc60 con 0x7fe17c09ec20 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe164001610 con 0x7fe17c09ec20 2024-09-16T04:11:05.783 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 5 monclient: authenticate success, global_id 5857 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: dump: 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.784 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe164001900 con 0x7fe17c09ec20 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 1 -- 172.21.15.171:0/3447809606 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe164001d10 con 0x7fe17c09ec20 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 4 set_mon_vals no callback set 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:05.785 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe17affd640 10 monclient: dump: 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.786 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 10 monclient: shutdown 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 1 -- 172.21.15.171:0/3447809606 >> v1:172.21.15.171:6789/0 conn(0x7fe17c09ec20 legacy=0x7fe17c133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 1 -- 172.21.15.171:0/3447809606 shutdown_connections 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 1 -- 172.21.15.171:0/3447809606 >> 172.21.15.171:0/3447809606 conn(0x7fe17c0a52a0 msgr2=0x7fe17c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 1 -- 172.21.15.171:0/3447809606 shutdown_connections 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.772+0000 7fe1820a6640 1 -- 172.21.15.171:0/3447809606 wait complete. 2024-09-16T04:11:05.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.773+0000 7fe1820a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.773+0000 7fe1820a6640 10 monclient: monmap: 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.773+0000 7fe1820a6640 1 Processor -- start 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.773+0000 7fe1820a6640 1 -- start start 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.773+0000 7fe1820a6640 10 monclient: init 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c09ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient(hunting): picked mon.noname-a con 0x7fe17c132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c1c99c0 con 0x7fe17c132e60 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c06ceb0 con 0x7fe17c132a80 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09c800 con 0x7fe17c09ec20 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient(hunting): authenticate will time out at 1041.279161s 2024-09-16T04:11:05.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe17b7fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe17c09ec20 legacy=0x7fe17c131cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe173fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe17c132a80 0x7fe17c1ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43044/0 (socket says 172.21.15.171:43044) 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe173fff640 1 -- 172.21.15.171:0/255712499 learned_addr learned my addr 172.21.15.171:0/255712499 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333256754 0 0) 0x7fe17c06ceb0 con 0x7fe17c132a80 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): my global_id is 5860 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe15c003520 con 0x7fe17c132a80 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241754324 0 0) 0x7fe174003d70 con 0x7fe17c132e60 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:05.790 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): my global_id is 64299 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe17c06ceb0 con 0x7fe17c132e60 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2198762306 0 0) 0x7fe15c003520 con 0x7fe17c132a80 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe174003d70 con 0x7fe17c132a80 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe1640015d0 con 0x7fe17c132a80 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): dump: 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.791 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1914219515 0 0) 0x7fe174003d70 con 0x7fe17c132a80 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 >> v1:172.21.15.148:6790/0 conn(0x7fe17c09ec20 legacy=0x7fe17c131cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 >> v1:172.21.15.148:6789/0 conn(0x7fe17c132e60 legacy=0x7fe17c1d1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:11:05.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 monclient: found mon.b 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c11f3b0 con 0x7fe17c132a80 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: _finish_auth 0 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 5 monclient: authenticate success, global_id 5860 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient: _renew_subs 2024-09-16T04:11:05.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c11f690 con 0x7fe17c132a80 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient: _renew_subs 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe17c1c8f60 con 0x7fe17c132a80 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe164001900 con 0x7fe17c132a80 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe164004cc0 con 0x7fe17c132a80 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:05.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe17a7fc640 4 set_mon_vals no callback set 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.774+0000 7fe178ff9640 10 monclient: dump: 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:05.795 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.775+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe164004ee0 con 0x7fe17c132a80 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.775+0000 7fe1820a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.775+0000 7fe1820a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.775+0000 7fe1820a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.775+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe160005080 con 0x7fe17c132a80 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.775+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(164..164 src has 1..164) ==== 9023+0+0 (unknown 2427898 0 0) 0x7fe164048470 con 0x7fe17c132a80 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.776+0000 7fe17b7fe640 10 monclient: get_auth_request con 0x7fe15c038f20 auth_method 0 2024-09-16T04:11:05.796 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.780+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe16401e300 con 0x7fe17c132a80 2024-09-16T04:11:05.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.780+0000 7fe178ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:05.797 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.780+0000 7fe178ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:05.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.956+0000 7fe1820a6640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.956+0000 7fe1820a6640 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.959 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.956+0000 7fe1820a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:05.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.956+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fe160005370 con 0x7fe17c132a80 2024-09-16T04:11:05.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.957+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) ==== 83+0+2320 (unknown 3080134077 0 832910183) 0x7fe164013b40 con 0x7fe17c132a80 2024-09-16T04:11:05.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.957+0000 7fe178ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:05.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:05.957+0000 7fe178ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:06.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.062+0000 7fe1820a6640 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:06.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.062+0000 7fe1820a6640 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:06.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.062+0000 7fe1820a6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.062+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) -- 0x7fe1600065c0 con 0x7fe17c132a80 2024-09-16T04:11:06.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.063+0000 7fe178ff9640 1 -- 172.21.15.171:0/255712499 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v165) ==== 83+0+2320 (unknown 3080134077 0 832910183) 0x7fe164044d30 con 0x7fe17c132a80 2024-09-16T04:11:06.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.063+0000 7fe178ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-16T04:11:06.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.063+0000 7fe178ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 >> v1:172.21.15.171:6812/7327273 conn(0x7fe15c038f20 legacy=0x7fe15c03b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 10 monclient: shutdown 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 >> v1:172.21.15.171:6789/0 conn(0x7fe17c132a80 legacy=0x7fe17c1ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 shutdown_connections 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 >> 172.21.15.171:0/255712499 conn(0x7fe17c0a52a0 msgr2=0x7fe17c09f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 shutdown_connections 2024-09-16T04:11:06.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.065+0000 7fe1820a6640 1 -- 172.21.15.171:0/255712499 wait complete. 2024-09-16T04:11:06.076 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T04:11:06.076 INFO:tasks.workunit.client.0.smithi171.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-16T04:11:06.076 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.152+0000 7f2a81cc9640 10 monclient: get_monmap_and_config 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.152+0000 7f2a81cc9640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.152+0000 7f2a81cc9640 10 monclient: monmap: 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:06.153349+0000 2024-09-16T04:11:06.154 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:06.153349+0000 2024-09-16T04:11:06.155 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.155 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:06.155 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:06.155 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:06.155 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:06.155 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:06.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 1 Processor -- start 2024-09-16T04:11:06.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 1 -- start start 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient: init 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7c132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f2a7c09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f2a7c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c1332b0 con 0x7f2a7c09ec20 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c134310 con 0x7f2a7c132a80 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c09d980 con 0x7f2a7c132e60 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a81cc9640 10 monclient(hunting): authenticate will time out at 1041.658791s 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a7b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2a7c132a80 0x7f2a7c1334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43054/0 (socket says 172.21.15.171:43054) 2024-09-16T04:11:06.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a7b7fe640 1 -- 172.21.15.171:0/2095687841 learned_addr learned my addr 172.21.15.171:0/2095687841 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.153+0000 7f2a7bfff640 1 -- 172.21.15.171:0/2095687841 >> v1:172.21.15.148:6790/0 conn(0x7f2a7c132e60 legacy=0x7f2a7c13c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359842799 0 0) 0x7f2a7c134310 con 0x7f2a7c132a80 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient(hunting): my global_id is 5863 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a64003520 con 0x7f2a7c132a80 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1867771376 0 0) 0x7f2a64003520 con 0x7f2a7c132a80 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 >> v1:172.21.15.148:6790/0 conn(0x7f2a7c132e60 legacy=0x7f2a7c13c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 >> v1:172.21.15.148:6789/0 conn(0x7f2a7c09ec20 legacy=0x7f2a7c133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:06.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c09dc60 con 0x7f2a7c132a80 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2a6c001610 con 0x7f2a7c132a80 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: dump: 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.165 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 5 monclient: authenticate success, global_id 5863 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a6c001870 con 0x7f2a7c132a80 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 1 -- 172.21.15.171:0/2095687841 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2a6c001c40 con 0x7f2a7c132a80 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:06.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 4 set_mon_vals no callback set 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a7a7fc640 10 monclient: dump: 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:06.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 10 monclient: shutdown 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.154+0000 7f2a81cc9640 1 -- 172.21.15.171:0/2095687841 >> v1:172.21.15.171:6789/0 conn(0x7f2a7c132a80 legacy=0x7f2a7c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 1 -- 172.21.15.171:0/2095687841 shutdown_connections 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 1 -- 172.21.15.171:0/2095687841 >> 172.21.15.171:0/2095687841 conn(0x7f2a7c0a52a0 msgr2=0x7f2a7c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 1 -- 172.21.15.171:0/2095687841 shutdown_connections 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 1 -- 172.21.15.171:0/2095687841 wait complete. 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 10 monclient: monmap: 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:06.168 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.155+0000 7f2a81cc9640 1 Processor -- start 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 1 -- start start 2024-09-16T04:11:06.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient: init 2024-09-16T04:11:06.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:06.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient(hunting): picked mon.noname-a con 0x7f2a7c09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:06.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient(hunting): picked mon.noname-b con 0x7f2a7c132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:06.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:06.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c06ceb0 con 0x7f2a7c09ec20 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c09c7c0 con 0x7f2a7c132a80 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c06fd10 con 0x7f2a7c132e60 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a81cc9640 10 monclient(hunting): authenticate will time out at 1041.661577s 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a7bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f2a7c132e60 legacy=0x7f2a7c1d3f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a7affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f2a7c132a80 0x7f2a7c1d0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43056/0 (socket says 172.21.15.171:43056) 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a7affd640 1 -- 172.21.15.171:0/112104228 learned_addr learned my addr 172.21.15.171:0/112104228 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786759252 0 0) 0x7f2a7c09c7c0 con 0x7f2a7c132a80 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): my global_id is 5866 2024-09-16T04:11:06.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a68003520 con 0x7f2a7c132a80 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 63644859 0 0) 0x7f2a7c06ceb0 con 0x7f2a7c09ec20 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 10 monclient(hunting): my global_id is 64305 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.156+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a7c09c7c0 con 0x7f2a7c09ec20 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2268952025 0 0) 0x7f2a68003520 con 0x7f2a7c132a80 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a7c06ceb0 con 0x7f2a7c132a80 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2a700019e0 con 0x7f2a7c132a80 2024-09-16T04:11:06.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient(hunting): dump: 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:06.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3295084731 0 0) 0x7f2a7c06ceb0 con 0x7f2a7c132a80 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 >> v1:172.21.15.148:6790/0 conn(0x7f2a7c132e60 legacy=0x7f2a7c1d3f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 >> v1:172.21.15.148:6789/0 conn(0x7f2a7c09ec20 legacy=0x7f2a7c1cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 monclient: found mon.b 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c125860 con 0x7f2a7c132a80 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 5 monclient: authenticate success, global_id 5866 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 10 monclient: _renew_subs 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c125b40 con 0x7f2a7c132a80 2024-09-16T04:11:06.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 10 monclient: _renew_subs 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a7c131d60 con 0x7f2a7c132a80 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a700038c0 con 0x7f2a7c132a80 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2a70003c90 con 0x7f2a7c132a80 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a79ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.157+0000 7f2a63fff640 10 monclient: dump: 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:06.175 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2a70003e30 con 0x7f2a7c132a80 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a81cc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a81cc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a81cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5c005080 con 0x7f2a7c132a80 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(165..165 src has 1..165) ==== 9023+0+0 (unknown 592248245 0 0) 0x7f2a70048420 con 0x7f2a7c132a80 2024-09-16T04:11:06.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.158+0000 7f2a7b7fe640 10 monclient: get_auth_request con 0x7f2a68038bc0 auth_method 0 2024-09-16T04:11:06.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.162+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2a70015d40 con 0x7f2a7c132a80 2024-09-16T04:11:06.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.162+0000 7f2a63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.162+0000 7f2a63fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:06.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.335+0000 7f2a81cc9640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:11:06.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.335+0000 7f2a81cc9640 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:11:06.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.335+0000 7f2a81cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:06.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.335+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f2a5c005b80 con 0x7f2a7c132a80 2024-09-16T04:11:06.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.926+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v166) ==== 123+0+0 (unknown 2875036770 0 0) 0x7f2a700199f0 con 0x7f2a7c132a80 2024-09-16T04:11:06.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.926+0000 7f2a63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:11:06.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:06.926+0000 7f2a63fff640 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-16T04:11:07.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.031+0000 7f2a81cc9640 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:11:07.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.031+0000 7f2a81cc9640 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:11:07.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.031+0000 7f2a81cc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:07.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.031+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) -- 0x7f2a5c001090 con 0x7f2a7c132a80 2024-09-16T04:11:07.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.940+0000 7f2a63fff640 1 -- 172.21.15.171:0/112104228 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v167) ==== 149+0+0 (unknown 29425364 0 0) 0x7f2a7001e2b0 con 0x7f2a7c132a80 2024-09-16T04:11:07.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.940+0000 7f2a63fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-16T04:11:07.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.940+0000 7f2a63fff640 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T04:11:07.943 INFO:tasks.workunit.client.0.smithi171.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.943+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 >> v1:172.21.15.171:6812/7327273 conn(0x7f2a68038bc0 legacy=0x7f2a6803b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.943+0000 7f2a81cc9640 10 monclient: shutdown 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.943+0000 7f2a81cc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.943+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 >> v1:172.21.15.171:6789/0 conn(0x7f2a7c132a80 legacy=0x7f2a7c1d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.944+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 shutdown_connections 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.944+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 >> 172.21.15.171:0/112104228 conn(0x7f2a7c0a52a0 msgr2=0x7f2a7c0a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.944+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 shutdown_connections 2024-09-16T04:11:07.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:07.944+0000 7f2a81cc9640 1 -- 172.21.15.171:0/112104228 wait complete. 2024-09-16T04:11:07.956 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-16T04:11:08.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.041+0000 7f7e21653640 10 monclient: get_monmap_and_config 2024-09-16T04:11:08.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.041+0000 7f7e21653640 10 monclient: build_initial_monmap 2024-09-16T04:11:08.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.041+0000 7f7e21653640 10 monclient: monmap: 2024-09-16T04:11:08.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:08.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:08.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:08.042062+0000 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:08.042062+0000 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:08.044 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:08.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 1 Processor -- start 2024-09-16T04:11:08.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 1 -- start start 2024-09-16T04:11:08.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient: init 2024-09-16T04:11:08.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:08.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:08.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1c136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:08.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1c137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:08.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1c1375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c09b2c0 con 0x7f7e1c1375a0 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c09b5a0 con 0x7f7e1c136d60 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c0a6640 con 0x7f7e1c137140 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e21653640 10 monclient(hunting): authenticate will time out at 1043.547761s 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e1affd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7e1c136d60 0x7f7e1c09bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42598/0 (socket says 172.21.15.171:42598) 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e1affd640 1 -- 172.21.15.171:0/1549277480 learned_addr learned my addr 172.21.15.171:0/1549277480 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.042+0000 7f7e13fff640 1 -- 172.21.15.171:0/1549277480 >> v1:172.21.15.148:6790/0 conn(0x7f7e1c137140 legacy=0x7f7e1c13bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687922171 0 0) 0x7f7e1c09b5a0 con 0x7f7e1c136d60 2024-09-16T04:11:08.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient(hunting): my global_id is 5869 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e08003520 con 0x7f7e1c136d60 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 593168370 0 0) 0x7f7e08003520 con 0x7f7e1c136d60 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 >> v1:172.21.15.148:6790/0 conn(0x7f7e1c137140 legacy=0x7f7e1c13bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 >> v1:172.21.15.148:6789/0 conn(0x7f7e1c1375a0 legacy=0x7f7e1c13f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:08.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c143b20 con 0x7f7e1c136d60 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e0c001610 con 0x7f7e1c136d60 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: dump: 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:08.058 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:08.059 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e21653640 5 monclient: authenticate success, global_id 5869 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e21653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e0c001900 con 0x7f7e1c136d60 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 1 -- 172.21.15.171:0/1549277480 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e0c001cd0 con 0x7f7e1c136d60 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e21653640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:08.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e21653640 4 set_mon_vals no callback set 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.043+0000 7f7e1a7fc640 10 monclient: dump: 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:08.061 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 10 monclient: shutdown 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 1 -- 172.21.15.171:0/1549277480 >> v1:172.21.15.171:6789/0 conn(0x7f7e1c136d60 legacy=0x7f7e1c09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 1 -- 172.21.15.171:0/1549277480 shutdown_connections 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 1 -- 172.21.15.171:0/1549277480 >> 172.21.15.171:0/1549277480 conn(0x7f7e1c0a9ae0 msgr2=0x7f7e1c0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 1 -- 172.21.15.171:0/1549277480 shutdown_connections 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 1 -- 172.21.15.171:0/1549277480 wait complete. 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 10 monclient: build_initial_monmap 2024-09-16T04:11:08.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.044+0000 7f7e21653640 10 monclient: monmap: 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 1 Processor -- start 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 1 -- start start 2024-09-16T04:11:08.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient: init 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient(hunting): picked mon.noname-a con 0x7f7e1c136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1c137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1c1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c12e750 con 0x7f7e1c136d60 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c0a8940 con 0x7f7e1c137140 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1c0a83f0 con 0x7f7e1c1375a0 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e21653640 10 monclient(hunting): authenticate will time out at 1043.550631s 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e13fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7e1c137140 0x7f7e1c09d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42612/0 (socket says 172.21.15.171:42612) 2024-09-16T04:11:08.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e13fff640 1 -- 172.21.15.171:0/4008137862 learned_addr learned my addr 172.21.15.171:0/4008137862 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e1b7fe640 1 -- 172.21.15.171:0/4008137862 >> v1:172.21.15.148:6790/0 conn(0x7f7e1c1375a0 legacy=0x7f7e1c09dee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168376844 0 0) 0x7f7e1c0a8940 con 0x7f7e1c137140 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e137fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e137fe640 10 monclient(hunting): my global_id is 5872 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.045+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dfc003520 con 0x7f7e1c137140 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353414573 0 0) 0x7f7e1c12e750 con 0x7f7e1c136d60 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): my global_id is 64311 2024-09-16T04:11:08.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e1c0a8940 con 0x7f7e1c136d60 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2873186015 0 0) 0x7f7dfc003520 con 0x7f7e1c137140 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e1c12e750 con 0x7f7e1c137140 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e04004100 con 0x7f7e1c137140 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): dump: 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:08.066 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3682662420 0 0) 0x7f7e1c12e750 con 0x7f7e1c137140 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 >> v1:172.21.15.148:6790/0 conn(0x7f7e1c1375a0 legacy=0x7f7e1c09dee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 >> v1:172.21.15.148:6789/0 conn(0x7f7e1c136d60 legacy=0x7f7e1c09d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 monclient: found mon.b 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c0a03b0 con 0x7f7e1c137140 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:08.067 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 5 monclient: authenticate success, global_id 5872 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 10 monclient: _renew_subs 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c12e470 con 0x7f7e1c137140 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 10 monclient: _renew_subs 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e1c1d4a90 con 0x7f7e1c137140 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e04004360 con 0x7f7e1c137140 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e040046b0 con 0x7f7e1c137140 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:08.068 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e137fe640 10 monclient: dump: 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.046+0000 7f7e19ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:08.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7e04004850 con 0x7f7e1c137140 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e21653640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e21653640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e21653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(167..167 src has 1..167) ==== 8334+0+0 (unknown 3448083006 0 0) 0x7f7e04047e80 con 0x7f7e1c137140 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1c0008d0 con 0x7f7e1c137140 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.047+0000 7f7e1affd640 10 monclient: get_auth_request con 0x7f7dfc039000 auth_method 0 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.051+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7e04015ad0 con 0x7f7e1c137140 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.051+0000 7f7e137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:08.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.051+0000 7f7e137fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:08.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.217+0000 7f7e21653640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:08.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.217+0000 7f7e21653640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:08.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.217+0000 7f7e21653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:08.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.217+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e1c1d4c60 con 0x7f7e1c137140 2024-09-16T04:11:08.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.940+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v168) ==== 148+0+0 (unknown 3885969402 0 0) 0x7f7e04019780 con 0x7f7e1c137140 2024-09-16T04:11:08.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.940+0000 7f7e137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:08.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:08.940+0000 7f7e137fe640 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-16T04:11:09.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.044+0000 7f7e21653640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:09.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.045+0000 7f7e21653640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:09.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.045+0000 7f7e21653640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.045+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e1c1d4fc0 con 0x7f7e1c137140 2024-09-16T04:11:09.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.046+0000 7f7e137fe640 1 -- 172.21.15.171:0/4008137862 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v168) ==== 155+0+0 (unknown 1175874072 0 0) 0x7f7e0401e040 con 0x7f7e1c137140 2024-09-16T04:11:09.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.046+0000 7f7e137fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:09.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.046+0000 7f7e137fe640 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-16T04:11:09.052 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'cool' does not exist 2024-09-16T04:11:09.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 >> v1:172.21.15.171:6812/7327273 conn(0x7f7dfc039000 legacy=0x7f7dfc03b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:09.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 10 monclient: shutdown 2024-09-16T04:11:09.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:09.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 >> v1:172.21.15.171:6789/0 conn(0x7f7e1c137140 legacy=0x7f7e1c09d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:09.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 shutdown_connections 2024-09-16T04:11:09.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 >> 172.21.15.171:0/4008137862 conn(0x7f7e1c0a9ae0 msgr2=0x7f7e1c072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:09.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 shutdown_connections 2024-09-16T04:11:09.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.048+0000 7f7e21653640 1 -- 172.21.15.171:0/4008137862 wait complete. 2024-09-16T04:11:09.059 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-16T04:11:09.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.139+0000 7f915714b640 10 monclient: get_monmap_and_config 2024-09-16T04:11:09.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.139+0000 7f915714b640 10 monclient: build_initial_monmap 2024-09-16T04:11:09.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.139+0000 7f915714b640 10 monclient: monmap: 2024-09-16T04:11:09.141 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:09.141 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:09.140320+0000 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:09.140320+0000 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:09.142 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 1 Processor -- start 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 1 -- start start 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient: init 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient(hunting): picked mon.noname-a con 0x7f9150133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient(hunting): picked mon.noname-c con 0x7f91501341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient(hunting): picked mon.noname-b con 0x7f915013a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915009da90 con 0x7f9150133700 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915009dd70 con 0x7f91501341f0 2024-09-16T04:11:09.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915009e050 con 0x7f915013a350 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f915714b640 10 monclient(hunting): authenticate will time out at 1044.645865s 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f914ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f91501341f0 0x7f9150138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42624/0 (socket says 172.21.15.171:42624) 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f914ffff640 1 -- 172.21.15.171:0/3631419217 learned_addr learned my addr 172.21.15.171:0/3631419217 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.140+0000 7f91556e8640 1 -- 172.21.15.171:0/3631419217 >> v1:172.21.15.148:6790/0 conn(0x7f915013a350 legacy=0x7f915013c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893541625 0 0) 0x7f915009dd70 con 0x7f91501341f0 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): my global_id is 5875 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9140003520 con 0x7f91501341f0 2024-09-16T04:11:09.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3338452497 0 0) 0x7f915009da90 con 0x7f9150133700 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient(hunting): my global_id is 64314 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f915009dd70 con 0x7f9150133700 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2397351433 0 0) 0x7f9140003520 con 0x7f91501341f0 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 >> v1:172.21.15.148:6790/0 conn(0x7f915013a350 legacy=0x7f915013c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 >> v1:172.21.15.148:6789/0 conn(0x7f9150133700 legacy=0x7f9150133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915009d4f0 con 0x7f91501341f0 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f913c001610 con 0x7f91501341f0 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f915714b640 5 monclient: authenticate success, global_id 5875 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f915714b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: dump: 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.153 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f915714b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f913c001870 con 0x7f91501341f0 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f915714b640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 1 -- 172.21.15.171:0/3631419217 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f913c001c40 con 0x7f91501341f0 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f915714b640 4 set_mon_vals no callback set 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:09.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.141+0000 7f914f7fe640 10 monclient: dump: 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 10 monclient: shutdown 2024-09-16T04:11:09.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 1 -- 172.21.15.171:0/3631419217 >> v1:172.21.15.171:6789/0 conn(0x7f91501341f0 legacy=0x7f9150138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 1 -- 172.21.15.171:0/3631419217 shutdown_connections 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 1 -- 172.21.15.171:0/3631419217 >> 172.21.15.171:0/3631419217 conn(0x7f91500a5450 msgr2=0x7f91500a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 1 -- 172.21.15.171:0/3631419217 shutdown_connections 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 1 -- 172.21.15.171:0/3631419217 wait complete. 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 10 monclient: build_initial_monmap 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 10 monclient: monmap: 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:09.156 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.142+0000 7f915714b640 1 Processor -- start 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 1 -- start start 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient: init 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient(hunting): picked mon.noname-a con 0x7f9150133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient(hunting): picked mon.noname-c con 0x7f91501341f0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient(hunting): picked mon.noname-b con 0x7f915013a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:09.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915011c1f0 con 0x7f9150133700 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915011c470 con 0x7f915013a350 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915011d920 con 0x7f91501341f0 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f915714b640 10 monclient(hunting): authenticate will time out at 1044.648677s 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914ffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f91501341f0 legacy=0x7f91501cd710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f91556e8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f915013a350 0x7f91501d0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42632/0 (socket says 172.21.15.171:42632) 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f91556e8640 1 -- 172.21.15.171:0/948495660 learned_addr learned my addr 172.21.15.171:0/948495660 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904653046 0 0) 0x7f915011c470 con 0x7f915013a350 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914d7fa640 10 monclient(hunting): my global_id is 5878 2024-09-16T04:11:09.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.143+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9134005730 con 0x7f915013a350 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228544752 0 0) 0x7f915011c1f0 con 0x7f9150133700 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): my global_id is 64317 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f915011c470 con 0x7f9150133700 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 757692233 0 0) 0x7f9134005730 con 0x7f915013a350 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f915011c1f0 con 0x7f915013a350 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9148004440 con 0x7f915013a350 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): dump: 2024-09-16T04:11:09.159 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2454424350 0 0) 0x7f915011c1f0 con 0x7f915013a350 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 >> v1:172.21.15.148:6790/0 conn(0x7f91501341f0 legacy=0x7f91501cd710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:09.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 >> v1:172.21.15.148:6789/0 conn(0x7f9150133700 legacy=0x7f91501c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 monclient: found mon.b 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91500a3e80 con 0x7f915013a350 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 5 monclient: authenticate success, global_id 5878 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 10 monclient: _renew_subs 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f915011d360 con 0x7f915013a350 2024-09-16T04:11:09.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 10 monclient: _renew_subs 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f91501395d0 con 0x7f915013a350 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f91480046f0 con 0x7f915013a350 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9148004ac0 con 0x7f915013a350 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914d7fa640 10 monclient: dump: 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:09.162 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.144+0000 7f914effd640 4 set_mon_vals no callback set 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9148004c60 con 0x7f915013a350 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f915714b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f915714b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f915714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f915011db60 con 0x7f915013a350 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(168..168 src has 1..168) ==== 7663+0+0 (unknown 1928016466 0 0) 0x7f9148047f50 con 0x7f915013a350 2024-09-16T04:11:09.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.145+0000 7f9154ee7640 10 monclient: get_auth_request con 0x7f913403ada0 auth_method 0 2024-09-16T04:11:09.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.150+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9148015f50 con 0x7f915013a350 2024-09-16T04:11:09.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.150+0000 7f914d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:09.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.150+0000 7f914d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:09.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.320+0000 7f915714b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:09.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.320+0000 7f915714b640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:09.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.320+0000 7f915714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:09.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.320+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) -- 0x7f915006bd60 con 0x7f915013a350 2024-09-16T04:11:09.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.951+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v169) ==== 148+0+0 (unknown 2891678725 0 0) 0x7f9148015850 con 0x7f915013a350 2024-09-16T04:11:09.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.951+0000 7f914d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:09.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:09.951+0000 7f914d7fa640 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-16T04:11:10.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.055+0000 7f915714b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:10.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.055+0000 7f915714b640 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:10.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.055+0000 7f915714b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.055+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) -- 0x7f915009bca0 con 0x7f915013a350 2024-09-16T04:11:10.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.056+0000 7f914d7fa640 1 -- 172.21.15.171:0/948495660 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v169) ==== 155+0+0 (unknown 2400769086 0 0) 0x7f9148019ba0 con 0x7f915013a350 2024-09-16T04:11:10.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.056+0000 7f914d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-16T04:11:10.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.056+0000 7f914d7fa640 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-16T04:11:10.058 INFO:tasks.workunit.client.0.smithi171.stderr:pool 'cold' does not exist 2024-09-16T04:11:10.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 >> v1:172.21.15.171:6812/7327273 conn(0x7f913403ada0 legacy=0x7f913403d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:10.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 10 monclient: shutdown 2024-09-16T04:11:10.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:10.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 >> v1:172.21.15.171:6789/0 conn(0x7f915013a350 legacy=0x7f91501d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:10.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 shutdown_connections 2024-09-16T04:11:10.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 >> 172.21.15.171:0/948495660 conn(0x7f91500a5450 msgr2=0x7f91500703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:10.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 shutdown_connections 2024-09-16T04:11:10.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.058+0000 7f915714b640 1 -- 172.21.15.171:0/948495660 wait complete. 2024-09-16T04:11:10.069 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T04:11:10.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.149+0000 7fe9bf552640 10 monclient: get_monmap_and_config 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.149+0000 7fe9bf552640 10 monclient: build_initial_monmap 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.149+0000 7fe9bf552640 10 monclient: monmap: 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:10.149843+0000 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:10.149843+0000 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:10.151 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:10.152 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.149+0000 7fe9bf552640 1 Processor -- start 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.149+0000 7fe9bf552640 1 -- start start 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.149+0000 7fe9bf552640 10 monclient: init 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b8136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b8139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b813d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b80a1340 con 0x7fe9b8139b10 2024-09-16T04:11:10.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b8137650 con 0x7fe9b8136b60 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b80a0aa0 con 0x7fe9b813d620 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 10 monclient(hunting): authenticate will time out at 1045.655474s 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bd2ee640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe9b8136b60 0x7fe9b8136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42638/0 (socket says 172.21.15.171:42638) 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bd2ee640 1 -- 172.21.15.171:0/2456919521 learned_addr learned my addr 172.21.15.171:0/2456919521 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bdaef640 1 -- 172.21.15.171:0/2456919521 >> v1:172.21.15.148:6790/0 conn(0x7fe9b813d620 legacy=0x7fe9b813fa10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1312065267 0 0) 0x7fe9b8137650 con 0x7fe9b8136b60 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient(hunting): my global_id is 5881 2024-09-16T04:11:10.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9a0003520 con 0x7fe9b8136b60 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3941320222 0 0) 0x7fe9a0003520 con 0x7fe9b8136b60 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 >> v1:172.21.15.148:6790/0 conn(0x7fe9b813d620 legacy=0x7fe9b813fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 >> v1:172.21.15.148:6789/0 conn(0x7fe9b8139b10 legacy=0x7fe9b813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 monclient: found mon.noname-c 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b80a0550 con 0x7fe9b8136b60 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9a8001990 con 0x7fe9b8136b60 2024-09-16T04:11:10.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 5 monclient: authenticate success, global_id 5881 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9bf552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.150+0000 7fe9b7fff640 10 monclient: dump: 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:10.162 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9a8001c70 con 0x7fe9b8136b60 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9b7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9b7fff640 1 -- 172.21.15.171:0/2456919521 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9a8002080 con 0x7fe9b8136b60 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 4 set_mon_vals no callback set 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9b7fff640 10 monclient: dump: 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:10.163 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 10 monclient: shutdown 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 1 -- 172.21.15.171:0/2456919521 >> v1:172.21.15.171:6789/0 conn(0x7fe9b8136b60 legacy=0x7fe9b8136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 1 -- 172.21.15.171:0/2456919521 shutdown_connections 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 1 -- 172.21.15.171:0/2456919521 >> 172.21.15.171:0/2456919521 conn(0x7fe9b80a98b0 msgr2=0x7fe9b80a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:10.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 1 -- 172.21.15.171:0/2456919521 shutdown_connections 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.151+0000 7fe9bf552640 1 -- 172.21.15.171:0/2456919521 wait complete. 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient: build_initial_monmap 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient: monmap: 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:10.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 1 Processor -- start 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 1 -- start start 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient: init 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b8136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b8139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b813d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b809ebd0 con 0x7fe9b8136b60 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.152+0000 7fe9bf552640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b80a0240 con 0x7fe9b8139b10 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b80a27f0 con 0x7fe9b813d620 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 10 monclient(hunting): authenticate will time out at 1045.658015s 2024-09-16T04:11:10.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bcaed640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe9b8139b10 0x7fe9b809d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42644/0 (socket says 172.21.15.171:42644) 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bdaef640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe9b813d620 legacy=0x7fe9b809dea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bcaed640 1 -- 172.21.15.171:0/3782279337 learned_addr learned my addr 172.21.15.171:0/3782279337 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315342200 0 0) 0x7fe9b80a0240 con 0x7fe9b8139b10 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): my global_id is 5884 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9a4003520 con 0x7fe9b8139b10 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174458066 0 0) 0x7fe9b809ebd0 con 0x7fe9b8136b60 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:10.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): my global_id is 64323 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9b80a0240 con 0x7fe9b8136b60 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3040216490 0 0) 0x7fe9a4003520 con 0x7fe9b8139b10 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9b809ebd0 con 0x7fe9b8139b10 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9ac0019e0 con 0x7fe9b8139b10 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): dump: 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.168 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 447749310 0 0) 0x7fe9b809ebd0 con 0x7fe9b8139b10 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 >> v1:172.21.15.148:6790/0 conn(0x7fe9b813d620 legacy=0x7fe9b809dea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 >> v1:172.21.15.148:6789/0 conn(0x7fe9b8136b60 legacy=0x7fe9b809d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 monclient: found mon.b 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:10.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b811f6a0 con 0x7fe9b8139b10 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 5 monclient: authenticate success, global_id 5884 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 10 monclient: _renew_subs 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b811fbf0 con 0x7fe9b8139b10 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9ac0038c0 con 0x7fe9b8139b10 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9ac003c90 con 0x7fe9b8139b10 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b5ffb640 10 monclient: dump: 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:10.171 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 10 monclient: _renew_subs 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9b813cfe0 con 0x7fe9b8139b10 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.153+0000 7fe9b77fe640 4 set_mon_vals no callback set 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe9ac003e30 con 0x7fe9b8139b10 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9bf552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9bf552640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:10.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9bf552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9b80008d0 con 0x7fe9b8139b10 2024-09-16T04:11:10.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(169..169 src has 1..169) ==== 6856+0+0 (unknown 1398092525 0 0) 0x7fe9ac047b50 con 0x7fe9b8139b10 2024-09-16T04:11:10.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.154+0000 7fe9bd2ee640 10 monclient: get_auth_request con 0x7fe9a4038bc0 auth_method 0 2024-09-16T04:11:10.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.159+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe9ac015d70 con 0x7fe9b8139b10 2024-09-16T04:11:10.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.159+0000 7fe9b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:10.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.159+0000 7fe9b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:10.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.331+0000 7fe9bf552640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:10.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.331+0000 7fe9bf552640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:10.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.331+0000 7fe9bf552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:10.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.331+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fe9b80a4ca0 con 0x7fe9b8139b10 2024-09-16T04:11:10.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.957+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v170) ==== 78+0+0 (unknown 1576154579 0 0) 0x7fe9ac015670 con 0x7fe9b8139b10 2024-09-16T04:11:10.960 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.957+0000 7fe9b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:10.961 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:10.957+0000 7fe9b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:11.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.073+0000 7fe9bf552640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:11.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.073+0000 7fe9bf552640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:11.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.073+0000 7fe9bf552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:11.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.073+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fe9b81d4800 con 0x7fe9b8139b10 2024-09-16T04:11:11.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.962+0000 7fe9b5ffb640 1 -- 172.21.15.171:0/3782279337 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v171) ==== 78+0+0 (unknown 473137142 0 0) 0x7fe9ac0199c0 con 0x7fe9b8139b10 2024-09-16T04:11:11.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.962+0000 7fe9b5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:11.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.962+0000 7fe9b5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:11.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.974+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 >> v1:172.21.15.171:6812/7327273 conn(0x7fe9a4038bc0 legacy=0x7fe9a403b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:11.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.974+0000 7fe9bf552640 10 monclient: shutdown 2024-09-16T04:11:11.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.974+0000 7fe9bf552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:11.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.974+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 >> v1:172.21.15.171:6789/0 conn(0x7fe9b8139b10 legacy=0x7fe9b809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:11.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.975+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 shutdown_connections 2024-09-16T04:11:11.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.975+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 >> 172.21.15.171:0/3782279337 conn(0x7fe9b80a98b0 msgr2=0x7fe9b81391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:11.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.975+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 shutdown_connections 2024-09-16T04:11:11.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:11.975+0000 7fe9bf552640 1 -- 172.21.15.171:0/3782279337 wait complete. 2024-09-16T04:11:11.992 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T04:11:12.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.072+0000 7ff827c1e640 10 monclient: get_monmap_and_config 2024-09-16T04:11:12.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.072+0000 7ff827c1e640 10 monclient: build_initial_monmap 2024-09-16T04:11:12.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.072+0000 7ff827c1e640 10 monclient: monmap: 2024-09-16T04:11:12.075 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:12.075 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:12.074334+0000 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:12.074334+0000 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:12.076 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 1 Processor -- start 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 1 -- start start 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient: init 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient(hunting): picked mon.noname-c con 0x7ff82009ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8200a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient(hunting): picked mon.noname-b con 0x7ff82009e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82009fbf0 con 0x7ff8200a0750 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8200a0e10 con 0x7ff82009ba30 2024-09-16T04:11:12.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82009f640 con 0x7ff82009e570 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff827c1e640 10 monclient(hunting): authenticate will time out at 1047.579850s 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff8259ba640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff82009ba30 0x7ff8200a0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42692/0 (socket says 172.21.15.171:42692) 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff8259ba640 1 -- 172.21.15.171:0/2290118796 learned_addr learned my addr 172.21.15.171:0/2290118796 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.073+0000 7ff8261bb640 1 -- 172.21.15.171:0/2290118796 >> v1:172.21.15.148:6790/0 conn(0x7ff82009e570 legacy=0x7ff82013d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077560499 0 0) 0x7ff8200a0e10 con 0x7ff82009ba30 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient(hunting): _negotiate 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient(hunting): my global_id is 5887 2024-09-16T04:11:12.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff80c003520 con 0x7ff82009ba30 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2724749440 0 0) 0x7ff80c003520 con 0x7ff82009ba30 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 >> v1:172.21.15.148:6790/0 conn(0x7ff82009e570 legacy=0x7ff82013d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 >> v1:172.21.15.148:6789/0 conn(0x7ff8200a0750 legacy=0x7ff8201399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: _finish_hunting 0 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 monclient: found mon.noname-c 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82009f810 con 0x7ff82009ba30 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: _finish_auth 0 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: _check_auth_tickets 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff810001610 con 0x7ff82009ba30 2024-09-16T04:11:12.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff827c1e640 5 monclient: authenticate success, global_id 5887 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: dump: 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:12.086 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff827c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff810001970 con 0x7ff82009ba30 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 1 -- 172.21.15.171:0/2290118796 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff8100048a0 con 0x7ff82009ba30 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff8249b8640 10 monclient: dump: 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:12.087 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff827c1e640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.074+0000 7ff827c1e640 4 set_mon_vals no callback set 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 10 monclient: shutdown 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 1 -- 172.21.15.171:0/2290118796 >> v1:172.21.15.171:6789/0 conn(0x7ff82009ba30 legacy=0x7ff8200a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 1 -- 172.21.15.171:0/2290118796 shutdown_connections 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 1 -- 172.21.15.171:0/2290118796 >> 172.21.15.171:0/2290118796 conn(0x7ff8200a52a0 msgr2=0x7ff8200a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 1 -- 172.21.15.171:0/2290118796 shutdown_connections 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 1 -- 172.21.15.171:0/2290118796 wait complete. 2024-09-16T04:11:12.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 10 monclient: build_initial_monmap 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 10 monclient: monmap: 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.075+0000 7ff827c1e640 1 Processor -- start 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 1 -- start start 2024-09-16T04:11:12.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient: init 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient(hunting): picked mon.noname-b con 0x7ff82009ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient(hunting): picked mon.noname-a con 0x7ff82009e570 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient(hunting): picked mon.noname-c con 0x7ff8200a0750 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82012ef20 con 0x7ff82009e570 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82012d6a0 con 0x7ff82009ba30 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82012dd50 con 0x7ff8200a0750 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff827c1e640 10 monclient(hunting): authenticate will time out at 1047.582525s 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8261bb640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7ff8200a0750 legacy=0x7ff8201d2650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8259ba640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff82009ba30 0x7ff820127750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42702/0 (socket says 172.21.15.171:42702) 2024-09-16T04:11:12.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8259ba640 1 -- 172.21.15.171:0/3634189634 learned_addr learned my addr 172.21.15.171:0/3634189634 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039527548 0 0) 0x7ff82012d6a0 con 0x7ff82009ba30 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): my global_id is 5890 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff804003520 con 0x7ff82009ba30 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734645988 0 0) 0x7ff82012ef20 con 0x7ff82009e570 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): my global_id is 64329 2024-09-16T04:11:12.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff82012d6a0 con 0x7ff82009e570 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1039043588 0 0) 0x7ff804003520 con 0x7ff82009ba30 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff82012ef20 con 0x7ff82009ba30 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff810001430 con 0x7ff82009ba30 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): dump: 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:12.092 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.076+0000 7ff8167fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 875636343 0 0) 0x7ff82012ef20 con 0x7ff82009ba30 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 >> v1:172.21.15.148:6790/0 conn(0x7ff8200a0750 legacy=0x7ff8201d2650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 >> v1:172.21.15.148:6789/0 conn(0x7ff82009e570 legacy=0x7ff8201cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 monclient: found mon.b 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82012d210 con 0x7ff82009ba30 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:12.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 5 monclient: authenticate success, global_id 5890 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 10 monclient: _renew_subs 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff8201329e0 con 0x7ff82009ba30 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 10 monclient: _renew_subs 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff82012c8f0 con 0x7ff82009ba30 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff810001970 con 0x7ff82009ba30 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff810004dc0 con 0x7ff82009ba30 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:12.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 10 monclient: dump: 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff817fff640 4 set_mon_vals no callback set 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff8100101f0 con 0x7ff82009ba30 2024-09-16T04:11:12.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.078+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(171..171 src has 1..171) ==== 6433+0+0 (unknown 3954986824 0 0) 0x7ff810047890 con 0x7ff82009ba30 2024-09-16T04:11:12.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.078+0000 7ff8251b9640 10 monclient: get_auth_request con 0x7ff804038fb0 auth_method 0 2024-09-16T04:11:12.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.078+0000 7ff827c1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:12.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.078+0000 7ff827c1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:12.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.078+0000 7ff827c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:12.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.078+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff808005080 con 0x7ff82009ba30 2024-09-16T04:11:12.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.082+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff810015d30 con 0x7ff82009ba30 2024-09-16T04:11:12.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.082+0000 7ff8167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:12.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.082+0000 7ff8167fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:12.258 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.255+0000 7ff827c1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:12.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.255+0000 7ff827c1e640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:12.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.255+0000 7ff827c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:12.259 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.255+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7ff808005370 con 0x7ff82009ba30 2024-09-16T04:11:12.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.964+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v172) ==== 82+0+0 (unknown 3073799927 0 0) 0x7ff810015630 con 0x7ff82009ba30 2024-09-16T04:11:12.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.964+0000 7ff8167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:12.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:12.964+0000 7ff8167fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:13.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.076+0000 7ff827c1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:13.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.076+0000 7ff827c1e640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:13.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.076+0000 7ff827c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.076+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7ff808002210 con 0x7ff82009ba30 2024-09-16T04:11:13.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.077+0000 7ff8167fc640 1 -- 172.21.15.171:0/3634189634 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v172) ==== 115+0+0 (unknown 3379692229 0 0) 0x7ff81001e180 con 0x7ff82009ba30 2024-09-16T04:11:13.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.077+0000 7ff8167fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:13.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.077+0000 7ff8167fc640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T04:11:13.080 INFO:tasks.workunit.client.0.smithi171.stderr:compat weight-set already created 2024-09-16T04:11:13.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 >> v1:172.21.15.171:6812/7327273 conn(0x7ff804038fb0 legacy=0x7ff80403b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:13.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 10 monclient: shutdown 2024-09-16T04:11:13.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:13.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 >> v1:172.21.15.171:6789/0 conn(0x7ff82009ba30 legacy=0x7ff820127750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:13.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 shutdown_connections 2024-09-16T04:11:13.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 >> 172.21.15.171:0/3634189634 conn(0x7ff8200a52a0 msgr2=0x7ff820131d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:13.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 shutdown_connections 2024-09-16T04:11:13.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.079+0000 7ff827c1e640 1 -- 172.21.15.171:0/3634189634 wait complete. 2024-09-16T04:11:13.092 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket fooo host 2024-09-16T04:11:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.170+0000 7fb75fdb7640 10 monclient: get_monmap_and_config 2024-09-16T04:11:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.170+0000 7fb75fdb7640 10 monclient: build_initial_monmap 2024-09-16T04:11:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.170+0000 7fb75fdb7640 10 monclient: monmap: 2024-09-16T04:11:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:13.172335+0000 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:13.172335+0000 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:13.174 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 1 Processor -- start 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 1 -- start start 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient: init 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient(hunting): picked mon.noname-c con 0x7fb75809aa20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient(hunting): picked mon.noname-a con 0x7fb75809ee90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient(hunting): picked mon.noname-b con 0x7fb75809b510 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75809d000 con 0x7fb75809ee90 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75809d5c0 con 0x7fb75809aa20 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75809d8a0 con 0x7fb75809b510 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.171+0000 7fb75fdb7640 10 monclient(hunting): authenticate will time out at 1048.677916s 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75db53640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb75809aa20 0x7fb75809ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42714/0 (socket says 172.21.15.171:42714) 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75db53640 1 -- 172.21.15.171:0/797992582 learned_addr learned my addr 172.21.15.171:0/797992582 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75e354640 1 -- 172.21.15.171:0/797992582 >> v1:172.21.15.148:6790/0 conn(0x7fb75809b510 legacy=0x7fb75813bae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1646700328 0 0) 0x7fb75809d5c0 con 0x7fb75809aa20 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): _negotiate 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): my global_id is 5893 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb744003520 con 0x7fb75809aa20 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048115763 0 0) 0x7fb75809d000 con 0x7fb75809ee90 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): _negotiate 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient(hunting): my global_id is 64332 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb75809d5c0 con 0x7fb75809ee90 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4015940881 0 0) 0x7fb744003520 con 0x7fb75809aa20 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 >> v1:172.21.15.148:6790/0 conn(0x7fb75809b510 legacy=0x7fb75813bae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 >> v1:172.21.15.148:6789/0 conn(0x7fb75809ee90 legacy=0x7fb7581383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: _finish_hunting 0 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 monclient: found mon.noname-c 2024-09-16T04:11:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7580a0830 con 0x7fb75809aa20 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: _finish_auth 0 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: _check_auth_tickets 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb754001610 con 0x7fb75809aa20 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75fdb7640 5 monclient: authenticate success, global_id 5893 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75fdb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: dump: 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75fdb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb754001930 con 0x7fb75809aa20 2024-09-16T04:11:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 1 -- 172.21.15.171:0/797992582 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb754001d00 con 0x7fb75809aa20 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75fdb7640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75fdb7640 4 set_mon_vals no callback set 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.172+0000 7fb75cb51640 10 monclient: dump: 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 10 monclient: shutdown 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 1 -- 172.21.15.171:0/797992582 >> v1:172.21.15.171:6789/0 conn(0x7fb75809aa20 legacy=0x7fb75809ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 1 -- 172.21.15.171:0/797992582 shutdown_connections 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 1 -- 172.21.15.171:0/797992582 >> 172.21.15.171:0/797992582 conn(0x7fb7580a5370 msgr2=0x7fb7580a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 1 -- 172.21.15.171:0/797992582 shutdown_connections 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 1 -- 172.21.15.171:0/797992582 wait complete. 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 10 monclient: build_initial_monmap 2024-09-16T04:11:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.173+0000 7fb75fdb7640 10 monclient: monmap: 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 1 Processor -- start 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 1 -- start start 2024-09-16T04:11:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient: init 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient(hunting): picked mon.noname-c con 0x7fb75809aa20 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient(hunting): picked mon.noname-b con 0x7fb75809b510 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient(hunting): picked mon.noname-a con 0x7fb75809ee90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7581c3850 con 0x7fb75809ee90 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7581a7fe0 con 0x7fb75809b510 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7581aaaa0 con 0x7fb75809aa20 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75fdb7640 10 monclient(hunting): authenticate will time out at 1048.680615s 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75db53640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb75809aa20 legacy=0x7fb7581c3140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75d352640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb75809b510 0x7fb7581ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42728/0 (socket says 172.21.15.171:42728) 2024-09-16T04:11:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb75d352640 1 -- 172.21.15.171:0/592171088 learned_addr learned my addr 172.21.15.171:0/592171088 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2029751559 0 0) 0x7fb7581a7fe0 con 0x7fb75809b510 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb74e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb74e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb74e7fc640 10 monclient(hunting): my global_id is 5896 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.174+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb73c003520 con 0x7fb75809b510 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 880784124 0 0) 0x7fb7581c3850 con 0x7fb75809ee90 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): my global_id is 64335 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7581a7fe0 con 0x7fb75809ee90 2024-09-16T04:11:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2299810947 0 0) 0x7fb73c003520 con 0x7fb75809b510 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7581c3850 con 0x7fb75809b510 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb748001910 con 0x7fb75809b510 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): dump: 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2017697950 0 0) 0x7fb7581c3850 con 0x7fb75809b510 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 >> v1:172.21.15.148:6790/0 conn(0x7fb75809aa20 legacy=0x7fb7581c3140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 >> v1:172.21.15.148:6789/0 conn(0x7fb75809ee90 legacy=0x7fb7581d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 monclient: found mon.b 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb758072c20 con 0x7fb75809b510 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:13.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 5 monclient: authenticate success, global_id 5896 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 10 monclient: _renew_subs 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7581a87d0 con 0x7fb75809b510 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 10 monclient: _renew_subs 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7581a9550 con 0x7fb75809b510 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb748003610 con 0x7fb75809b510 2024-09-16T04:11:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:13.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb748003960 con 0x7fb75809b510 2024-09-16T04:11:13.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:13.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:13.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74e7fc640 10 monclient: dump: 2024-09-16T04:11:13.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.175+0000 7fb74ffff640 4 set_mon_vals no callback set 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb748003b80 con 0x7fb75809b510 2024-09-16T04:11:13.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb75fdb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:13.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb75fdb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:13.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb75fdb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb740005080 con 0x7fb75809b510 2024-09-16T04:11:13.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(172..172 src has 1..172) ==== 6761+0+0 (unknown 4267742823 0 0) 0x7fb748047a50 con 0x7fb75809b510 2024-09-16T04:11:13.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.176+0000 7fb75db53640 10 monclient: get_auth_request con 0x7fb73c038fb0 auth_method 0 2024-09-16T04:11:13.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.180+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb748015d10 con 0x7fb75809b510 2024-09-16T04:11:13.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.180+0000 7fb74e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:13.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.180+0000 7fb74e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:13.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.349+0000 7fb75fdb7640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T04:11:13.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.349+0000 7fb75fdb7640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T04:11:13.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.349+0000 7fb75fdb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:13.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.349+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) -- 0x7fb740005b80 con 0x7fb75809b510 2024-09-16T04:11:13.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.973+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v173) ==== 140+0+0 (unknown 3737524602 0 0) 0x7fb7480153a0 con 0x7fb75809b510 2024-09-16T04:11:13.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.973+0000 7fb74e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T04:11:13.976 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:13.973+0000 7fb74e7fc640 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-16T04:11:14.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.075+0000 7fb75fdb7640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T04:11:14.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.075+0000 7fb75fdb7640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T04:11:14.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.075+0000 7fb75fdb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.075+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) -- 0x7fb740001090 con 0x7fb75809b510 2024-09-16T04:11:14.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.076+0000 7fb74e7fc640 1 -- 172.21.15.171:0/592171088 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v173) ==== 128+0+0 (unknown 351310710 0 0) 0x7fb748015610 con 0x7fb75809b510 2024-09-16T04:11:14.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.076+0000 7fb74e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-16T04:11:14.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.076+0000 7fb74e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-16T04:11:14.080 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'fooo' already exists 2024-09-16T04:11:14.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 >> v1:172.21.15.171:6812/7327273 conn(0x7fb73c038fb0 legacy=0x7fb73c03b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:14.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 10 monclient: shutdown 2024-09-16T04:11:14.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:14.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 >> v1:172.21.15.171:6789/0 conn(0x7fb75809b510 legacy=0x7fb7581ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:14.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 shutdown_connections 2024-09-16T04:11:14.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 >> 172.21.15.171:0/592171088 conn(0x7fb7580a5370 msgr2=0x7fb758071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:14.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 shutdown_connections 2024-09-16T04:11:14.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.078+0000 7fb75fdb7640 1 -- 172.21.15.171:0/592171088 wait complete. 2024-09-16T04:11:14.090 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush move fooo root=default 2024-09-16T04:11:14.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.168+0000 7fb982b9f640 10 monclient: get_monmap_and_config 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.169+0000 7fb982b9f640 10 monclient: build_initial_monmap 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.169+0000 7fb982b9f640 10 monclient: monmap: 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:14.170792+0000 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:14.170792+0000 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:14.172 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.169+0000 7fb982b9f640 1 Processor -- start 2024-09-16T04:11:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.169+0000 7fb982b9f640 1 -- start start 2024-09-16T04:11:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.169+0000 7fb982b9f640 10 monclient: init 2024-09-16T04:11:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient(hunting): picked mon.noname-b con 0x7fb97c09fa10 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient(hunting): picked mon.noname-c con 0x7fb97c073580 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient(hunting): picked mon.noname-a con 0x7fb97c0a0500 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c0a11d0 con 0x7fb97c0a0500 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c0a0ba0 con 0x7fb97c073580 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c09bca0 con 0x7fb97c09fa10 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 10 monclient(hunting): authenticate will time out at 1049.676287s 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb973fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb97c073580 0x7fb97c13beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42738/0 (socket says 172.21.15.171:42738) 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb973fff640 1 -- 172.21.15.171:0/2344036972 learned_addr learned my addr 172.21.15.171:0/2344036972 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb98093b640 1 -- 172.21.15.171:0/2344036972 >> v1:172.21.15.148:6790/0 conn(0x7fb97c09fa10 legacy=0x7fb97c09fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506052878 0 0) 0x7fb97c0a0ba0 con 0x7fb97c073580 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): my global_id is 5899 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb964003520 con 0x7fb97c073580 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388288979 0 0) 0x7fb97c0a11d0 con 0x7fb97c0a0500 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient(hunting): my global_id is 64338 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb97c0a0ba0 con 0x7fb97c0a0500 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 229443429 0 0) 0x7fb964003520 con 0x7fb97c073580 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 >> v1:172.21.15.148:6790/0 conn(0x7fb97c09fa10 legacy=0x7fb97c09fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 >> v1:172.21.15.148:6789/0 conn(0x7fb97c0a0500 legacy=0x7fb97c13f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 monclient: found mon.noname-c 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c0a1630 con 0x7fb97c073580 2024-09-16T04:11:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 5 monclient: authenticate success, global_id 5899 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb96c001610 con 0x7fb97c073580 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: dump: 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb982b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb96c001930 con 0x7fb97c073580 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.170+0000 7fb9737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb9737fe640 1 -- 172.21.15.171:0/2344036972 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb96c001d00 con 0x7fb97c073580 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb9737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 4 set_mon_vals no callback set 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb9737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb9737fe640 10 monclient: dump: 2024-09-16T04:11:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 10 monclient: shutdown 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 1 -- 172.21.15.171:0/2344036972 >> v1:172.21.15.171:6789/0 conn(0x7fb97c073580 legacy=0x7fb97c13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 1 -- 172.21.15.171:0/2344036972 shutdown_connections 2024-09-16T04:11:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 1 -- 172.21.15.171:0/2344036972 >> 172.21.15.171:0/2344036972 conn(0x7fb97c0a1b10 msgr2=0x7fb97c0a1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 1 -- 172.21.15.171:0/2344036972 shutdown_connections 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.171+0000 7fb982b9f640 1 -- 172.21.15.171:0/2344036972 wait complete. 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 10 monclient: build_initial_monmap 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 10 monclient: monmap: 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 1 Processor -- start 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 1 -- start start 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 10 monclient: init 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.172+0000 7fb982b9f640 10 monclient(hunting): picked mon.noname-c con 0x7fb97c073580 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient(hunting): picked mon.noname-b con 0x7fb97c09fa10 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient(hunting): picked mon.noname-a con 0x7fb97c0a0500 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c09c200 con 0x7fb97c0a0500 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c122290 con 0x7fb97c09fa10 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb97c122570 con 0x7fb97c073580 2024-09-16T04:11:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient(hunting): authenticate will time out at 1049.679055s 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb98093b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb97c073580 legacy=0x7fb97c1cb540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb973fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb97c09fa10 0x7fb97c134e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42740/0 (socket says 172.21.15.171:42740) 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb973fff640 1 -- 172.21.15.171:0/2048526402 learned_addr learned my addr 172.21.15.171:0/2048526402 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536579597 0 0) 0x7fb97c122290 con 0x7fb97c09fa10 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): my global_id is 5902 2024-09-16T04:11:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb968003520 con 0x7fb97c09fa10 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 114432553 0 0) 0x7fb97c0a0ba0 con 0x7fb97c0a0500 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): my global_id is 64341 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb97c122290 con 0x7fb97c0a0500 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3561783418 0 0) 0x7fb968003520 con 0x7fb97c09fa10 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb97c0a0ba0 con 0x7fb97c09fa10 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb96c0015d0 con 0x7fb97c09fa10 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): dump: 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2396060904 0 0) 0x7fb97c122290 con 0x7fb97c0a0500 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb968003520 con 0x7fb97c0a0500 2024-09-16T04:11:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb974001c30 con 0x7fb97c0a0500 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient(hunting): dump: 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2730292385 0 0) 0x7fb97c0a0ba0 con 0x7fb97c09fa10 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 >> v1:172.21.15.148:6790/0 conn(0x7fb97c073580 legacy=0x7fb97c1cb540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 >> v1:172.21.15.148:6789/0 conn(0x7fb97c0a0500 legacy=0x7fb97c1d38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: _finish_hunting 0 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 monclient: found mon.b 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c136810 con 0x7fb97c09fa10 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 5 monclient: authenticate success, global_id 5902 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient: _renew_subs 2024-09-16T04:11:14.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb97c120e40 con 0x7fb97c09fa10 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient: _renew_subs 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb97c135780 con 0x7fb97c09fa10 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb96c004cc0 con 0x7fb97c09fa10 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb96c005010 con 0x7fb97c09fa10 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb9717fa640 10 monclient: dump: 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.196 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.173+0000 7fb972ffd640 4 set_mon_vals no callback set 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.174+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb96c010440 con 0x7fb97c09fa10 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.174+0000 7fb982b9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.174+0000 7fb982b9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.174+0000 7fb982b9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.174+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97c0a1160 con 0x7fb97c09fa10 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.174+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(173..173 src has 1..173) ==== 6789+0+0 (unknown 144836719 0 0) 0x7fb96c047b10 con 0x7fb97c09fa10 2024-09-16T04:11:14.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.175+0000 7fb98093b640 10 monclient: get_auth_request con 0x7fb968038ea0 auth_method 0 2024-09-16T04:11:14.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.179+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb97c0a1160 con 0x7fb97c09fa10 2024-09-16T04:11:14.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.179+0000 7fb9717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:14.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.179+0000 7fb9717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:14.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.350+0000 7fb982b9f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T04:11:14.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.350+0000 7fb982b9f640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T04:11:14.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.350+0000 7fb982b9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:14.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.350+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) -- 0x7fb97c0a2f10 con 0x7fb97c09fa10 2024-09-16T04:11:14.977 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.975+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v174) ==== 173+0+0 (unknown 3722164873 0 0) 0x7fb97c0a2f10 con 0x7fb97c09fa10 2024-09-16T04:11:14.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.975+0000 7fb9717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T04:11:14.978 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:14.975+0000 7fb9717fa640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T04:11:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.080+0000 7fb982b9f640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T04:11:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.080+0000 7fb982b9f640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T04:11:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.080+0000 7fb982b9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.080+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) -- 0x7fb97c069a20 con 0x7fb97c09fa10 2024-09-16T04:11:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.081+0000 7fb9717fa640 1 -- 172.21.15.171:0/2048526402 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v174) ==== 183+0+0 (unknown 2005285716 0 0) 0x7fb96c015e50 con 0x7fb97c09fa10 2024-09-16T04:11:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.081+0000 7fb9717fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-16T04:11:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.081+0000 7fb9717fa640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T04:11:15.084 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 >> v1:172.21.15.171:6812/7327273 conn(0x7fb968038ea0 legacy=0x7fb96803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 10 monclient: shutdown 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 >> v1:172.21.15.171:6789/0 conn(0x7fb97c09fa10 legacy=0x7fb97c134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 shutdown_connections 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 >> 172.21.15.171:0/2048526402 conn(0x7fb97c0a1b10 msgr2=0x7fb97c135ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 shutdown_connections 2024-09-16T04:11:15.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.083+0000 7fb982b9f640 1 -- 172.21.15.171:0/2048526402 wait complete. 2024-09-16T04:11:15.095 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket barr rack 2024-09-16T04:11:15.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.172+0000 7fb675eba640 10 monclient: get_monmap_and_config 2024-09-16T04:11:15.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.172+0000 7fb675eba640 10 monclient: build_initial_monmap 2024-09-16T04:11:15.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.172+0000 7fb675eba640 10 monclient: monmap: 2024-09-16T04:11:15.175 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:15.174557+0000 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:15.174557+0000 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:15.176 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:15.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.173+0000 7fb675eba640 1 Processor -- start 2024-09-16T04:11:15.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.173+0000 7fb675eba640 1 -- start start 2024-09-16T04:11:15.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.173+0000 7fb675eba640 10 monclient: init 2024-09-16T04:11:15.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.173+0000 7fb675eba640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:15.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.173+0000 7fb675eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 10 monclient(hunting): picked mon.noname-a con 0x7fb6701327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 10 monclient(hunting): picked mon.noname-c con 0x7fb67009eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 10 monclient(hunting): picked mon.noname-b con 0x7fb670139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67009ef30 con 0x7fb6701327c0 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67009d880 con 0x7fb67009eb20 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67009db60 con 0x7fb670139ea0 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 10 monclient(hunting): authenticate will time out at 1050.680124s 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb67009eb20 0x7fb670138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42750/0 (socket says 172.21.15.171:42750) 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66effd640 1 -- 172.21.15.171:0/1523541103 learned_addr learned my addr 172.21.15.171:0/1523541103 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66ffff640 1 -- 172.21.15.171:0/1523541103 >> v1:172.21.15.148:6790/0 conn(0x7fb670139ea0 legacy=0x7fb67013c290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:15.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 991162399 0 0) 0x7fb67009d880 con 0x7fb67009eb20 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): my global_id is 5905 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb658003520 con 0x7fb67009eb20 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122185207 0 0) 0x7fb67009ef30 con 0x7fb6701327c0 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient(hunting): my global_id is 64344 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb67009d880 con 0x7fb6701327c0 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3393033075 0 0) 0x7fb658003520 con 0x7fb67009eb20 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 >> v1:172.21.15.148:6790/0 conn(0x7fb670139ea0 legacy=0x7fb67013c290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:15.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 >> v1:172.21.15.148:6789/0 conn(0x7fb6701327c0 legacy=0x7fb670132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67009de40 con 0x7fb67009eb20 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 5 monclient: authenticate success, global_id 5905 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb664001610 con 0x7fb67009eb20 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:15.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: dump: 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6640018c0 con 0x7fb67009eb20 2024-09-16T04:11:15.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 1 -- 172.21.15.171:0/1523541103 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb664001c90 con 0x7fb67009eb20 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb675eba640 4 set_mon_vals no callback set 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.174+0000 7fb66e7fc640 10 monclient: dump: 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:15.189 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 10 monclient: shutdown 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 1 -- 172.21.15.171:0/1523541103 >> v1:172.21.15.171:6789/0 conn(0x7fb67009eb20 legacy=0x7fb670138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 1 -- 172.21.15.171:0/1523541103 shutdown_connections 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 1 -- 172.21.15.171:0/1523541103 >> 172.21.15.171:0/1523541103 conn(0x7fb6700a5370 msgr2=0x7fb6700a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 1 -- 172.21.15.171:0/1523541103 shutdown_connections 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 1 -- 172.21.15.171:0/1523541103 wait complete. 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 10 monclient: build_initial_monmap 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.175+0000 7fb675eba640 10 monclient: monmap: 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:15.190 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 1 Processor -- start 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 1 -- start start 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient: init 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient(hunting): picked mon.noname-b con 0x7fb67009eb20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:15.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient(hunting): picked mon.noname-a con 0x7fb6701327c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient(hunting): picked mon.noname-c con 0x7fb670139ea0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6701cb900 con 0x7fb6701327c0 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb670120020 con 0x7fb67009eb20 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb670120300 con 0x7fb670139ea0 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb675eba640 10 monclient(hunting): authenticate will time out at 1050.682908s 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb66f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb67009eb20 0x7fb670131930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42764/0 (socket says 172.21.15.171:42764) 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.176+0000 7fb66f7fe640 1 -- 172.21.15.171:0/3217962728 learned_addr learned my addr 172.21.15.171:0/3217962728 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:15.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb66ffff640 1 -- 172.21.15.171:0/3217962728 >> v1:172.21.15.148:6790/0 conn(0x7fb670139ea0 legacy=0x7fb6701d3e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408459439 0 0) 0x7fb670120020 con 0x7fb67009eb20 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): my global_id is 5908 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb65c003520 con 0x7fb67009eb20 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2311828469 0 0) 0x7fb65c003520 con 0x7fb67009eb20 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb670120020 con 0x7fb67009eb20 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb6600018d0 con 0x7fb67009eb20 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): dump: 2024-09-16T04:11:15.193 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989528549 0 0) 0x7fb67009de40 con 0x7fb6701327c0 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:15.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient(hunting): my global_id is 64347 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb65c003520 con 0x7fb6701327c0 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2038347689 0 0) 0x7fb670120020 con 0x7fb67009eb20 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 >> v1:172.21.15.148:6790/0 conn(0x7fb670139ea0 legacy=0x7fb6701d3e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 >> v1:172.21.15.148:6789/0 conn(0x7fb6701327c0 legacy=0x7fb6701d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 monclient: found mon.b 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb670139b90 con 0x7fb67009eb20 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 5 monclient: authenticate success, global_id 5908 2024-09-16T04:11:15.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 10 monclient: _renew_subs 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb670139160 con 0x7fb67009eb20 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 10 monclient: _renew_subs 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb670138740 con 0x7fb67009eb20 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb660001c90 con 0x7fb67009eb20 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb660003ae0 con 0x7fb67009eb20 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb66dffb640 4 set_mon_vals no callback set 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.177+0000 7fb657fff640 10 monclient: dump: 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:15.196 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb660003d00 con 0x7fb67009eb20 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb675eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb675eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb675eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb650005080 con 0x7fb67009eb20 2024-09-16T04:11:15.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(174..174 src has 1..174) ==== 6977+0+0 (unknown 4050757488 0 0) 0x7fb660047bb0 con 0x7fb67009eb20 2024-09-16T04:11:15.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.178+0000 7fb66effd640 10 monclient: get_auth_request con 0x7fb65c038fc0 auth_method 0 2024-09-16T04:11:15.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.183+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb660015d10 con 0x7fb67009eb20 2024-09-16T04:11:15.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.183+0000 7fb657fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:15.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.183+0000 7fb657fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:15.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.351+0000 7fb675eba640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T04:11:15.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.351+0000 7fb675eba640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T04:11:15.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.351+0000 7fb675eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.351+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) -- 0x7fb650005b80 con 0x7fb67009eb20 2024-09-16T04:11:15.820 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.817+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v175) ==== 140+0+0 (unknown 1175284939 0 0) 0x7fb6600199c0 con 0x7fb67009eb20 2024-09-16T04:11:15.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.817+0000 7fb657fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T04:11:15.821 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.817+0000 7fb657fff640 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-16T04:11:15.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.922+0000 7fb675eba640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T04:11:15.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.922+0000 7fb675eba640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T04:11:15.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.922+0000 7fb675eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:15.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.922+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) -- 0x7fb650001090 con 0x7fb67009eb20 2024-09-16T04:11:15.926 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.923+0000 7fb657fff640 1 -- 172.21.15.171:0/3217962728 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v175) ==== 128+0+0 (unknown 1342079467 0 0) 0x7fb66001e280 con 0x7fb67009eb20 2024-09-16T04:11:15.926 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.923+0000 7fb657fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-16T04:11:15.926 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.923+0000 7fb657fff640 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-16T04:11:15.926 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'barr' already exists 2024-09-16T04:11:15.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 >> v1:172.21.15.171:6812/7327273 conn(0x7fb65c038fc0 legacy=0x7fb65c03b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 10 monclient: shutdown 2024-09-16T04:11:15.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:15.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 >> v1:172.21.15.171:6789/0 conn(0x7fb67009eb20 legacy=0x7fb670131930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:15.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 shutdown_connections 2024-09-16T04:11:15.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 >> 172.21.15.171:0/3217962728 conn(0x7fb6700a5370 msgr2=0x7fb6700a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:15.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 shutdown_connections 2024-09-16T04:11:15.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:15.925+0000 7fb675eba640 1 -- 172.21.15.171:0/3217962728 wait complete. 2024-09-16T04:11:15.937 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush move barr root=default 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.018+0000 7fefce15b640 10 monclient: get_monmap_and_config 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.018+0000 7fefce15b640 10 monclient: build_initial_monmap 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.018+0000 7fefce15b640 10 monclient: monmap: 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:16.019931+0000 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:16.019931+0000 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:16.021 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:16.022 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:16.022 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:16.022 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.018+0000 7fefce15b640 1 Processor -- start 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.018+0000 7fefce15b640 1 -- start start 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.018+0000 7fefce15b640 10 monclient: init 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient(hunting): picked mon.noname-c con 0x7fefc8136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient(hunting): picked mon.noname-b con 0x7fefc8139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient(hunting): picked mon.noname-a con 0x7fefc813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc80a1340 con 0x7fefc813d620 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc8137650 con 0x7fefc8136b60 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc80a0aa0 con 0x7fefc8139b10 2024-09-16T04:11:16.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 10 monclient(hunting): authenticate will time out at 1051.525389s 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fefc8136b60 0x7fefc8136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42778/0 (socket says 172.21.15.171:42778) 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc77fe640 1 -- 172.21.15.171:0/614175709 learned_addr learned my addr 172.21.15.171:0/614175709 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc6ffd640 1 -- 172.21.15.171:0/614175709 >> v1:172.21.15.148:6790/0 conn(0x7fefc8139b10 legacy=0x7fefc813bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141424974 0 0) 0x7fefc8137650 con 0x7fefc8136b60 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient(hunting): my global_id is 5911 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefb0003520 con 0x7fefc8136b60 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 565887772 0 0) 0x7fefb0003520 con 0x7fefc8136b60 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 >> v1:172.21.15.148:6790/0 conn(0x7fefc8139b10 legacy=0x7fefc813bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:16.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 >> v1:172.21.15.148:6789/0 conn(0x7fefc813d620 legacy=0x7fefc813fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc80a0550 con 0x7fefc8136b60 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 5 monclient: authenticate success, global_id 5911 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fefb8001610 con 0x7fefc8136b60 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:16.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefc67fc640 10 monclient: dump: 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.019+0000 7fefce15b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:16.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefb80018f0 con 0x7fefc8136b60 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefc67fc640 1 -- 172.21.15.171:0/614175709 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fefb8001d00 con 0x7fefc8136b60 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 4 set_mon_vals no callback set 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefc67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefc67fc640 10 monclient: dump: 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:16.033 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 10 monclient: shutdown 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 1 -- 172.21.15.171:0/614175709 >> v1:172.21.15.171:6789/0 conn(0x7fefc8136b60 legacy=0x7fefc8136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 1 -- 172.21.15.171:0/614175709 shutdown_connections 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 1 -- 172.21.15.171:0/614175709 >> 172.21.15.171:0/614175709 conn(0x7fefc80a98b0 msgr2=0x7fefc80a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 1 -- 172.21.15.171:0/614175709 shutdown_connections 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.020+0000 7fefce15b640 1 -- 172.21.15.171:0/614175709 wait complete. 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.021+0000 7fefce15b640 10 monclient: build_initial_monmap 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.021+0000 7fefce15b640 10 monclient: monmap: 2024-09-16T04:11:16.034 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.021+0000 7fefce15b640 1 Processor -- start 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.021+0000 7fefce15b640 1 -- start start 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.021+0000 7fefce15b640 10 monclient: init 2024-09-16T04:11:16.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient(hunting): picked mon.noname-b con 0x7fefc8136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient(hunting): picked mon.noname-c con 0x7fefc8139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient(hunting): picked mon.noname-a con 0x7fefc813d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc81317e0 con 0x7fefc813d620 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc80a8980 con 0x7fefc8136b60 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc80a0240 con 0x7fefc8139b10 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient(hunting): authenticate will time out at 1051.528161s 2024-09-16T04:11:16.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefc77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fefc8136b60 0x7fefc81309c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42784/0 (socket says 172.21.15.171:42784) 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefc77fe640 1 -- 172.21.15.171:0/2087563121 learned_addr learned my addr 172.21.15.171:0/2087563121 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefc6ffd640 1 -- 172.21.15.171:0/2087563121 >> v1:172.21.15.148:6790/0 conn(0x7fefc8139b10 legacy=0x7fefc81310d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 972342227 0 0) 0x7fefc80a8980 con 0x7fefc8136b60 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): my global_id is 5914 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefb4003520 con 0x7fefc8136b60 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 741757811 0 0) 0x7fefb4003520 con 0x7fefc8136b60 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefc80a8980 con 0x7fefc8136b60 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fefb8001c40 con 0x7fefc8136b60 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:16.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): dump: 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:16.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886786635 0 0) 0x7fefc81317e0 con 0x7fefc813d620 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient(hunting): my global_id is 64353 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefb4003520 con 0x7fefc813d620 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3647357435 0 0) 0x7fefc80a8980 con 0x7fefc8136b60 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 >> v1:172.21.15.148:6790/0 conn(0x7fefc8139b10 legacy=0x7fefc81310d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 >> v1:172.21.15.148:6789/0 conn(0x7fefc813d620 legacy=0x7fefc81d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 monclient: found mon.b 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:16.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc8133520 con 0x7fefc8136b60 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: _finish_auth 0 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 5 monclient: authenticate success, global_id 5914 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient: _renew_subs 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fefc8133800 con 0x7fefc8136b60 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient: _renew_subs 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fefc813ced0 con 0x7fefc8136b60 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefb80018f0 con 0x7fefc8136b60 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fefb8004ee0 con 0x7fefc8136b60 2024-09-16T04:11:16.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefaffff640 10 monclient: dump: 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:16.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.022+0000 7fefc5ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.023+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fefb8005100 con 0x7fefc8136b60 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.023+0000 7fefce15b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.023+0000 7fefce15b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.023+0000 7fefce15b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.023+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa8005080 con 0x7fefc8136b60 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.023+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(175..175 src has 1..175) ==== 7005+0+0 (unknown 2341167999 0 0) 0x7fefb8047cf0 con 0x7fefc8136b60 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.024+0000 7fefc6ffd640 10 monclient: get_auth_request con 0x7fefb4038f70 auth_method 0 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.028+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fefb8015f00 con 0x7fefc8136b60 2024-09-16T04:11:16.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.028+0000 7fefaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:16.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.028+0000 7fefaffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.199+0000 7fefce15b640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T04:11:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.199+0000 7fefce15b640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T04:11:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.199+0000 7fefce15b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:16.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.199+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) -- 0x7fefa8005b80 con 0x7fefc8136b60 2024-09-16T04:11:16.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.985+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v176) ==== 173+0+0 (unknown 522474870 0 0) 0x7fefb80432a0 con 0x7fefc8136b60 2024-09-16T04:11:16.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.985+0000 7fefaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T04:11:16.988 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:16.985+0000 7fefaffff640 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T04:11:17.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.090+0000 7fefce15b640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T04:11:17.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.090+0000 7fefce15b640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T04:11:17.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.090+0000 7fefce15b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.090+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) -- 0x7fefa8001090 con 0x7fefc8136b60 2024-09-16T04:11:17.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.091+0000 7fefaffff640 1 -- 172.21.15.171:0/2087563121 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v176) ==== 183+0+0 (unknown 3026868651 0 0) 0x7fefb80434e0 con 0x7fefc8136b60 2024-09-16T04:11:17.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.091+0000 7fefaffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-16T04:11:17.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.091+0000 7fefaffff640 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T04:11:17.094 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-16T04:11:17.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 >> v1:172.21.15.171:6812/7327273 conn(0x7fefb4038f70 legacy=0x7fefb403b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:17.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 10 monclient: shutdown 2024-09-16T04:11:17.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:17.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 >> v1:172.21.15.171:6789/0 conn(0x7fefc8136b60 legacy=0x7fefc81309c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:17.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 shutdown_connections 2024-09-16T04:11:17.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 >> 172.21.15.171:0/2087563121 conn(0x7fefc80a98b0 msgr2=0x7fefc80a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:17.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 shutdown_connections 2024-09-16T04:11:17.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.093+0000 7fefce15b640 1 -- 172.21.15.171:0/2087563121 wait complete. 2024-09-16T04:11:17.105 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush move fooo rack=barr 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.182+0000 7fdcfd882640 10 monclient: get_monmap_and_config 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.182+0000 7fdcfd882640 10 monclient: build_initial_monmap 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.182+0000 7fdcfd882640 10 monclient: monmap: 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:17.184081+0000 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:17.184081+0000 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:17.185 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.186 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:17.186 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:17.186 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:17.186 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.182+0000 7fdcfd882640 1 Processor -- start 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.182+0000 7fdcfd882640 1 -- start start 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.182+0000 7fdcfd882640 10 monclient: init 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf8132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf809ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:17.195 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf8132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf81332b0 con 0x7fdcf8132e60 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf8134310 con 0x7fdcf809ec20 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf809d980 con 0x7fdcf8132a80 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcfd882640 10 monclient(hunting): authenticate will time out at 1052.689506s 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf67fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdcf809ec20 0x7fdcf8133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42794/0 (socket says 172.21.15.171:42794) 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf67fc640 1 -- 172.21.15.171:0/1586191830 learned_addr learned my addr 172.21.15.171:0/1586191830 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf6ffd640 1 -- 172.21.15.171:0/1586191830 >> v1:172.21.15.148:6790/0 conn(0x7fdcf8132a80 legacy=0x7fdcf81334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010263551 0 0) 0x7fdcf8134310 con 0x7fdcf809ec20 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:17.196 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): my global_id is 5917 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdce0003520 con 0x7fdcf809ec20 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385441196 0 0) 0x7fdcf81332b0 con 0x7fdcf8132e60 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 10 monclient(hunting): my global_id is 64356 2024-09-16T04:11:17.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.183+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcf8134310 con 0x7fdcf8132e60 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3089931041 0 0) 0x7fdce0003520 con 0x7fdcf809ec20 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 >> v1:172.21.15.148:6790/0 conn(0x7fdcf8132a80 legacy=0x7fdcf81334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 >> v1:172.21.15.148:6789/0 conn(0x7fdcf8132e60 legacy=0x7fdcf813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 monclient: found mon.noname-c 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf809dc60 con 0x7fdcf809ec20 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcec001990 con 0x7fdcf809ec20 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 5 monclient: authenticate success, global_id 5917 2024-09-16T04:11:17.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: dump: 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdcec001c40 con 0x7fdcf809ec20 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 1 -- 172.21.15.171:0/1586191830 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcec002010 con 0x7fdcf809ec20 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcf5ffb640 10 monclient: dump: 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:17.200 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 4 set_mon_vals no callback set 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 10 monclient: shutdown 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 1 -- 172.21.15.171:0/1586191830 >> v1:172.21.15.171:6789/0 conn(0x7fdcf809ec20 legacy=0x7fdcf8133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 1 -- 172.21.15.171:0/1586191830 shutdown_connections 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 1 -- 172.21.15.171:0/1586191830 >> 172.21.15.171:0/1586191830 conn(0x7fdcf80a52a0 msgr2=0x7fdcf80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 1 -- 172.21.15.171:0/1586191830 shutdown_connections 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.184+0000 7fdcfd882640 1 -- 172.21.15.171:0/1586191830 wait complete. 2024-09-16T04:11:17.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.185+0000 7fdcfd882640 10 monclient: build_initial_monmap 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.185+0000 7fdcfd882640 10 monclient: monmap: 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:17.202 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.185+0000 7fdcfd882640 1 Processor -- start 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.185+0000 7fdcfd882640 1 -- start start 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.185+0000 7fdcfd882640 10 monclient: init 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf809ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf8132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf8132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf811c8d0 con 0x7fdcf8132a80 2024-09-16T04:11:17.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf81c81f0 con 0x7fdcf809ec20 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf80a3d20 con 0x7fdcf8132e60 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient(hunting): authenticate will time out at 1052.692274s 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcf6ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdcf809ec20 0x7fdcf81c73d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:42796/0 (socket says 172.21.15.171:42796) 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcf6ffd640 1 -- 172.21.15.171:0/508196513 learned_addr learned my addr 172.21.15.171:0/508196513 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcf77fe640 1 -- 172.21.15.171:0/508196513 >> v1:172.21.15.148:6790/0 conn(0x7fdcf8132e60 legacy=0x7fdcf81cf630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511197347 0 0) 0x7fdcf81c81f0 con 0x7fdcf809ec20 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): my global_id is 5920 2024-09-16T04:11:17.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdce4003520 con 0x7fdcf809ec20 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2613044092 0 0) 0x7fdcf811c8d0 con 0x7fdcf8132a80 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): my global_id is 64359 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcf81c81f0 con 0x7fdcf8132a80 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4203004087 0 0) 0x7fdce4003520 con 0x7fdcf809ec20 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcf811c8d0 con 0x7fdcf809ec20 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcf00041d0 con 0x7fdcf809ec20 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): dump: 2024-09-16T04:11:17.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 965439693 0 0) 0x7fdcf81c81f0 con 0x7fdcf8132a80 2024-09-16T04:11:17.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdce4003520 con 0x7fdcf8132a80 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcec001f90 con 0x7fdcf8132a80 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient(hunting): dump: 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.207 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2635591423 0 0) 0x7fdcf811c8d0 con 0x7fdcf809ec20 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 >> v1:172.21.15.148:6790/0 conn(0x7fdcf8132e60 legacy=0x7fdcf81cf630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 >> v1:172.21.15.148:6789/0 conn(0x7fdcf8132a80 legacy=0x7fdcf81c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 monclient: found mon.b 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf809c9a0 con 0x7fdcf809ec20 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient: _finish_auth 0 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 5 monclient: authenticate success, global_id 5920 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient: _renew_subs 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.186+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf811e180 con 0x7fdcf809ec20 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcfd882640 10 monclient: _renew_subs 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcfd882640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdcf8072620 con 0x7fdcf809ec20 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdcf0004430 con 0x7fdcf809ec20 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcdffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:17.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdcf0004780 con 0x7fdcf809ec20 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcdffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcf57fa640 4 set_mon_vals no callback set 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.187+0000 7fdcdffff640 10 monclient: dump: 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:17.210 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdcf0004920 con 0x7fdcf809ec20 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcfd882640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcfd882640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcfd882640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd8005080 con 0x7fdcf809ec20 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(176..176 src has 1..176) ==== 7193+0+0 (unknown 1702981045 0 0) 0x7fdcf0047af0 con 0x7fdcf809ec20 2024-09-16T04:11:17.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.188+0000 7fdcf67fc640 10 monclient: get_auth_request con 0x7fdce4038ef0 auth_method 0 2024-09-16T04:11:17.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.192+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdcf0015be0 con 0x7fdcf809ec20 2024-09-16T04:11:17.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.192+0000 7fdcdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:17.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.192+0000 7fdcdffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:17.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.362+0000 7fdcfd882640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T04:11:17.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.362+0000 7fdcfd882640 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T04:11:17.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.362+0000 7fdcfd882640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:17.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.362+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) -- 0x7fdcd8005b80 con 0x7fdcf809ec20 2024-09-16T04:11:17.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.992+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v177) ==== 167+0+0 (unknown 374039905 0 0) 0x7fdcf00154e0 con 0x7fdcf809ec20 2024-09-16T04:11:17.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.992+0000 7fdcdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T04:11:17.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:17.992+0000 7fdcdffff640 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T04:11:18.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.096+0000 7fdcfd882640 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T04:11:18.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.096+0000 7fdcfd882640 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T04:11:18.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.096+0000 7fdcfd882640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.096+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) -- 0x7fdcd8001090 con 0x7fdcf809ec20 2024-09-16T04:11:18.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.097+0000 7fdcdffff640 1 -- 172.21.15.171:0/508196513 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v177) ==== 177+0+0 (unknown 1323730099 0 0) 0x7fdcf0019830 con 0x7fdcf809ec20 2024-09-16T04:11:18.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.097+0000 7fdcdffff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-16T04:11:18.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.097+0000 7fdcdffff640 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T04:11:18.100 INFO:tasks.workunit.client.0.smithi171.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-16T04:11:18.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.099+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 >> v1:172.21.15.171:6812/7327273 conn(0x7fdce4038ef0 legacy=0x7fdce403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:18.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.099+0000 7fdcfd882640 10 monclient: shutdown 2024-09-16T04:11:18.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.099+0000 7fdcfd882640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:18.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.099+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 >> v1:172.21.15.171:6789/0 conn(0x7fdcf809ec20 legacy=0x7fdcf81c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:18.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.099+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 shutdown_connections 2024-09-16T04:11:18.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.099+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 >> 172.21.15.171:0/508196513 conn(0x7fdcf80a52a0 msgr2=0x7fdcf80a4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:18.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.100+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 shutdown_connections 2024-09-16T04:11:18.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.100+0000 7fdcfd882640 1 -- 172.21.15.171:0/508196513 wait complete. 2024-09-16T04:11:18.111 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm fooo 2024-09-16T04:11:18.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.187+0000 7f50d7faf640 10 monclient: get_monmap_and_config 2024-09-16T04:11:18.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.187+0000 7f50d7faf640 10 monclient: build_initial_monmap 2024-09-16T04:11:18.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.187+0000 7f50d7faf640 10 monclient: monmap: 2024-09-16T04:11:18.190 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:18.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:18.189341+0000 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:18.189341+0000 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:18.191 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:18.198 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 1 Processor -- start 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 1 -- start start 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient: init 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient(hunting): picked mon.noname-c con 0x7f50d0132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient(hunting): picked mon.noname-a con 0x7f50d009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient(hunting): picked mon.noname-b con 0x7f50d0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d01332b0 con 0x7f50d009ec20 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d0134310 con 0x7f50d0132a80 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d009d980 con 0x7f50d0132e60 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:18.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d7faf640 10 monclient(hunting): authenticate will time out at 1053.694903s 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d654c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f50d0132e60 legacy=0x7f50d013c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d5d4b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f50d0132a80 0x7f50d01334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34234/0 (socket says 172.21.15.171:34234) 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.188+0000 7f50d5d4b640 1 -- 172.21.15.171:0/1177176389 learned_addr learned my addr 172.21.15.171:0/1177176389 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903365234 0 0) 0x7f50d0134310 con 0x7f50d0132a80 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): _negotiate 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): my global_id is 5923 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50bc003520 con 0x7f50d0132a80 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 282532572 0 0) 0x7f50d01332b0 con 0x7f50d009ec20 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): _negotiate 2024-09-16T04:11:18.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient(hunting): my global_id is 64362 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d0134310 con 0x7f50d009ec20 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1927357561 0 0) 0x7f50bc003520 con 0x7f50d0132a80 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 >> v1:172.21.15.148:6790/0 conn(0x7f50d0132e60 legacy=0x7f50d013c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 >> v1:172.21.15.148:6789/0 conn(0x7f50d009ec20 legacy=0x7f50d0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: _finish_hunting 0 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 monclient: found mon.noname-c 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d009dc60 con 0x7f50d0132a80 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: _finish_auth 0 2024-09-16T04:11:18.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: _check_auth_tickets 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50cc001610 con 0x7f50d0132a80 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d7faf640 5 monclient: authenticate success, global_id 5923 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d7faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: dump: 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:18.202 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d7faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50cc0018c0 con 0x7f50d0132a80 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 1 -- 172.21.15.171:0/1177176389 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50cc001c90 con 0x7f50d0132a80 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d7faf640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d7faf640 4 set_mon_vals no callback set 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:18.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.189+0000 7f50d4d49640 10 monclient: dump: 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 10 monclient: shutdown 2024-09-16T04:11:18.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 1 -- 172.21.15.171:0/1177176389 >> v1:172.21.15.171:6789/0 conn(0x7f50d0132a80 legacy=0x7f50d01334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 1 -- 172.21.15.171:0/1177176389 shutdown_connections 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 1 -- 172.21.15.171:0/1177176389 >> 172.21.15.171:0/1177176389 conn(0x7f50d00a52a0 msgr2=0x7f50d00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 1 -- 172.21.15.171:0/1177176389 shutdown_connections 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 1 -- 172.21.15.171:0/1177176389 wait complete. 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 10 monclient: build_initial_monmap 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 10 monclient: monmap: 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:18.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.190+0000 7f50d7faf640 1 Processor -- start 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 1 -- start start 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient: init 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient(hunting): picked mon.noname-b con 0x7f50d009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient(hunting): picked mon.noname-a con 0x7f50d0132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient(hunting): picked mon.noname-c con 0x7f50d0132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d0125b30 con 0x7f50d0132a80 2024-09-16T04:11:18.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d0124b90 con 0x7f50d009ec20 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d0123740 con 0x7f50d0132e60 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d7faf640 10 monclient(hunting): authenticate will time out at 1053.697561s 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d5d4b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f50d009ec20 0x7f50d01cb680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34248/0 (socket says 172.21.15.171:34248) 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d5d4b640 1 -- 172.21.15.171:0/1693052460 learned_addr learned my addr 172.21.15.171:0/1693052460 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50d654c640 1 -- 172.21.15.171:0/1693052460 >> v1:172.21.15.148:6790/0 conn(0x7f50d0132e60 legacy=0x7f50d01d4000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819530951 0 0) 0x7f50d0124b90 con 0x7f50d009ec20 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): my global_id is 5926 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50b4003520 con 0x7f50d009ec20 2024-09-16T04:11:18.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393346248 0 0) 0x7f50d0134310 con 0x7f50d0132a80 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 10 monclient(hunting): my global_id is 64365 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d0124b90 con 0x7f50d0132a80 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.191+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 982428563 0 0) 0x7f50b4003520 con 0x7f50d009ec20 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50d0134310 con 0x7f50d009ec20 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50cc001c10 con 0x7f50d009ec20 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient(hunting): dump: 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:18.208 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 512212652 0 0) 0x7f50d0124b90 con 0x7f50d0132a80 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50b4003520 con 0x7f50d0132a80 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1118731667 0 0) 0x7f50d0134310 con 0x7f50d009ec20 2024-09-16T04:11:18.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 >> v1:172.21.15.148:6790/0 conn(0x7f50d0132e60 legacy=0x7f50d01d4000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 >> v1:172.21.15.148:6789/0 conn(0x7f50d0132a80 legacy=0x7f50d01d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 monclient: found mon.b 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d01239c0 con 0x7f50d009ec20 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 5 monclient: authenticate success, global_id 5926 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 10 monclient: _renew_subs 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50d0124e70 con 0x7f50d009ec20 2024-09-16T04:11:18.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 10 monclient: _renew_subs 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50d01248b0 con 0x7f50d009ec20 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50cc0018c0 con 0x7f50d009ec20 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50cc004de0 con 0x7f50d009ec20 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c7fff640 4 set_mon_vals no callback set 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.192+0000 7f50c67fc640 10 monclient: dump: 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:18.211 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f50cc010210 con 0x7f50d009ec20 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50d7faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50d7faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50d7faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b8005080 con 0x7f50d009ec20 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(177..177 src has 1..177) ==== 7193+0+0 (unknown 146601034 0 0) 0x7f50cc047c00 con 0x7f50d009ec20 2024-09-16T04:11:18.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.193+0000 7f50d554a640 10 monclient: get_auth_request con 0x7f50b4038e90 auth_method 0 2024-09-16T04:11:18.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.197+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f50cc015db0 con 0x7f50d009ec20 2024-09-16T04:11:18.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.197+0000 7f50c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:18.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.198+0000 7f50c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:18.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.369+0000 7f50d7faf640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T04:11:18.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.369+0000 7f50d7faf640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T04:11:18.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.369+0000 7f50d7faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:18.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:18.369+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) -- 0x7f50b8005370 con 0x7f50d009ec20 2024-09-16T04:11:19.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.001+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v178) ==== 122+0+0 (unknown 1386201458 0 0) 0x7f50cc0156b0 con 0x7f50d009ec20 2024-09-16T04:11:19.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.001+0000 7f50c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T04:11:19.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.001+0000 7f50c67fc640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-16T04:11:19.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.104+0000 7f50d7faf640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T04:11:19.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.104+0000 7f50d7faf640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T04:11:19.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.104+0000 7f50d7faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.104+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) -- 0x7f50b8002210 con 0x7f50d009ec20 2024-09-16T04:11:19.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.105+0000 7f50c67fc640 1 -- 172.21.15.171:0/1693052460 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v178) ==== 122+0+0 (unknown 2051004323 0 0) 0x7f50cc01e200 con 0x7f50d009ec20 2024-09-16T04:11:19.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.105+0000 7f50c67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-16T04:11:19.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.105+0000 7f50c67fc640 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-16T04:11:19.108 INFO:tasks.workunit.client.0.smithi171.stderr:device 'fooo' does not appear in the crush map 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 >> v1:172.21.15.171:6812/7327273 conn(0x7f50b4038e90 legacy=0x7f50b403b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 10 monclient: shutdown 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 >> v1:172.21.15.171:6789/0 conn(0x7f50d009ec20 legacy=0x7f50d01cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 shutdown_connections 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 >> 172.21.15.171:0/1693052460 conn(0x7f50d00a52a0 msgr2=0x7f50d0131ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:19.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 shutdown_connections 2024-09-16T04:11:19.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.107+0000 7f50d7faf640 1 -- 172.21.15.171:0/1693052460 wait complete. 2024-09-16T04:11:19.119 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm barr 2024-09-16T04:11:19.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.194+0000 7f16924c0640 10 monclient: get_monmap_and_config 2024-09-16T04:11:19.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.194+0000 7f16924c0640 10 monclient: build_initial_monmap 2024-09-16T04:11:19.197 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.194+0000 7f16924c0640 10 monclient: monmap: 2024-09-16T04:11:19.197 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:19.197 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:19.197 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:19.196480+0000 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:19.196480+0000 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:19.198 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:19.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.195+0000 7f16924c0640 1 Processor -- start 2024-09-16T04:11:19.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.195+0000 7f16924c0640 1 -- start start 2024-09-16T04:11:19.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.195+0000 7f16924c0640 10 monclient: init 2024-09-16T04:11:19.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.195+0000 7f16924c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:19.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.195+0000 7f16924c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.195+0000 7f16924c0640 10 monclient(hunting): picked mon.noname-b con 0x7f168c132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 10 monclient(hunting): picked mon.noname-c con 0x7f168c09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 10 monclient(hunting): picked mon.noname-a con 0x7f168c132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c1332b0 con 0x7f168c132e60 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c134310 con 0x7f168c09ec20 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c09d980 con 0x7f168c132a80 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 10 monclient(hunting): authenticate will time out at 1054.702062s 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f168c09ec20 0x7f168c133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34250/0 (socket says 172.21.15.171:34250) 2024-09-16T04:11:19.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f168c132a80 legacy=0x7f168c1334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168b7fe640 1 -- 172.21.15.171:0/1586366704 learned_addr learned my addr 172.21.15.171:0/1586366704 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3407140277 0 0) 0x7f168c134310 con 0x7f168c09ec20 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient(hunting): my global_id is 5929 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1674003520 con 0x7f168c09ec20 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2849880191 0 0) 0x7f1674003520 con 0x7f168c09ec20 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 >> v1:172.21.15.148:6790/0 conn(0x7f168c132a80 legacy=0x7f168c1334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 >> v1:172.21.15.148:6789/0 conn(0x7f168c132e60 legacy=0x7f168c13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:19.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 monclient: found mon.noname-c 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168c09dc60 con 0x7f168c09ec20 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1680001610 con 0x7f168c09ec20 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 5 monclient: authenticate success, global_id 5929 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:19.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: dump: 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:19.209 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1680001880 con 0x7f168c09ec20 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 1 -- 172.21.15.171:0/1586366704 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1680001c50 con 0x7f168c09ec20 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f16924c0640 4 set_mon_vals no callback set 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.196+0000 7f168affd640 10 monclient: dump: 2024-09-16T04:11:19.210 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 10 monclient: shutdown 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:19.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 1 -- 172.21.15.171:0/1586366704 >> v1:172.21.15.171:6789/0 conn(0x7f168c09ec20 legacy=0x7f168c133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 1 -- 172.21.15.171:0/1586366704 shutdown_connections 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 1 -- 172.21.15.171:0/1586366704 >> 172.21.15.171:0/1586366704 conn(0x7f168c0a52a0 msgr2=0x7f168c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 1 -- 172.21.15.171:0/1586366704 shutdown_connections 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 1 -- 172.21.15.171:0/1586366704 wait complete. 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 10 monclient: build_initial_monmap 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.197+0000 7f16924c0640 10 monclient: monmap: 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:19.212 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 1 Processor -- start 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 1 -- start start 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient: init 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:19.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient(hunting): picked mon.noname-c con 0x7f168c09ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient(hunting): picked mon.noname-a con 0x7f168c132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient(hunting): picked mon.noname-b con 0x7f168c132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c1c7ed0 con 0x7f168c132a80 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c120140 con 0x7f168c132e60 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f168c120800 con 0x7f168c09ec20 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f16924c0640 10 monclient(hunting): authenticate will time out at 1054.704772s 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f168bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f168c09ec20 legacy=0x7f168c1c70b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f1690a5d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f168c132e60 0x7f168c1cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34254/0 (socket says 172.21.15.171:34254) 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f1690a5d640 1 -- 172.21.15.171:0/442715902 learned_addr learned my addr 172.21.15.171:0/442715902 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:19.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.198+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228132988 0 0) 0x7f168c120140 con 0x7f168c132e60 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): my global_id is 5932 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1678003520 con 0x7f168c132e60 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3074180128 0 0) 0x7f168c1c7ed0 con 0x7f168c132a80 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:19.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): my global_id is 64371 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f168c120140 con 0x7f168c132a80 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 504338522 0 0) 0x7f1678003520 con 0x7f168c132e60 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f168c1c7ed0 con 0x7f168c132e60 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1684001cf0 con 0x7f168c132e60 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): dump: 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:19.216 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1660101233 0 0) 0x7f168c1c7ed0 con 0x7f168c132e60 2024-09-16T04:11:19.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 >> v1:172.21.15.148:6790/0 conn(0x7f168c09ec20 legacy=0x7f168c1c70b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 >> v1:172.21.15.148:6789/0 conn(0x7f168c132a80 legacy=0x7f168c1c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 monclient: found mon.b 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168c131a30 con 0x7f168c132e60 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: _finish_auth 0 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:11:19.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 5 monclient: authenticate success, global_id 5932 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 10 monclient: _renew_subs 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f168c0a3420 con 0x7f168c132e60 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 10 monclient: _renew_subs 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f168c1216f0 con 0x7f168c132e60 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1684003c40 con 0x7f168c132e60 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1684003e70 con 0x7f168c132e60 2024-09-16T04:11:19.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f168a7fc640 4 set_mon_vals no callback set 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.199+0000 7f1688ff9640 10 monclient: dump: 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:19.220 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1684004010 con 0x7f168c132e60 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f16924c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f16924c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f16924c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c005080 con 0x7f168c132e60 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(178..178 src has 1..178) ==== 6977+0+0 (unknown 1008287097 0 0) 0x7f1684047e40 con 0x7f168c132e60 2024-09-16T04:11:19.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.200+0000 7f168bfff640 10 monclient: get_auth_request con 0x7f1678038bb0 auth_method 0 2024-09-16T04:11:19.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.205+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1684016080 con 0x7f168c132e60 2024-09-16T04:11:19.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.205+0000 7f1688ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:19.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.205+0000 7f1688ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:19.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.376+0000 7f16924c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T04:11:19.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.376+0000 7f16924c0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T04:11:19.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.376+0000 7f16924c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:19.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:19.376+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) -- 0x7f166c005370 con 0x7f168c132e60 2024-09-16T04:11:20.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.010+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v179) ==== 122+0+0 (unknown 1908779493 0 0) 0x7f1684015980 con 0x7f168c132e60 2024-09-16T04:11:20.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.010+0000 7f1688ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T04:11:20.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.010+0000 7f1688ff9640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-16T04:11:20.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.118+0000 7f16924c0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T04:11:20.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.118+0000 7f16924c0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T04:11:20.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.118+0000 7f16924c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.118+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) -- 0x7f166c002210 con 0x7f168c132e60 2024-09-16T04:11:20.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.119+0000 7f1688ff9640 1 -- 172.21.15.171:0/442715902 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v179) ==== 122+0+0 (unknown 974834810 0 0) 0x7f168401e530 con 0x7f168c132e60 2024-09-16T04:11:20.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.119+0000 7f1688ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-16T04:11:20.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.119+0000 7f1688ff9640 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-16T04:11:20.122 INFO:tasks.workunit.client.0.smithi171.stderr:device 'barr' does not appear in the crush map 2024-09-16T04:11:20.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 >> v1:172.21.15.171:6812/7327273 conn(0x7f1678038bb0 legacy=0x7f167803b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:20.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 10 monclient: shutdown 2024-09-16T04:11:20.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:20.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 >> v1:172.21.15.171:6789/0 conn(0x7f168c132e60 legacy=0x7f168c1cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:20.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 shutdown_connections 2024-09-16T04:11:20.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 >> 172.21.15.171:0/442715902 conn(0x7f168c0a52a0 msgr2=0x7f168c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:20.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 shutdown_connections 2024-09-16T04:11:20.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.121+0000 7f16924c0640 1 -- 172.21.15.171:0/442715902 wait complete. 2024-09-16T04:11:20.134 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.210+0000 7f80d7511640 10 monclient: get_monmap_and_config 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.210+0000 7f80d7511640 10 monclient: build_initial_monmap 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.210+0000 7f80d7511640 10 monclient: monmap: 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:20.212047+0000 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:20.212047+0000 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:20.213 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:20.214 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:20.214 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:20.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.210+0000 7f80d7511640 1 Processor -- start 2024-09-16T04:11:20.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 1 -- start start 2024-09-16T04:11:20.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient: init 2024-09-16T04:11:20.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient(hunting): picked mon.noname-b con 0x7f80d0136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient(hunting): picked mon.noname-c con 0x7f80d0139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient(hunting): picked mon.noname-a con 0x7f80d013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d00a1340 con 0x7f80d013d620 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d0137650 con 0x7f80d0139b10 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d00a0aa0 con 0x7f80d0136b60 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d7511640 10 monclient(hunting): authenticate will time out at 1055.717557s 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d4aac640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f80d0139b10 0x7f80d013bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34256/0 (socket says 172.21.15.171:34256) 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d4aac640 1 -- 172.21.15.171:0/2963045031 learned_addr learned my addr 172.21.15.171:0/2963045031 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80d52ad640 1 -- 172.21.15.171:0/2963045031 >> v1:172.21.15.148:6790/0 conn(0x7f80d0136b60 legacy=0x7f80d0136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:20.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218589001 0 0) 0x7f80d0137650 con 0x7f80d0139b10 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80c7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80c7fff640 10 monclient(hunting): my global_id is 5935 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.211+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80bc003520 con 0x7f80d0139b10 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4293830169 0 0) 0x7f80bc003520 con 0x7f80d0139b10 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 >> v1:172.21.15.148:6790/0 conn(0x7f80d0136b60 legacy=0x7f80d0136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 >> v1:172.21.15.148:6789/0 conn(0x7f80d013d620 legacy=0x7f80d013fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 monclient: found mon.noname-c 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d00a0550 con 0x7f80d0139b10 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f80c0001610 con 0x7f80d0139b10 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: dump: 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:20.223 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 5 monclient: authenticate success, global_id 5935 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f80c0001970 con 0x7f80d0139b10 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 1 -- 172.21.15.171:0/2963045031 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f80c00048a0 con 0x7f80d0139b10 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 4 set_mon_vals no callback set 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80c7fff640 10 monclient: dump: 2024-09-16T04:11:20.224 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 10 monclient: shutdown 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 1 -- 172.21.15.171:0/2963045031 >> v1:172.21.15.171:6789/0 conn(0x7f80d0139b10 legacy=0x7f80d013bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 1 -- 172.21.15.171:0/2963045031 shutdown_connections 2024-09-16T04:11:20.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 1 -- 172.21.15.171:0/2963045031 >> 172.21.15.171:0/2963045031 conn(0x7f80d00a98b0 msgr2=0x7f80d00a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 1 -- 172.21.15.171:0/2963045031 shutdown_connections 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.212+0000 7f80d7511640 1 -- 172.21.15.171:0/2963045031 wait complete. 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.213+0000 7f80d7511640 10 monclient: build_initial_monmap 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.213+0000 7f80d7511640 10 monclient: monmap: 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:20.226 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.213+0000 7f80d7511640 1 Processor -- start 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.213+0000 7f80d7511640 1 -- start start 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.213+0000 7f80d7511640 10 monclient: init 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient(hunting): picked mon.noname-b con 0x7f80d0136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient(hunting): picked mon.noname-c con 0x7f80d0139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient(hunting): picked mon.noname-a con 0x7f80d013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d009acc0 con 0x7f80d013d620 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d013ad10 con 0x7f80d0136b60 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d013bb30 con 0x7f80d0139b10 2024-09-16T04:11:20.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient(hunting): authenticate will time out at 1055.720284s 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d52ad640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f80d0136b60 0x7f80d0139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34268/0 (socket says 172.21.15.171:34268) 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d52ad640 1 -- 172.21.15.171:0/361853789 learned_addr learned my addr 172.21.15.171:0/361853789 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d4aac640 1 -- 172.21.15.171:0/361853789 >> v1:172.21.15.148:6790/0 conn(0x7f80d0139b10 legacy=0x7f80d013a600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938240050 0 0) 0x7f80d013ad10 con 0x7f80d0136b60 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): my global_id is 5938 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80b4003520 con 0x7f80d0136b60 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 54475969 0 0) 0x7f80d009acc0 con 0x7f80d013d620 2024-09-16T04:11:20.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): my global_id is 64377 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80d013ad10 con 0x7f80d013d620 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3049188912 0 0) 0x7f80b4003520 con 0x7f80d0136b60 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f80d009acc0 con 0x7f80d0136b60 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f80cc004180 con 0x7f80d0136b60 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): dump: 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.229 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 207219997 0 0) 0x7f80d009acc0 con 0x7f80d0136b60 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 >> v1:172.21.15.148:6790/0 conn(0x7f80d0139b10 legacy=0x7f80d013a600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 >> v1:172.21.15.148:6789/0 conn(0x7f80d013d620 legacy=0x7f80d01d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 monclient: found mon.b 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:20.230 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d00a20d0 con 0x7f80d0136b60 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 5 monclient: authenticate success, global_id 5938 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient: _renew_subs 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.214+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f80d00a0240 con 0x7f80d0136b60 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f80cc0043e0 con 0x7f80d0136b60 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f80cc0047b0 con 0x7f80d0136b60 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:20.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 10 monclient: dump: 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c77fe640 4 set_mon_vals no callback set 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80d7511640 10 monclient: _renew_subs 2024-09-16T04:11:20.232 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80d7511640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f80d013b8b0 con 0x7f80d0136b60 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.215+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f80cc0049d0 con 0x7f80d0136b60 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.216+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(179..179 src has 1..179) ==== 6761+0+0 (unknown 4116546022 0 0) 0x7f80cc0478b0 con 0x7f80d0136b60 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.216+0000 7f80d7511640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.216+0000 7f80d7511640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.216+0000 7f80d7511640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.216+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d00008d0 con 0x7f80d0136b60 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.216+0000 7f80d4aac640 10 monclient: get_auth_request con 0x7f80b4038c10 auth_method 0 2024-09-16T04:11:20.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.220+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f80cc015c90 con 0x7f80d0136b60 2024-09-16T04:11:20.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.220+0000 7f80c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:20.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.220+0000 7f80c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:20.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.393+0000 7f80d7511640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:20.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.393+0000 7f80d7511640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:20.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.393+0000 7f80d7511640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:20.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:20.393+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f80d013cf40 con 0x7f80d0136b60 2024-09-16T04:11:21.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.017+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v180) ==== 78+0+0 (unknown 3538827938 0 0) 0x7f80cc015590 con 0x7f80d0136b60 2024-09-16T04:11:21.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.017+0000 7f80c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:21.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.017+0000 7f80c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:21.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.127+0000 7f80d7511640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:21.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.127+0000 7f80d7511640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:21.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.127+0000 7f80d7511640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:21.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:21.127+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7f80d00a4ca0 con 0x7f80d0136b60 2024-09-16T04:11:22.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.022+0000 7f80c5ffb640 1 -- 172.21.15.171:0/361853789 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v181) ==== 78+0+0 (unknown 2469359751 0 0) 0x7f80cc01e140 con 0x7f80d0136b60 2024-09-16T04:11:22.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.022+0000 7f80c5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:11:22.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.022+0000 7f80c5ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 >> v1:172.21.15.171:6812/7327273 conn(0x7f80b4038c10 legacy=0x7f80b403b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 10 monclient: shutdown 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 >> v1:172.21.15.171:6789/0 conn(0x7f80d0136b60 legacy=0x7f80d0139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 shutdown_connections 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 >> 172.21.15.171:0/361853789 conn(0x7f80d00a98b0 msgr2=0x7f80d0072c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 shutdown_connections 2024-09-16T04:11:22.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.025+0000 7f80d7511640 1 -- 172.21.15.171:0/361853789 wait complete. 2024-09-16T04:11:22.040 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set create-compat 2024-09-16T04:11:22.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient: get_monmap_and_config 2024-09-16T04:11:22.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient: build_initial_monmap 2024-09-16T04:11:22.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient: monmap: 2024-09-16T04:11:22.124 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:22.124 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:22.123463+0000 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:22.123463+0000 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:22.125 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:22.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 1 Processor -- start 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 1 -- start start 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient: init 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d81338e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d809aa20 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d813d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d81343d0 con 0x7fc3d809aa20 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d8075db0 con 0x7fc3d81338e0 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d809b1e0 con 0x7fc3d813d240 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.122+0000 7fc3dce53640 10 monclient(hunting): authenticate will time out at 1057.628989s 2024-09-16T04:11:22.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d6d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc3d813d240 legacy=0x7fc3d813f630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc3d81338e0 0x7fc3d8133cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34282/0 (socket says 172.21.15.171:34282) 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d659c640 1 -- 172.21.15.171:0/1670120349 learned_addr learned my addr 172.21.15.171:0/1670120349 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214357136 0 0) 0x7fc3d8075db0 con 0x7fc3d81338e0 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): my global_id is 5941 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c0003520 con 0x7fc3d81338e0 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123590352 0 0) 0x7fc3d81343d0 con 0x7fc3d809aa20 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:22.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient(hunting): my global_id is 64380 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d8075db0 con 0x7fc3d809aa20 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2523322390 0 0) 0x7fc3c0003520 con 0x7fc3d81338e0 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 >> v1:172.21.15.148:6790/0 conn(0x7fc3d813d240 legacy=0x7fc3d813f630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 >> v1:172.21.15.148:6789/0 conn(0x7fc3d809aa20 legacy=0x7fc3d813bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 monclient: found mon.noname-c 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d809b4c0 con 0x7fc3d81338e0 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: _finish_auth 0 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c8001610 con 0x7fc3d81338e0 2024-09-16T04:11:22.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3dce53640 5 monclient: authenticate success, global_id 5941 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3dce53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: dump: 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:22.136 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3dce53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3c80018c0 con 0x7fc3d81338e0 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 1 -- 172.21.15.171:0/1670120349 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c8001c90 con 0x7fc3d81338e0 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3dce53640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3dce53640 4 set_mon_vals no callback set 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.123+0000 7fc3d559a640 10 monclient: dump: 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:22.137 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 10 monclient: shutdown 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 1 -- 172.21.15.171:0/1670120349 >> v1:172.21.15.171:6789/0 conn(0x7fc3d81338e0 legacy=0x7fc3d8133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 1 -- 172.21.15.171:0/1670120349 shutdown_connections 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 1 -- 172.21.15.171:0/1670120349 >> 172.21.15.171:0/1670120349 conn(0x7fc3d80a28d0 msgr2=0x7fc3d80a2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:22.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 1 -- 172.21.15.171:0/1670120349 shutdown_connections 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.124+0000 7fc3dce53640 1 -- 172.21.15.171:0/1670120349 wait complete. 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient: build_initial_monmap 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient: monmap: 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:22.139 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 1 Processor -- start 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 1 -- start start 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient: init 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d809aa20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d81338e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d813d240 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d813c700 con 0x7fc3d81338e0 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d809d4d0 con 0x7fc3d809aa20 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d80a5a80 con 0x7fc3d813d240 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:22.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.125+0000 7fc3dce53640 10 monclient(hunting): authenticate will time out at 1057.631971s 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3d6d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc3d813d240 legacy=0x7fc3d81d3da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3d659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc3d809aa20 0x7fc3d8132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34296/0 (socket says 172.21.15.171:34296) 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3d659c640 1 -- 172.21.15.171:0/3043077622 learned_addr learned my addr 172.21.15.171:0/3043077622 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4124301780 0 0) 0x7fc3d809b4c0 con 0x7fc3d809aa20 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): my global_id is 5944 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b8003520 con 0x7fc3d809aa20 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232443734 0 0) 0x7fc3d813c700 con 0x7fc3d81338e0 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:22.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): my global_id is 64383 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d809b4c0 con 0x7fc3d81338e0 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2724636536 0 0) 0x7fc3b8003520 con 0x7fc3d809aa20 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3d813c700 con 0x7fc3d809aa20 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c8001c10 con 0x7fc3d809aa20 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:22.142 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1202720070 0 0) 0x7fc3d813c700 con 0x7fc3d809aa20 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 >> v1:172.21.15.148:6790/0 conn(0x7fc3d813d240 legacy=0x7fc3d81d3da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 >> v1:172.21.15.148:6789/0 conn(0x7fc3d81338e0 legacy=0x7fc3d81d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 monclient: found mon.b 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:22.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d809d0c0 con 0x7fc3d809aa20 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 5 monclient: authenticate success, global_id 5944 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 10 monclient: _renew_subs 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d811da10 con 0x7fc3d809aa20 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 10 monclient: _renew_subs 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3d8132e20 con 0x7fc3d809aa20 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3c80018c0 con 0x7fc3d809aa20 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:22.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c8004de0 con 0x7fc3d809aa20 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3d4d99640 4 set_mon_vals no callback set 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.126+0000 7fc3c6ffd640 10 monclient: dump: 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:22.145 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc3c8004f80 con 0x7fc3d809aa20 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3dce53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3dce53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3dce53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3bc005080 con 0x7fc3d809aa20 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(181..181 src has 1..181) ==== 6433+0+0 (unknown 2217704116 0 0) 0x7fc3c8047a10 con 0x7fc3d809aa20 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.127+0000 7fc3d5d9b640 10 monclient: get_auth_request con 0x7fc3b8038b90 auth_method 0 2024-09-16T04:11:22.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.132+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc3d809d4d0 con 0x7fc3d809aa20 2024-09-16T04:11:22.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.132+0000 7fc3c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:22.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.132+0000 7fc3c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:22.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.306+0000 7fc3dce53640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:22.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.306+0000 7fc3dce53640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:22.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.306+0000 7fc3dce53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:22.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:22.306+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fc3bc005370 con 0x7fc3d809aa20 2024-09-16T04:11:23.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.023+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v182) ==== 82+0+0 (unknown 328874346 0 0) 0x7fc3c8015e50 con 0x7fc3d809aa20 2024-09-16T04:11:23.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.023+0000 7fc3c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:23.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.023+0000 7fc3c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:23.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.134+0000 7fc3dce53640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:23.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.134+0000 7fc3dce53640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:23.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.134+0000 7fc3dce53640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.134+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) -- 0x7fc3bc002210 con 0x7fc3d809aa20 2024-09-16T04:11:23.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.135+0000 7fc3c6ffd640 1 -- 172.21.15.171:0/3043077622 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v182) ==== 115+0+0 (unknown 2606304275 0 0) 0x7fc3c8015750 con 0x7fc3d809aa20 2024-09-16T04:11:23.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.135+0000 7fc3c6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-16T04:11:23.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.135+0000 7fc3c6ffd640 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-16T04:11:23.138 INFO:tasks.workunit.client.0.smithi171.stderr:compat weight-set already created 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 >> v1:172.21.15.171:6812/7327273 conn(0x7fc3b8038b90 legacy=0x7fc3b803b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 10 monclient: shutdown 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 >> v1:172.21.15.171:6789/0 conn(0x7fc3d809aa20 legacy=0x7fc3d8132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 shutdown_connections 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 >> 172.21.15.171:0/3043077622 conn(0x7fc3d80a28d0 msgr2=0x7fc3d813ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 shutdown_connections 2024-09-16T04:11:23.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.137+0000 7fc3dce53640 1 -- 172.21.15.171:0/3043077622 wait complete. 2024-09-16T04:11:23.149 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-16T04:11:23.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.229+0000 7f5c28c65640 10 monclient: get_monmap_and_config 2024-09-16T04:11:23.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.229+0000 7f5c28c65640 10 monclient: build_initial_monmap 2024-09-16T04:11:23.231 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.229+0000 7f5c28c65640 10 monclient: monmap: 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:23.230599+0000 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:23.230599+0000 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:23.232 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.229+0000 7f5c28c65640 1 Processor -- start 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.229+0000 7f5c28c65640 1 -- start start 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.229+0000 7f5c28c65640 10 monclient: init 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient(hunting): picked mon.noname-b con 0x7f5c24136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient(hunting): picked mon.noname-a con 0x7f5c24137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient(hunting): picked mon.noname-c con 0x7f5c241375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c2409b2c0 con 0x7f5c24137140 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c2409b5a0 con 0x7f5c241375a0 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c240a6640 con 0x7f5c24136d60 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:23.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient(hunting): authenticate will time out at 1058.736125s 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c22d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5c241375a0 0x7f5c2413f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34300/0 (socket says 172.21.15.171:34300) 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c22d9d640 1 -- 172.21.15.171:0/927465604 learned_addr learned my addr 172.21.15.171:0/927465604 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2259c640 1 -- 172.21.15.171:0/927465604 >> v1:172.21.15.148:6790/0 conn(0x7f5c24136d60 legacy=0x7f5c2409bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153075123 0 0) 0x7f5c2409b5a0 con 0x7f5c241375a0 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient(hunting): my global_id is 5947 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c10003520 con 0x7f5c241375a0 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 381712084 0 0) 0x7f5c10003520 con 0x7f5c241375a0 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 >> v1:172.21.15.148:6790/0 conn(0x7f5c24136d60 legacy=0x7f5c2409bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 >> v1:172.21.15.148:6789/0 conn(0x7f5c24137140 legacy=0x7f5c2413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:23.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 monclient: found mon.noname-c 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c24143b20 con 0x7f5c241375a0 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: _finish_auth 0 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 5 monclient: authenticate success, global_id 5947 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c1c001610 con 0x7f5c241375a0 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: dump: 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:23.243 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c1c001900 con 0x7f5c241375a0 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 1 -- 172.21.15.171:0/927465604 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c1c001d10 con 0x7f5c241375a0 2024-09-16T04:11:23.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c28c65640 4 set_mon_vals no callback set 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.230+0000 7f5c2159a640 10 monclient: dump: 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:23.245 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 10 monclient: shutdown 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 1 -- 172.21.15.171:0/927465604 >> v1:172.21.15.171:6789/0 conn(0x7f5c241375a0 legacy=0x7f5c2413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 1 -- 172.21.15.171:0/927465604 shutdown_connections 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 1 -- 172.21.15.171:0/927465604 >> 172.21.15.171:0/927465604 conn(0x7f5c240a9ae0 msgr2=0x7f5c240a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 1 -- 172.21.15.171:0/927465604 shutdown_connections 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.231+0000 7f5c28c65640 1 -- 172.21.15.171:0/927465604 wait complete. 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient: build_initial_monmap 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient: monmap: 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:23.246 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 1 Processor -- start 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 1 -- start start 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient: init 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient(hunting): picked mon.noname-c con 0x7f5c24136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient(hunting): picked mon.noname-b con 0x7f5c24137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:23.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 10 monclient(hunting): picked mon.noname-a con 0x7f5c241375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c24135b20 con 0x7f5c241375a0 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c2259c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f5c24136d60 legacy=0x7f5c2412b090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.232+0000 7f5c28c65640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c240a8880 con 0x7f5c24137140 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c240a8350 con 0x7f5c24136d60 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 10 monclient(hunting): authenticate will time out at 1058.739022s 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c21d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5c24137140 0x7f5c2412b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34308/0 (socket says 172.21.15.171:34308) 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c21d9b640 1 -- 172.21.15.171:0/4250252127 learned_addr learned my addr 172.21.15.171:0/4250252127 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1687859548 0 0) 0x7f5c2409b2c0 con 0x7f5c24137140 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): my global_id is 5950 2024-09-16T04:11:23.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c04003520 con 0x7f5c24137140 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092882808 0 0) 0x7f5c24135b20 con 0x7f5c241375a0 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): my global_id is 64389 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c2409b2c0 con 0x7f5c241375a0 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3565080970 0 0) 0x7f5c04003520 con 0x7f5c24137140 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c24135b20 con 0x7f5c24137140 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c0c001930 con 0x7f5c24137140 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:23.249 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): dump: 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3350189557 0 0) 0x7f5c2409b2c0 con 0x7f5c241375a0 2024-09-16T04:11:23.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5c04003520 con 0x7f5c241375a0 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c1c001430 con 0x7f5c241375a0 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient(hunting): dump: 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:23.251 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3804850405 0 0) 0x7f5c24135b20 con 0x7f5c24137140 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 >> v1:172.21.15.148:6790/0 conn(0x7f5c24136d60 legacy=0x7f5c2412b090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 >> v1:172.21.15.148:6789/0 conn(0x7f5c241375a0 legacy=0x7f5c241d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 monclient: found mon.b 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2412c8e0 con 0x7f5c24137140 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 5 monclient: authenticate success, global_id 5950 2024-09-16T04:11:23.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 10 monclient: _renew_subs 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2412e8d0 con 0x7f5c24137140 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 10 monclient: _renew_subs 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5c2412ee90 con 0x7f5c24137140 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5c0c001d00 con 0x7f5c24137140 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5c0c003a60 con 0x7f5c24137140 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c20d99640 4 set_mon_vals no callback set 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.233+0000 7f5c1affd640 10 monclient: dump: 2024-09-16T04:11:23.253 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5c0c003c80 con 0x7f5c24137140 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c28c65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c28c65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:23.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c28c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c08005080 con 0x7f5c24137140 2024-09-16T04:11:23.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(182..182 src has 1..182) ==== 6761+0+0 (unknown 1612554811 0 0) 0x7f5c0c047a50 con 0x7f5c24137140 2024-09-16T04:11:23.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.234+0000 7f5c2259c640 10 monclient: get_auth_request con 0x7f5c04038ea0 auth_method 0 2024-09-16T04:11:23.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.239+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5c240a8880 con 0x7f5c24137140 2024-09-16T04:11:23.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.239+0000 7f5c1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:23.255 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.239+0000 7f5c1affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:23.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.408+0000 7f5c28c65640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T04:11:23.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.408+0000 7f5c28c65640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T04:11:23.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.408+0000 7f5c28c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:23.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:23.408+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) -- 0x7f5c08005370 con 0x7f5c24137140 2024-09-16T04:11:24.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.040+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v183) ==== 176+0+0 (unknown 2894638766 0 0) 0x7f5c0c015d10 con 0x7f5c24137140 2024-09-16T04:11:24.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.040+0000 7f5c1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T04:11:24.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.040+0000 7f5c1affd640 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-16T04:11:24.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.144+0000 7f5c28c65640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T04:11:24.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.144+0000 7f5c28c65640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T04:11:24.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.144+0000 7f5c28c65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.144+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) -- 0x7f5c08001170 con 0x7f5c24137140 2024-09-16T04:11:24.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.145+0000 7f5c1affd640 1 -- 172.21.15.171:0/4250252127 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v183) ==== 150+0+0 (unknown 3574563992 0 0) 0x7f5c08001170 con 0x7f5c24137140 2024-09-16T04:11:24.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.145+0000 7f5c1affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-16T04:11:24.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.145+0000 7f5c1affd640 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-16T04:11:24.147 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'r1' already exists 2024-09-16T04:11:24.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 >> v1:172.21.15.171:6812/7327273 conn(0x7f5c04038ea0 legacy=0x7f5c0403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:24.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 10 monclient: shutdown 2024-09-16T04:11:24.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:24.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 >> v1:172.21.15.171:6789/0 conn(0x7f5c24137140 legacy=0x7f5c2412b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:24.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 shutdown_connections 2024-09-16T04:11:24.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 >> 172.21.15.171:0/4250252127 conn(0x7f5c240a9ae0 msgr2=0x7f5c240a25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:24.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 shutdown_connections 2024-09-16T04:11:24.150 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.147+0000 7f5c28c65640 1 -- 172.21.15.171:0/4250252127 wait complete. 2024-09-16T04:11:24.158 INFO:tasks.workunit.client.0.smithi171.stderr:++ seq 1 32 2024-09-16T04:11:24.173 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:24.174 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-16T04:11:24.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.254+0000 7fb7ad34b640 10 monclient: get_monmap_and_config 2024-09-16T04:11:24.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.254+0000 7fb7ad34b640 10 monclient: build_initial_monmap 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.254+0000 7fb7ad34b640 10 monclient: monmap: 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:24.255440+0000 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:24.255440+0000 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:24.257 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:24.258 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 1 Processor -- start 2024-09-16T04:11:24.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 1 -- start start 2024-09-16T04:11:24.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient: init 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7a80a3b50 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7a80a3f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7a813cec0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a80a4cf0 con 0x7fb7a80a3f30 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a809aa90 con 0x7fb7a813cec0 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a809af70 con 0x7fb7a80a3b50 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:24.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.255+0000 7fb7ad34b640 10 monclient(hunting): authenticate will time out at 1059.760985s 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a6ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb7a80a3b50 legacy=0x7fb7a809bc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb7a813cec0 0x7fb7a813f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34312/0 (socket says 172.21.15.171:34312) 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a77fe640 1 -- 172.21.15.171:0/3432160609 learned_addr learned my addr 172.21.15.171:0/3432160609 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2837447101 0 0) 0x7fb7a809aa90 con 0x7fb7a813cec0 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient(hunting): my global_id is 5953 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb794003520 con 0x7fb7a813cec0 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2211324344 0 0) 0x7fb794003520 con 0x7fb7a813cec0 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 >> v1:172.21.15.148:6790/0 conn(0x7fb7a80a3b50 legacy=0x7fb7a809bc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 >> v1:172.21.15.148:6789/0 conn(0x7fb7a80a3f30 legacy=0x7fb7a80a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:24.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a809b250 con 0x7fb7a813cec0 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7a0001990 con 0x7fb7a813cec0 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7ad34b640 5 monclient: authenticate success, global_id 5953 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7ad34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:24.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: dump: 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7ad34b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7a0001c80 con 0x7fb7a813cec0 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:24.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7ad34b640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 1 -- 172.21.15.171:0/3432160609 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7a0002090 con 0x7fb7a813cec0 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7ad34b640 4 set_mon_vals no callback set 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.256+0000 7fb7a67fc640 10 monclient: dump: 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:24.270 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 10 monclient: shutdown 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 1 -- 172.21.15.171:0/3432160609 >> v1:172.21.15.171:6789/0 conn(0x7fb7a813cec0 legacy=0x7fb7a813f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 1 -- 172.21.15.171:0/3432160609 shutdown_connections 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 1 -- 172.21.15.171:0/3432160609 >> 172.21.15.171:0/3432160609 conn(0x7fb7a80a9940 msgr2=0x7fb7a80a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 1 -- 172.21.15.171:0/3432160609 shutdown_connections 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 1 -- 172.21.15.171:0/3432160609 wait complete. 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 10 monclient: build_initial_monmap 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.257+0000 7fb7ad34b640 10 monclient: monmap: 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:24.271 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 1 Processor -- start 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 1 -- start start 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient: init 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient(hunting): picked mon.noname-b con 0x7fb7a80a3b50 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient(hunting): picked mon.noname-a con 0x7fb7a80a3f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:24.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient(hunting): picked mon.noname-c con 0x7fb7a813cec0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a813cbc0 con 0x7fb7a80a3f30 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a813bd80 con 0x7fb7a80a3b50 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a80a0770 con 0x7fb7a813cec0 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7ad34b640 10 monclient(hunting): authenticate will time out at 1059.763627s 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7a6ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb7a80a3b50 0x7fb7a8138790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34326/0 (socket says 172.21.15.171:34326) 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7a77fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb7a813cec0 legacy=0x7fb7a81395b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb7a6ffd640 1 -- 172.21.15.171:0/975959646 learned_addr learned my addr 172.21.15.171:0/975959646 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876075547 0 0) 0x7fb7a813bd80 con 0x7fb7a80a3b50 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:24.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): my global_id is 5956 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb788003520 con 0x7fb7a80a3b50 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353400335 0 0) 0x7fb7a80a4cf0 con 0x7fb7a80a3f30 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 10 monclient(hunting): my global_id is 64395 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.258+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7a813bd80 con 0x7fb7a80a3f30 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2459534195 0 0) 0x7fb788003520 con 0x7fb7a80a3b50 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7a80a4cf0 con 0x7fb7a80a3b50 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb7980041d0 con 0x7fb7a80a3b50 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:24.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:24.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1500827974 0 0) 0x7fb7a813bd80 con 0x7fb7a80a3f30 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb788003520 con 0x7fb7a80a3f30 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb790001830 con 0x7fb7a80a3f30 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.276 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 627445287 0 0) 0x7fb7a80a4cf0 con 0x7fb7a80a3b50 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 >> v1:172.21.15.148:6790/0 conn(0x7fb7a813cec0 legacy=0x7fb7a81395b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 >> v1:172.21.15.148:6789/0 conn(0x7fb7a80a3f30 legacy=0x7fb7a8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 monclient: found mon.b 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a80a8490 con 0x7fb7a80a3b50 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:24.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 5 monclient: authenticate success, global_id 5956 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 10 monclient: _renew_subs 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a80a0340 con 0x7fb7a80a3b50 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 10 monclient: _renew_subs 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7a813c040 con 0x7fb7a80a3b50 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb798004430 con 0x7fb7a80a3b50 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb798004780 con 0x7fb7a80a3b50 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb7a5ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:24.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.259+0000 7fb79f7fe640 10 monclient: dump: 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb798004920 con 0x7fb7a80a3b50 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb7ad34b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:24.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb7ad34b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb7ad34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb78c005080 con 0x7fb7a80a3b50 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(183..183 src has 1..183) ==== 6971+0+0 (unknown 3506316678 0 0) 0x7fb798047a50 con 0x7fb7a80a3b50 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.260+0000 7fb79ffff640 10 monclient: get_auth_request con 0x7fb788038ef0 auth_method 0 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.265+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb798015bc0 con 0x7fb7a80a3b50 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.265+0000 7fb79f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:24.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.265+0000 7fb79f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:24.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.434+0000 7fb7ad34b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:24.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.434+0000 7fb7ad34b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:24.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.434+0000 7fb7ad34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:24.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:24.434+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fb78c005370 con 0x7fb7a80a3b50 2024-09-16T04:11:25.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.050+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v184) ==== 166+0+0 (unknown 583486770 0 0) 0x7fb798019870 con 0x7fb7a80a3b50 2024-09-16T04:11:25.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.050+0000 7fb79f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.050+0000 7fb79f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-16T04:11:25.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.155+0000 7fb7ad34b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.155+0000 7fb7ad34b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.155+0000 7fb7ad34b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.155+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fb78c005bd0 con 0x7fb7a80a3b50 2024-09-16T04:11:25.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.156+0000 7fb79f7fe640 1 -- 172.21.15.171:0/975959646 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v184) ==== 145+0+0 (unknown 2346422746 0 0) 0x7fb79801e130 con 0x7fb7a80a3b50 2024-09-16T04:11:25.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.156+0000 7fb79f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.156+0000 7fb79f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-16T04:11:25.158 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h1' already exists 2024-09-16T04:11:25.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 >> v1:172.21.15.171:6812/7327273 conn(0x7fb788038ef0 legacy=0x7fb78803b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 10 monclient: shutdown 2024-09-16T04:11:25.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:25.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 >> v1:172.21.15.171:6789/0 conn(0x7fb7a80a3b50 legacy=0x7fb7a8138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 shutdown_connections 2024-09-16T04:11:25.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 >> 172.21.15.171:0/975959646 conn(0x7fb7a80a9940 msgr2=0x7fb7a80a1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:25.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 shutdown_connections 2024-09-16T04:11:25.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.158+0000 7fb7ad34b640 1 -- 172.21.15.171:0/975959646 wait complete. 2024-09-16T04:11:25.170 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:25.170 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-16T04:11:25.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.250+0000 7f72f4fdf640 10 monclient: get_monmap_and_config 2024-09-16T04:11:25.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.250+0000 7f72f4fdf640 10 monclient: build_initial_monmap 2024-09-16T04:11:25.252 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.250+0000 7f72f4fdf640 10 monclient: monmap: 2024-09-16T04:11:25.252 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:25.252 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:25.252 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:25.251297+0000 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:25.251297+0000 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:25.253 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 1 Processor -- start 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 1 -- start start 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient: init 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient(hunting): picked mon.noname-b con 0x7f72f009ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient(hunting): picked mon.noname-c con 0x7f72f00a0370 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:25.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient(hunting): picked mon.noname-a con 0x7f72f00a0800 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f00a0f20 con 0x7f72f00a0800 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f00a1200 con 0x7f72f00a0370 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f009bca0 con 0x7f72f009ff90 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72f4fdf640 10 monclient(hunting): authenticate will time out at 1060.756763s 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72ee59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f72f009ff90 legacy=0x7f72f009f7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72edd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f72f00a0370 0x7f72f0139a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34330/0 (socket says 172.21.15.171:34330) 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.251+0000 7f72edd9b640 1 -- 172.21.15.171:0/710357546 learned_addr learned my addr 172.21.15.171:0/710357546 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972790147 0 0) 0x7f72f00a1200 con 0x7f72f00a0370 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:25.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): my global_id is 5959 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d8003520 con 0x7f72f00a0370 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993221583 0 0) 0x7f72f00a0f20 con 0x7f72f00a0800 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient(hunting): my global_id is 64398 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72f00a1200 con 0x7f72f00a0800 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1299992298 0 0) 0x7f72d8003520 con 0x7f72f00a0370 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 >> v1:172.21.15.148:6790/0 conn(0x7f72f009ff90 legacy=0x7f72f009f7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 >> v1:172.21.15.148:6789/0 conn(0x7f72f00a0800 legacy=0x7f72f013d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 monclient: found mon.noname-c 2024-09-16T04:11:25.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f009e830 con 0x7f72f00a0370 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: _finish_auth 0 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72e4001610 con 0x7f72f00a0370 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: dump: 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.263 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 5 monclient: authenticate success, global_id 5959 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72e4001930 con 0x7f72f00a0370 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 1 -- 172.21.15.171:0/710357546 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72e4001d00 con 0x7f72f00a0370 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:25.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 4 set_mon_vals no callback set 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72ed59a640 10 monclient: dump: 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 10 monclient: shutdown 2024-09-16T04:11:25.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.252+0000 7f72f4fdf640 1 -- 172.21.15.171:0/710357546 >> v1:172.21.15.171:6789/0 conn(0x7f72f00a0370 legacy=0x7f72f0139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.253+0000 7f72f4fdf640 1 -- 172.21.15.171:0/710357546 shutdown_connections 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.253+0000 7f72f4fdf640 1 -- 172.21.15.171:0/710357546 >> 172.21.15.171:0/710357546 conn(0x7f72f00a5450 msgr2=0x7f72f00a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.253+0000 7f72f4fdf640 1 -- 172.21.15.171:0/710357546 shutdown_connections 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.253+0000 7f72f4fdf640 1 -- 172.21.15.171:0/710357546 wait complete. 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.253+0000 7f72f4fdf640 10 monclient: build_initial_monmap 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.253+0000 7f72f4fdf640 10 monclient: monmap: 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:25.266 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 1 Processor -- start 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 1 -- start start 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient: init 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient(hunting): picked mon.noname-c con 0x7f72f009ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:25.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient(hunting): picked mon.noname-b con 0x7f72f00a0370 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient(hunting): picked mon.noname-a con 0x7f72f00a0800 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f013e6b0 con 0x7f72f00a0800 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f013c4c0 con 0x7f72f00a0370 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f006eaa0 con 0x7f72f009ff90 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72f4fdf640 10 monclient(hunting): authenticate will time out at 1060.759632s 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72edd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f72f00a0370 0x7f72f01d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34338/0 (socket says 172.21.15.171:34338) 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72edd9b640 1 -- 172.21.15.171:0/1949469534 learned_addr learned my addr 172.21.15.171:0/1949469534 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72ee59c640 1 -- 172.21.15.171:0/1949469534 >> v1:172.21.15.148:6790/0 conn(0x7f72f009ff90 legacy=0x7f72f01cbb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029071273 0 0) 0x7f72f013c4c0 con 0x7f72f00a0370 2024-09-16T04:11:25.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72deffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72deffd640 10 monclient(hunting): my global_id is 5962 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.254+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d0003520 con 0x7f72f00a0370 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555966868 0 0) 0x7f72f013e6b0 con 0x7f72f00a0800 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): my global_id is 64401 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72f013c4c0 con 0x7f72f00a0800 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1606953042 0 0) 0x7f72d0003520 con 0x7f72f00a0370 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72f013e6b0 con 0x7f72f00a0370 2024-09-16T04:11:25.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72e40015d0 con 0x7f72f00a0370 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): dump: 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:25.270 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2249410713 0 0) 0x7f72f013e6b0 con 0x7f72f00a0370 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 >> v1:172.21.15.148:6790/0 conn(0x7f72f009ff90 legacy=0x7f72f01cbb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 >> v1:172.21.15.148:6789/0 conn(0x7f72f00a0800 legacy=0x7f72f01d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 monclient: found mon.b 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f009c0b0 con 0x7f72f00a0370 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:25.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 5 monclient: authenticate success, global_id 5962 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 10 monclient: _renew_subs 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72f013baf0 con 0x7f72f00a0370 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 10 monclient: _renew_subs 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72f01342e0 con 0x7f72f00a0370 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72e4001930 con 0x7f72f00a0370 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f72e4004e60 con 0x7f72f00a0370 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72ecd99640 4 set_mon_vals no callback set 2024-09-16T04:11:25.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.255+0000 7f72deffd640 10 monclient: dump: 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f72e4010290 con 0x7f72f00a0370 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72f4fdf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:25.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72f4fdf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72f4fdf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d4005080 con 0x7f72f00a0370 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(184..184 src has 1..184) ==== 7181+0+0 (unknown 2163943818 0 0) 0x7f72e4047c50 con 0x7f72f00a0370 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.256+0000 7f72ee59c640 10 monclient: get_auth_request con 0x7f72d0039060 auth_method 0 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.260+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f72e4015e70 con 0x7f72f00a0370 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.260+0000 7f72deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:25.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.260+0000 7f72deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:25.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.432+0000 7f72f4fdf640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.432+0000 7f72f4fdf640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.432+0000 7f72f4fdf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.432+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f72d4005370 con 0x7f72f00a0370 2024-09-16T04:11:25.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.821+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v185) ==== 166+0+0 (unknown 115398724 0 0) 0x7f72e4015770 con 0x7f72f00a0370 2024-09-16T04:11:25.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.821+0000 7f72deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.823 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.821+0000 7f72deffd640 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-16T04:11:25.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.925+0000 7f72f4fdf640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.925+0000 7f72f4fdf640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.925+0000 7f72f4fdf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:25.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.925+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f72d4005bd0 con 0x7f72f00a0370 2024-09-16T04:11:25.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.926+0000 7f72deffd640 1 -- 172.21.15.171:0/1949469534 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v185) ==== 145+0+0 (unknown 3664088182 0 0) 0x7f72e4013ad0 con 0x7f72f00a0370 2024-09-16T04:11:25.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.926+0000 7f72deffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:25.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.926+0000 7f72deffd640 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-16T04:11:25.929 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h2' already exists 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 >> v1:172.21.15.171:6812/7327273 conn(0x7f72d0039060 legacy=0x7f72d003b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 10 monclient: shutdown 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 >> v1:172.21.15.171:6789/0 conn(0x7f72f00a0370 legacy=0x7f72f01d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 shutdown_connections 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 >> 172.21.15.171:0/1949469534 conn(0x7f72f00a5450 msgr2=0x7f72f009f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 shutdown_connections 2024-09-16T04:11:25.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:25.929+0000 7f72f4fdf640 1 -- 172.21.15.171:0/1949469534 wait complete. 2024-09-16T04:11:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:25.940 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.018+0000 7f0b69bd6640 10 monclient: get_monmap_and_config 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.018+0000 7f0b69bd6640 10 monclient: build_initial_monmap 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.018+0000 7f0b69bd6640 10 monclient: monmap: 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:26.019228+0000 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:26.019228+0000 2024-09-16T04:11:26.020 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:26.021 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.021 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:26.021 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:26.021 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:26.021 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 1 Processor -- start 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 1 -- start start 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient: init 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6409f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6409fba0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:26.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b641395d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b640a1490 con 0x7f0b641395d0 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6409d1b0 con 0x7f0b6409fba0 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b6409d490 con 0x7f0b6409f7c0 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b69bd6640 10 monclient(hunting): authenticate will time out at 1061.524706s 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b637fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0b6409f7c0 legacy=0x7f0b6409de70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b62ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0b6409fba0 0x7f0b640a0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34348/0 (socket says 172.21.15.171:34348) 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.019+0000 7f0b62ffd640 1 -- 172.21.15.171:0/461026762 learned_addr learned my addr 172.21.15.171:0/461026762 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578608197 0 0) 0x7f0b6409d1b0 con 0x7f0b6409fba0 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:26.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): my global_id is 5965 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b4c003520 con 0x7f0b6409fba0 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336647827 0 0) 0x7f0b640a1490 con 0x7f0b641395d0 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient(hunting): my global_id is 64404 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b6409d1b0 con 0x7f0b641395d0 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1507280535 0 0) 0x7f0b4c003520 con 0x7f0b6409fba0 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 >> v1:172.21.15.148:6790/0 conn(0x7f0b6409f7c0 legacy=0x7f0b6409de70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 >> v1:172.21.15.148:6789/0 conn(0x7f0b641395d0 legacy=0x7f0b6413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:26.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6409bca0 con 0x7f0b6409fba0 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0b58001610 con 0x7f0b6409fba0 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 5 monclient: authenticate success, global_id 5965 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:26.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: dump: 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.033 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b58001930 con 0x7f0b6409fba0 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 1 -- 172.21.15.171:0/461026762 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0b58001d00 con 0x7f0b6409fba0 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b627fc640 10 monclient: dump: 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.034 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 4 set_mon_vals no callback set 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 10 monclient: shutdown 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 1 -- 172.21.15.171:0/461026762 >> v1:172.21.15.171:6789/0 conn(0x7f0b6409fba0 legacy=0x7f0b640a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 1 -- 172.21.15.171:0/461026762 shutdown_connections 2024-09-16T04:11:26.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.020+0000 7f0b69bd6640 1 -- 172.21.15.171:0/461026762 >> 172.21.15.171:0/461026762 conn(0x7f0b640a5450 msgr2=0x7f0b640a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.021+0000 7f0b69bd6640 1 -- 172.21.15.171:0/461026762 shutdown_connections 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.021+0000 7f0b69bd6640 1 -- 172.21.15.171:0/461026762 wait complete. 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.021+0000 7f0b69bd6640 10 monclient: build_initial_monmap 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.021+0000 7f0b69bd6640 10 monclient: monmap: 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:26.036 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.021+0000 7f0b69bd6640 1 Processor -- start 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 1 -- start start 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient: init 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6409f7c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6409fba0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b641395d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b64122800 con 0x7f0b641395d0 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b64120640 con 0x7f0b6409f7c0 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b640a4200 con 0x7f0b6409fba0 2024-09-16T04:11:26.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b69bd6640 10 monclient(hunting): authenticate will time out at 1061.527483s 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b637fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0b6409f7c0 0x7f0b6411f820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34358/0 (socket says 172.21.15.171:34358) 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b62ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0b6409fba0 legacy=0x7f0b6411ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b637fe640 1 -- 172.21.15.171:0/1090730501 learned_addr learned my addr 172.21.15.171:0/1090730501 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994731295 0 0) 0x7f0b64120640 con 0x7f0b6409f7c0 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): my global_id is 5968 2024-09-16T04:11:26.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b50003520 con 0x7f0b6409f7c0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3134823106 0 0) 0x7f0b64122800 con 0x7f0b641395d0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): my global_id is 64407 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b64120640 con 0x7f0b641395d0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2418877780 0 0) 0x7f0b50003520 con 0x7f0b6409f7c0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b64122800 con 0x7f0b6409f7c0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0b54004180 con 0x7f0b6409f7c0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): dump: 2024-09-16T04:11:26.039 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.022+0000 7f0b4bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 189395940 0 0) 0x7f0b64120640 con 0x7f0b641395d0 2024-09-16T04:11:26.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0b50003520 con 0x7f0b641395d0 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0b5c001c30 con 0x7f0b641395d0 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient(hunting): dump: 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:26.041 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1783332574 0 0) 0x7f0b64122800 con 0x7f0b6409f7c0 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 >> v1:172.21.15.148:6790/0 conn(0x7f0b6409fba0 legacy=0x7f0b6411ff30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 >> v1:172.21.15.148:6789/0 conn(0x7f0b641395d0 legacy=0x7f0b641d3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 monclient: found mon.b 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b640a3cd0 con 0x7f0b6409f7c0 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: _finish_auth 0 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 5 monclient: authenticate success, global_id 5968 2024-09-16T04:11:26.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 10 monclient: _renew_subs 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0b64122dc0 con 0x7f0b6409f7c0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 10 monclient: _renew_subs 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0b64122ae0 con 0x7f0b6409f7c0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0b540043e0 con 0x7f0b6409f7c0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0b540047b0 con 0x7f0b6409f7c0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b61ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.023+0000 7f0b4bfff640 10 monclient: dump: 2024-09-16T04:11:26.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0b54004950 con 0x7f0b6409f7c0 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b69bd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b69bd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b69bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:26.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b44005080 con 0x7f0b6409f7c0 2024-09-16T04:11:26.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(185..185 src has 1..185) ==== 7391+0+0 (unknown 1331857852 0 0) 0x7f0b54047b80 con 0x7f0b6409f7c0 2024-09-16T04:11:26.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.024+0000 7f0b62ffd640 10 monclient: get_auth_request con 0x7f0b50038f60 auth_method 0 2024-09-16T04:11:26.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.028+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0b54015c10 con 0x7f0b6409f7c0 2024-09-16T04:11:26.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.028+0000 7f0b4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:26.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.028+0000 7f0b4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.199+0000 7f0b69bd6640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:26.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.199+0000 7f0b69bd6640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.199+0000 7f0b69bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:26.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:26.199+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0b44005370 con 0x7f0b6409f7c0 2024-09-16T04:11:27.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.060+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v186) ==== 166+0+0 (unknown 302430699 0 0) 0x7f0b54015510 con 0x7f0b6409f7c0 2024-09-16T04:11:27.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.060+0000 7f0b4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:27.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.060+0000 7f0b4bfff640 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-16T04:11:27.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.163+0000 7f0b69bd6640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:27.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.163+0000 7f0b69bd6640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.163+0000 7f0b69bd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.163+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0b44005bd0 con 0x7f0b6409f7c0 2024-09-16T04:11:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.164+0000 7f0b4bfff640 1 -- 172.21.15.171:0/1090730501 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v186) ==== 145+0+0 (unknown 157880584 0 0) 0x7f0b5401e0c0 con 0x7f0b6409f7c0 2024-09-16T04:11:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.164+0000 7f0b4bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.164+0000 7f0b4bfff640 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-16T04:11:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h3' already exists 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 >> v1:172.21.15.171:6812/7327273 conn(0x7f0b50038f60 legacy=0x7f0b5003b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 10 monclient: shutdown 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 >> v1:172.21.15.171:6789/0 conn(0x7f0b6409f7c0 legacy=0x7f0b6411f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 shutdown_connections 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 >> 172.21.15.171:0/1090730501 conn(0x7f0b640a5450 msgr2=0x7f0b641349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 shutdown_connections 2024-09-16T04:11:27.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.167+0000 7f0b69bd6640 1 -- 172.21.15.171:0/1090730501 wait complete. 2024-09-16T04:11:27.177 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:27.177 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.258+0000 7f3b2d16d640 10 monclient: get_monmap_and_config 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.258+0000 7f3b2d16d640 10 monclient: build_initial_monmap 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.258+0000 7f3b2d16d640 10 monclient: monmap: 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:27.258908+0000 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:27.258908+0000 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:27.260 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:27.261 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:27.261 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:27.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.258+0000 7f3b2d16d640 1 Processor -- start 2024-09-16T04:11:27.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.258+0000 7f3b2d16d640 1 -- start start 2024-09-16T04:11:27.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.258+0000 7f3b2d16d640 10 monclient: init 2024-09-16T04:11:27.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2809ff90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b280a0370 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b280a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b280a0f20 con 0x7f3b2809ff90 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b280a1200 con 0x7f3b280a0370 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2809bca0 con 0x7f3b280a0800 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient(hunting): authenticate will time out at 1062.764293s 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2759e640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3b280a0800 legacy=0x7f3b2813d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3b280a0370 0x7f3b28139a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33972/0 (socket says 172.21.15.171:33972) 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2659c640 1 -- 172.21.15.171:0/1224211588 learned_addr learned my addr 172.21.15.171:0/1224211588 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:27.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697125525 0 0) 0x7f3b280a1200 con 0x7f3b280a0370 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): my global_id is 5971 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b10003520 con 0x7f3b280a0370 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 812137975 0 0) 0x7f3b280a0f20 con 0x7f3b2809ff90 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient(hunting): my global_id is 64410 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b280a1200 con 0x7f3b2809ff90 2024-09-16T04:11:27.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 608569959 0 0) 0x7f3b10003520 con 0x7f3b280a0370 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 >> v1:172.21.15.148:6790/0 conn(0x7f3b280a0800 legacy=0x7f3b2813d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 >> v1:172.21.15.148:6789/0 conn(0x7f3b2809ff90 legacy=0x7f3b2809f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 monclient: found mon.noname-c 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2809e830 con 0x7f3b280a0370 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3b1c001610 con 0x7f3b280a0370 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:27.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: dump: 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:27.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 5 monclient: authenticate success, global_id 5971 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b1c001930 con 0x7f3b280a0370 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 1 -- 172.21.15.171:0/1224211588 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3b1c001d00 con 0x7f3b280a0370 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b2d16d640 4 set_mon_vals no callback set 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.259+0000 7f3b25d9b640 10 monclient: dump: 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:27.273 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 10 monclient: shutdown 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 1 -- 172.21.15.171:0/1224211588 >> v1:172.21.15.171:6789/0 conn(0x7f3b280a0370 legacy=0x7f3b28139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 1 -- 172.21.15.171:0/1224211588 shutdown_connections 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 1 -- 172.21.15.171:0/1224211588 >> 172.21.15.171:0/1224211588 conn(0x7f3b280a5450 msgr2=0x7f3b280a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:27.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 1 -- 172.21.15.171:0/1224211588 shutdown_connections 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.260+0000 7f3b2d16d640 1 -- 172.21.15.171:0/1224211588 wait complete. 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.261+0000 7f3b2d16d640 10 monclient: build_initial_monmap 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.261+0000 7f3b2d16d640 10 monclient: monmap: 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:27.275 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.261+0000 7f3b2d16d640 1 Processor -- start 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.261+0000 7f3b2d16d640 1 -- start start 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.261+0000 7f3b2d16d640 10 monclient: init 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2809ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b280a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b280a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2811bfa0 con 0x7f3b280a0370 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2813e2a0 con 0x7f3b2809ff90 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b28071670 con 0x7f3b280a0800 2024-09-16T04:11:27.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient(hunting): authenticate will time out at 1062.767172s 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2759e640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3b280a0800 legacy=0x7f3b281d1940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b26d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3b2809ff90 0x7f3b2813bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33974/0 (socket says 172.21.15.171:33974) 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b26d9d640 1 -- 172.21.15.171:0/3328860079 learned_addr learned my addr 172.21.15.171:0/3328860079 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1886232215 0 0) 0x7f3b2813e2a0 con 0x7f3b2809ff90 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): my global_id is 5974 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b14003520 con 0x7f3b2809ff90 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398708739 0 0) 0x7f3b2811bfa0 con 0x7f3b280a0370 2024-09-16T04:11:27.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): my global_id is 64413 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b2813e2a0 con 0x7f3b280a0370 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2271312807 0 0) 0x7f3b14003520 con 0x7f3b2809ff90 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b2811bfa0 con 0x7f3b2809ff90 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3b18001910 con 0x7f3b2809ff90 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:27.278 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3138797116 0 0) 0x7f3b2811bfa0 con 0x7f3b2809ff90 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 >> v1:172.21.15.148:6790/0 conn(0x7f3b280a0800 legacy=0x7f3b281d1940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 >> v1:172.21.15.148:6789/0 conn(0x7f3b280a0370 legacy=0x7f3b2813c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:27.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 monclient: found mon.b 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2811c560 con 0x7f3b2809ff90 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 5 monclient: authenticate success, global_id 5974 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient: _renew_subs 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2813de20 con 0x7f3b2809ff90 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient: _renew_subs 2024-09-16T04:11:27.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b2813cfe0 con 0x7f3b2809ff90 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b18001ce0 con 0x7f3b2809ff90 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3b18003a60 con 0x7f3b2809ff90 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b2559a640 4 set_mon_vals no callback set 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.262+0000 7f3b0f7fe640 10 monclient: dump: 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:27.281 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3b18003c00 con 0x7f3b2809ff90 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b2d16d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b2d16d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b08005080 con 0x7f3b2809ff90 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(186..186 src has 1..186) ==== 7601+0+0 (unknown 3338056268 0 0) 0x7f3b18047d90 con 0x7f3b2809ff90 2024-09-16T04:11:27.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.263+0000 7f3b2659c640 10 monclient: get_auth_request con 0x7f3b14038be0 auth_method 0 2024-09-16T04:11:27.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.268+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3b18015ca0 con 0x7f3b2809ff90 2024-09-16T04:11:27.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.268+0000 7f3b0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:27.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.268+0000 7f3b0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:27.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.438+0000 7f3b2d16d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:27.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.438+0000 7f3b2d16d640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:27.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.438+0000 7f3b2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:27.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:27.438+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3b08005370 con 0x7f3b2809ff90 2024-09-16T04:11:28.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.062+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v187) ==== 166+0+0 (unknown 1320153768 0 0) 0x7f3b180155a0 con 0x7f3b2809ff90 2024-09-16T04:11:28.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.062+0000 7f3b0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:28.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.062+0000 7f3b0f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-16T04:11:28.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.165+0000 7f3b2d16d640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:28.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.165+0000 7f3b2d16d640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:28.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.165+0000 7f3b2d16d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.165+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3b08005ab0 con 0x7f3b2809ff90 2024-09-16T04:11:28.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.166+0000 7f3b0f7fe640 1 -- 172.21.15.171:0/3328860079 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v187) ==== 145+0+0 (unknown 2031728430 0 0) 0x7f3b180198f0 con 0x7f3b2809ff90 2024-09-16T04:11:28.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.166+0000 7f3b0f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:28.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.166+0000 7f3b0f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-16T04:11:28.168 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h4' already exists 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 >> v1:172.21.15.171:6812/7327273 conn(0x7f3b14038be0 legacy=0x7f3b1403b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 10 monclient: shutdown 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 >> v1:172.21.15.171:6789/0 conn(0x7f3b2809ff90 legacy=0x7f3b2813bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 shutdown_connections 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 >> 172.21.15.171:0/3328860079 conn(0x7f3b280a5450 msgr2=0x7f3b280a2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:28.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 shutdown_connections 2024-09-16T04:11:28.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.168+0000 7f3b2d16d640 1 -- 172.21.15.171:0/3328860079 wait complete. 2024-09-16T04:11:28.179 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:28.179 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-16T04:11:28.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.258+0000 7f0504642640 10 monclient: get_monmap_and_config 2024-09-16T04:11:28.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.258+0000 7f0504642640 10 monclient: build_initial_monmap 2024-09-16T04:11:28.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.258+0000 7f0504642640 10 monclient: monmap: 2024-09-16T04:11:28.260 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:28.260 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:28.259524+0000 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:28.259524+0000 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:28.261 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 1 Processor -- start 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 1 -- start start 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 10 monclient: init 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 10 monclient(hunting): picked mon.noname-c con 0x7f04fc136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 10 monclient(hunting): picked mon.noname-a con 0x7f04fc137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 10 monclient(hunting): picked mon.noname-b con 0x7f04fc1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc09b2c0 con 0x7f04fc137140 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.259+0000 7f0504642640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc09b5a0 con 0x7f04fc136d60 2024-09-16T04:11:28.268 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc0a6640 con 0x7f04fc1375a0 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 10 monclient(hunting): authenticate will time out at 1063.765027s 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05023de640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f04fc136d60 0x7f04fc09bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33986/0 (socket says 172.21.15.171:33986) 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05023de640 1 -- 172.21.15.171:0/4081650255 learned_addr learned my addr 172.21.15.171:0/4081650255 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0502bdf640 1 -- 172.21.15.171:0/4081650255 >> v1:172.21.15.148:6790/0 conn(0x7f04fc1375a0 legacy=0x7f04fc13f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 740490427 0 0) 0x7f04fc09b5a0 con 0x7f04fc136d60 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): my global_id is 5977 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04e8003520 con 0x7f04fc136d60 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4170967078 0 0) 0x7f04fc09b2c0 con 0x7f04fc137140 2024-09-16T04:11:28.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient(hunting): my global_id is 64416 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04fc09b5a0 con 0x7f04fc137140 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2106634639 0 0) 0x7f04e8003520 con 0x7f04fc136d60 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 >> v1:172.21.15.148:6790/0 conn(0x7f04fc1375a0 legacy=0x7f04fc13f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 >> v1:172.21.15.148:6789/0 conn(0x7f04fc137140 legacy=0x7f04fc13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 monclient: found mon.noname-c 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc143b20 con 0x7f04fc136d60 2024-09-16T04:11:28.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: _finish_auth 0 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 5 monclient: authenticate success, global_id 5977 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f04f8001610 con 0x7f04fc136d60 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: dump: 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:28.271 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04f8001870 con 0x7f04fc136d60 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 1 -- 172.21.15.171:0/4081650255 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f04f8001bc0 con 0x7f04fc136d60 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:28.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f0504642640 4 set_mon_vals no callback set 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.260+0000 7f05013dc640 10 monclient: dump: 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:28.273 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 10 monclient: shutdown 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 1 -- 172.21.15.171:0/4081650255 >> v1:172.21.15.171:6789/0 conn(0x7f04fc136d60 legacy=0x7f04fc09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 1 -- 172.21.15.171:0/4081650255 shutdown_connections 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 1 -- 172.21.15.171:0/4081650255 >> 172.21.15.171:0/4081650255 conn(0x7f04fc0a9ae0 msgr2=0x7f04fc0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 1 -- 172.21.15.171:0/4081650255 shutdown_connections 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 1 -- 172.21.15.171:0/4081650255 wait complete. 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 10 monclient: build_initial_monmap 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.261+0000 7f0504642640 10 monclient: monmap: 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:28.274 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 1 Processor -- start 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 1 -- start start 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient: init 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient(hunting): picked mon.noname-a con 0x7f04fc136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient(hunting): picked mon.noname-b con 0x7f04fc137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:28.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient(hunting): picked mon.noname-c con 0x7f04fc1375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc12edd0 con 0x7f04fc136d60 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc12f0b0 con 0x7f04fc137140 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04fc0a2ac0 con 0x7f04fc1375a0 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0502bdf640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f04fc1375a0 legacy=0x7f04fc1d4410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0504642640 10 monclient(hunting): authenticate will time out at 1063.767668s 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0501bdd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f04fc137140 0x7f04fc1d0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:33996/0 (socket says 172.21.15.171:33996) 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f0501bdd640 1 -- 172.21.15.171:0/3757878567 learned_addr learned my addr 172.21.15.171:0/3757878567 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161705833 0 0) 0x7f04fc12f0b0 con 0x7f04fc137140 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:28.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 10 monclient(hunting): my global_id is 5980 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04e0003550 con 0x7f04fc137140 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.262+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462945280 0 0) 0x7f04fc12edd0 con 0x7f04fc136d60 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): my global_id is 64419 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04fc12f0b0 con 0x7f04fc136d60 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2074364039 0 0) 0x7f04e0003550 con 0x7f04fc137140 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04fc12edd0 con 0x7f04fc137140 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f04ec001970 con 0x7f04fc137140 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:28.277 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2912337184 0 0) 0x7f04fc12f0b0 con 0x7f04fc136d60 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04e0003550 con 0x7f04fc136d60 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f04f8001550 con 0x7f04fc136d60 2024-09-16T04:11:28.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1922554831 0 0) 0x7f04fc12edd0 con 0x7f04fc137140 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 >> v1:172.21.15.148:6790/0 conn(0x7f04fc1375a0 legacy=0x7f04fc1d4410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 >> v1:172.21.15.148:6789/0 conn(0x7f04fc136d60 legacy=0x7f04fc1cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 monclient: found mon.b 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc0a88d0 con 0x7f04fc137140 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 5 monclient: authenticate success, global_id 5980 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 10 monclient: _renew_subs 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04fc0a3210 con 0x7f04fc137140 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 10 monclient: _renew_subs 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04fc0a1a90 con 0x7f04fc137140 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f04ec001d20 con 0x7f04fc137140 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f04ec003a60 con 0x7f04fc137140 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f0500bdb640 4 set_mon_vals no callback set 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.263+0000 7f04f2ffd640 10 monclient: dump: 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.281 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f04ec003c00 con 0x7f04fc137140 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(187..187 src has 1..187) ==== 7811+0+0 (unknown 906771826 0 0) 0x7f04ec047dd0 con 0x7f04fc137140 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f05023de640 10 monclient: get_auth_request con 0x7f04e0038f90 auth_method 0 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f0504642640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f0504642640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f0504642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.264+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e4005080 con 0x7f04fc137140 2024-09-16T04:11:28.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.269+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f04ec015c50 con 0x7f04fc137140 2024-09-16T04:11:28.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.269+0000 7f04f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:28.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.269+0000 7f04f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:28.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.438+0000 7f0504642640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:28.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.438+0000 7f0504642640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:28.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.438+0000 7f0504642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:28.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:28.438+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f04e4005370 con 0x7f04fc137140 2024-09-16T04:11:29.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.067+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v188) ==== 166+0+0 (unknown 1128572032 0 0) 0x7f04ec004440 con 0x7f04fc137140 2024-09-16T04:11:29.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.067+0000 7f04f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:29.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.067+0000 7f04f2ffd640 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-16T04:11:29.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.169+0000 7f0504642640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:29.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.169+0000 7f0504642640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:29.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.169+0000 7f0504642640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.169+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f04e4005ab0 con 0x7f04fc137140 2024-09-16T04:11:29.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.170+0000 7f04f2ffd640 1 -- 172.21.15.171:0/3757878567 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v188) ==== 145+0+0 (unknown 2337465999 0 0) 0x7f04ec015550 con 0x7f04fc137140 2024-09-16T04:11:29.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.170+0000 7f04f2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:29.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.170+0000 7f04f2ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-16T04:11:29.172 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h5' already exists 2024-09-16T04:11:29.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 >> v1:172.21.15.171:6812/7327273 conn(0x7f04e0038f90 legacy=0x7f04e003b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:29.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 10 monclient: shutdown 2024-09-16T04:11:29.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:29.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 >> v1:172.21.15.171:6789/0 conn(0x7f04fc137140 legacy=0x7f04fc1d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:29.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 shutdown_connections 2024-09-16T04:11:29.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 >> 172.21.15.171:0/3757878567 conn(0x7f04fc0a9ae0 msgr2=0x7f04fc072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:29.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 shutdown_connections 2024-09-16T04:11:29.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.172+0000 7f0504642640 1 -- 172.21.15.171:0/3757878567 wait complete. 2024-09-16T04:11:29.183 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:29.183 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-16T04:11:29.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.261+0000 7fd8fc923640 10 monclient: get_monmap_and_config 2024-09-16T04:11:29.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.261+0000 7fd8fc923640 10 monclient: build_initial_monmap 2024-09-16T04:11:29.263 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.261+0000 7fd8fc923640 10 monclient: monmap: 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:29.262540+0000 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:29.262540+0000 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:29.264 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:29.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 1 Processor -- start 2024-09-16T04:11:29.272 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 1 -- start start 2024-09-16T04:11:29.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient: init 2024-09-16T04:11:29.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:29.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:29.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f809ff90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:29.273 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f80a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f80a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f80a0f20 con 0x7fd8f809ff90 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f80a1200 con 0x7fd8f80a0800 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f809bca0 con 0x7fd8f80a0370 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.262+0000 7fd8fc923640 10 monclient(hunting): authenticate will time out at 1064.767969s 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f6d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd8f80a0800 0x7fd8f813d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34004/0 (socket says 172.21.15.171:34004) 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f6d9d640 1 -- 172.21.15.171:0/132837640 learned_addr learned my addr 172.21.15.171:0/132837640 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8edd9b640 1 -- 172.21.15.171:0/132837640 >> v1:172.21.15.148:6790/0 conn(0x7fd8f80a0370 legacy=0x7fd8f8139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349413865 0 0) 0x7fd8f80a1200 con 0x7fd8f80a0800 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): my global_id is 5983 2024-09-16T04:11:29.274 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e4003520 con 0x7fd8f80a0800 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122107772 0 0) 0x7fd8f80a0f20 con 0x7fd8f809ff90 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient(hunting): my global_id is 64422 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8f80a1200 con 0x7fd8f809ff90 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2429404608 0 0) 0x7fd8e4003520 con 0x7fd8f80a0800 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 >> v1:172.21.15.148:6790/0 conn(0x7fd8f80a0370 legacy=0x7fd8f8139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 >> v1:172.21.15.148:6789/0 conn(0x7fd8f809ff90 legacy=0x7fd8f809f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 monclient: found mon.noname-c 2024-09-16T04:11:29.275 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f809e830 con 0x7fd8f80a0800 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8fc923640 5 monclient: authenticate success, global_id 5983 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd8f0001990 con 0x7fd8f80a0800 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:29.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: dump: 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8fc923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:29.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8f0001bf0 con 0x7fd8f80a0800 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 1 -- 172.21.15.171:0/132837640 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd8f0001fc0 con 0x7fd8f80a0800 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8fc923640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8fc923640 4 set_mon_vals no callback set 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.263+0000 7fd8f5d9b640 10 monclient: dump: 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:29.278 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 10 monclient: shutdown 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 1 -- 172.21.15.171:0/132837640 >> v1:172.21.15.171:6789/0 conn(0x7fd8f80a0800 legacy=0x7fd8f813d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 1 -- 172.21.15.171:0/132837640 shutdown_connections 2024-09-16T04:11:29.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 1 -- 172.21.15.171:0/132837640 >> 172.21.15.171:0/132837640 conn(0x7fd8f80a5450 msgr2=0x7fd8f80a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 1 -- 172.21.15.171:0/132837640 shutdown_connections 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 1 -- 172.21.15.171:0/132837640 wait complete. 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 10 monclient: build_initial_monmap 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.264+0000 7fd8fc923640 10 monclient: monmap: 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:29.280 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 1 Processor -- start 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 1 -- start start 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient: init 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f809ff90 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient(hunting): picked mon.noname-b con 0x7fd8f80a0370 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient(hunting): picked mon.noname-a con 0x7fd8f80a0800 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f809d110 con 0x7fd8f80a0800 2024-09-16T04:11:29.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f809df40 con 0x7fd8f80a0370 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f813c950 con 0x7fd8f809ff90 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8fc923640 10 monclient(hunting): authenticate will time out at 1064.770858s 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8f659c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fd8f809ff90 legacy=0x7fd8f809c2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8edd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd8f80a0370 0x7fd8f809ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34006/0 (socket says 172.21.15.171:34006) 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.265+0000 7fd8edd9b640 1 -- 172.21.15.171:0/2707154802 learned_addr learned my addr 172.21.15.171:0/2707154802 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2377636321 0 0) 0x7fd8f809df40 con 0x7fd8f80a0370 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): my global_id is 5986 2024-09-16T04:11:29.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8d8003520 con 0x7fd8f80a0370 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840861886 0 0) 0x7fd8f809d110 con 0x7fd8f80a0800 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): my global_id is 64425 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8f809df40 con 0x7fd8f80a0800 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2300933844 0 0) 0x7fd8d8003520 con 0x7fd8f80a0370 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8f809d110 con 0x7fd8f80a0370 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd8e0004150 con 0x7fd8f80a0370 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:29.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:29.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3495963455 0 0) 0x7fd8f809d110 con 0x7fd8f80a0370 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 >> v1:172.21.15.148:6790/0 conn(0x7fd8f809ff90 legacy=0x7fd8f809c2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 >> v1:172.21.15.148:6789/0 conn(0x7fd8f80a0800 legacy=0x7fd8f81d3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 monclient: found mon.b 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f813b2e0 con 0x7fd8f80a0370 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:29.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 5 monclient: authenticate success, global_id 5986 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 10 monclient: _renew_subs 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f813c0b0 con 0x7fd8f80a0370 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 10 monclient: _renew_subs 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8f8134450 con 0x7fd8f80a0370 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8e00043b0 con 0x7fd8f80a0370 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd8e0004700 con 0x7fd8f80a0370 2024-09-16T04:11:29.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:29.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:29.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8f559a640 4 set_mon_vals no callback set 2024-09-16T04:11:29.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.266+0000 7fd8ef7fe640 10 monclient: dump: 2024-09-16T04:11:29.287 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:29.287 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:29.287 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:29.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd8e00048a0 con 0x7fd8f80a0370 2024-09-16T04:11:29.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8fc923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:29.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8fc923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:29.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8fc923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8dc005080 con 0x7fd8f80a0370 2024-09-16T04:11:29.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(188..188 src has 1..188) ==== 8149+0+0 (unknown 637502527 0 0) 0x7fd8e0047ea0 con 0x7fd8f80a0370 2024-09-16T04:11:29.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.267+0000 7fd8f659c640 10 monclient: get_auth_request con 0x7fd8d8038f60 auth_method 0 2024-09-16T04:11:29.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.272+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd8e0076d40 con 0x7fd8f80a0370 2024-09-16T04:11:29.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.272+0000 7fd8ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:29.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.272+0000 7fd8ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:29.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.442+0000 7fd8fc923640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:29.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.442+0000 7fd8fc923640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:29.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.442+0000 7fd8fc923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:29.445 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:29.442+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fd8dc005370 con 0x7fd8f80a0370 2024-09-16T04:11:30.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.071+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v189) ==== 166+0+0 (unknown 1734554102 0 0) 0x7fd8e0019820 con 0x7fd8f80a0370 2024-09-16T04:11:30.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.071+0000 7fd8ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:30.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.071+0000 7fd8ef7fe640 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-16T04:11:30.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.174+0000 7fd8fc923640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:30.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.174+0000 7fd8fc923640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:30.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.174+0000 7fd8fc923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.174+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fd8dc005ab0 con 0x7fd8f80a0370 2024-09-16T04:11:30.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.175+0000 7fd8ef7fe640 1 -- 172.21.15.171:0/2707154802 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v189) ==== 145+0+0 (unknown 3672963875 0 0) 0x7fd8e001e0e0 con 0x7fd8f80a0370 2024-09-16T04:11:30.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.175+0000 7fd8ef7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:30.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.175+0000 7fd8ef7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-16T04:11:30.178 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h6' already exists 2024-09-16T04:11:30.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 >> v1:172.21.15.171:6812/7327273 conn(0x7fd8d8038f60 legacy=0x7fd8d803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:30.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 10 monclient: shutdown 2024-09-16T04:11:30.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:30.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 >> v1:172.21.15.171:6789/0 conn(0x7fd8f80a0370 legacy=0x7fd8f809ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:30.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 shutdown_connections 2024-09-16T04:11:30.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 >> 172.21.15.171:0/2707154802 conn(0x7fd8f80a5450 msgr2=0x7fd8f809f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:30.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 shutdown_connections 2024-09-16T04:11:30.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.177+0000 7fd8fc923640 1 -- 172.21.15.171:0/2707154802 wait complete. 2024-09-16T04:11:30.188 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:30.188 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-16T04:11:30.269 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.267+0000 7fbd79c1e640 10 monclient: get_monmap_and_config 2024-09-16T04:11:30.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.267+0000 7fbd79c1e640 10 monclient: build_initial_monmap 2024-09-16T04:11:30.270 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.267+0000 7fbd79c1e640 10 monclient: monmap: 2024-09-16T04:11:30.270 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:30.270 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:30.270 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:30.268662+0000 2024-09-16T04:11:30.270 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:30.268662+0000 2024-09-16T04:11:30.271 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:30.271 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:30.271 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:30.271 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:30.271 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:30.271 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.268+0000 7fbd79c1e640 1 Processor -- start 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.268+0000 7fbd79c1e640 1 -- start start 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.268+0000 7fbd79c1e640 10 monclient: init 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient(hunting): picked mon.noname-c con 0x7fbd74133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient(hunting): picked mon.noname-a con 0x7fbd741341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:30.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient(hunting): picked mon.noname-b con 0x7fbd7413a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd7409da90 con 0x7fbd741341f0 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd7409dd70 con 0x7fbd74133700 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd7409e050 con 0x7fbd7413a350 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient(hunting): authenticate will time out at 1065.774177s 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd737fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbd74133700 0x7fbd74133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34010/0 (socket says 172.21.15.171:34010) 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd737fe640 1 -- 172.21.15.171:0/2530465132 learned_addr learned my addr 172.21.15.171:0/2530465132 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd73fff640 1 -- 172.21.15.171:0/2530465132 >> v1:172.21.15.148:6790/0 conn(0x7fbd7413a350 legacy=0x7fbd7413c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650389431 0 0) 0x7fbd7409dd70 con 0x7fbd74133700 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:30.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient(hunting): my global_id is 5989 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd5c003520 con 0x7fbd74133700 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2832708425 0 0) 0x7fbd5c003520 con 0x7fbd74133700 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 >> v1:172.21.15.148:6790/0 conn(0x7fbd7413a350 legacy=0x7fbd7413c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 >> v1:172.21.15.148:6789/0 conn(0x7fbd741341f0 legacy=0x7fbd74138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7409d4f0 con 0x7fbd74133700 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:30.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd64001610 con 0x7fbd74133700 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 5 monclient: authenticate success, global_id 5989 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: dump: 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:30.284 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd64001970 con 0x7fbd74133700 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 1 -- 172.21.15.171:0/2530465132 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd64001d40 con 0x7fbd74133700 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd79c1e640 4 set_mon_vals no callback set 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.269+0000 7fbd727fc640 10 monclient: dump: 2024-09-16T04:11:30.285 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 10 monclient: shutdown 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 1 -- 172.21.15.171:0/2530465132 >> v1:172.21.15.171:6789/0 conn(0x7fbd74133700 legacy=0x7fbd74133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 1 -- 172.21.15.171:0/2530465132 shutdown_connections 2024-09-16T04:11:30.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 1 -- 172.21.15.171:0/2530465132 >> 172.21.15.171:0/2530465132 conn(0x7fbd740a5450 msgr2=0x7fbd740a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 1 -- 172.21.15.171:0/2530465132 shutdown_connections 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 1 -- 172.21.15.171:0/2530465132 wait complete. 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 10 monclient: build_initial_monmap 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.270+0000 7fbd79c1e640 10 monclient: monmap: 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:30.287 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 1 Processor -- start 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 1 -- start start 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 10 monclient: init 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 10 monclient(hunting): picked mon.noname-c con 0x7fbd74133700 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 10 monclient(hunting): picked mon.noname-a con 0x7fbd741341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.271+0000 7fbd79c1e640 10 monclient(hunting): picked mon.noname-b con 0x7fbd7413a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd7411ccc0 con 0x7fbd741341f0 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd740a4720 con 0x7fbd7413a350 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd740a18a0 con 0x7fbd74133700 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:30.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd737fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fbd74133700 legacy=0x7fbd741c7730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 10 monclient(hunting): authenticate will time out at 1065.777052s 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd73fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbd7413a350 0x7fbd741cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34012/0 (socket says 172.21.15.171:34012) 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd73fff640 1 -- 172.21.15.171:0/217628825 learned_addr learned my addr 172.21.15.171:0/217628825 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110014722 0 0) 0x7fbd740a4720 con 0x7fbd7413a350 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): my global_id is 5992 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd60003520 con 0x7fbd7413a350 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786403285 0 0) 0x7fbd7411ccc0 con 0x7fbd741341f0 2024-09-16T04:11:30.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): my global_id is 64431 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd740a4720 con 0x7fbd741341f0 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 453439049 0 0) 0x7fbd60003520 con 0x7fbd7413a350 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd7411ccc0 con 0x7fbd7413a350 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd6c0043b0 con 0x7fbd7413a350 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): dump: 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.291 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1872407216 0 0) 0x7fbd7411ccc0 con 0x7fbd7413a350 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 >> v1:172.21.15.148:6790/0 conn(0x7fbd74133700 legacy=0x7fbd741c7730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 >> v1:172.21.15.148:6789/0 conn(0x7fbd741341f0 legacy=0x7fbd741cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 monclient: found mon.b 2024-09-16T04:11:30.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7411c700 con 0x7fbd7413a350 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: _finish_auth 0 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 5 monclient: authenticate success, global_id 5992 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 10 monclient: _renew_subs 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd7411c9e0 con 0x7fbd7413a350 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 10 monclient: _renew_subs 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd7411d0e0 con 0x7fbd7413a350 2024-09-16T04:11:30.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd6c004610 con 0x7fbd7413a350 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd6c004960 con 0x7fbd7413a350 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd5bfff640 10 monclient: dump: 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:30.294 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.272+0000 7fbd71ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbd6c004b00 con 0x7fbd7413a350 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(189..189 src has 1..189) ==== 8359+0+0 (unknown 2496228511 0 0) 0x7fbd6c048190 con 0x7fbd7413a350 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd737fe640 10 monclient: get_auth_request con 0x7fbd60039000 auth_method 0 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd79c1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd79c1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd79c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.273+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd74076330 con 0x7fbd7413a350 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.278+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbd74076330 con 0x7fbd7413a350 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.278+0000 7fbd5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:30.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.278+0000 7fbd5bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:30.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.447+0000 7fbd79c1e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:30.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.447+0000 7fbd79c1e640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:30.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.447+0000 7fbd79c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:30.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:30.447+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fbd740717d0 con 0x7fbd7413a350 2024-09-16T04:11:31.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.083+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v190) ==== 166+0+0 (unknown 1938116697 0 0) 0x7fbd6c015d50 con 0x7fbd7413a350 2024-09-16T04:11:31.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.083+0000 7fbd5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:31.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.083+0000 7fbd5bfff640 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-16T04:11:31.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.185+0000 7fbd79c1e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:31.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.186+0000 7fbd79c1e640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:31.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.186+0000 7fbd79c1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.186+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fbd740008d0 con 0x7fbd7413a350 2024-09-16T04:11:31.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.187+0000 7fbd5bfff640 1 -- 172.21.15.171:0/217628825 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v190) ==== 145+0+0 (unknown 165701213 0 0) 0x7fbd6c015450 con 0x7fbd7413a350 2024-09-16T04:11:31.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.187+0000 7fbd5bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:31.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.187+0000 7fbd5bfff640 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-16T04:11:31.189 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h7' already exists 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 >> v1:172.21.15.171:6812/7327273 conn(0x7fbd60039000 legacy=0x7fbd6003b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 10 monclient: shutdown 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 >> v1:172.21.15.171:6789/0 conn(0x7fbd7413a350 legacy=0x7fbd741cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 shutdown_connections 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 >> 172.21.15.171:0/217628825 conn(0x7fbd740a5450 msgr2=0x7fbd740728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 shutdown_connections 2024-09-16T04:11:31.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.189+0000 7fbd79c1e640 1 -- 172.21.15.171:0/217628825 wait complete. 2024-09-16T04:11:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:31.200 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-16T04:11:31.281 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.280+0000 7f8bacfed640 10 monclient: get_monmap_and_config 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.280+0000 7f8bacfed640 10 monclient: build_initial_monmap 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.280+0000 7f8bacfed640 10 monclient: monmap: 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:31.280806+0000 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:31.280806+0000 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:31.282 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:31.283 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:31.283 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:31.283 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:31.283 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:31.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.280+0000 7f8bacfed640 1 Processor -- start 2024-09-16T04:11:31.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.280+0000 7f8bacfed640 1 -- start start 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.280+0000 7f8bacfed640 10 monclient: init 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba8136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba8137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba81375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba809b2c0 con 0x7f8ba81375a0 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba809b5a0 con 0x7f8ba8136d60 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba80a6640 con 0x7f8ba8137140 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 10 monclient(hunting): authenticate will time out at 1066.786309s 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8ba8136d60 0x7f8ba809bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34028/0 (socket says 172.21.15.171:34028) 2024-09-16T04:11:31.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba659c640 1 -- 172.21.15.171:0/1357581170 learned_addr learned my addr 172.21.15.171:0/1357581170 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba5d9b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f8ba8137140 legacy=0x7f8ba813bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3379562274 0 0) 0x7f8ba809b5a0 con 0x7f8ba8136d60 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient(hunting): my global_id is 5995 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b90003520 con 0x7f8ba8136d60 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2450797502 0 0) 0x7f8b90003520 con 0x7f8ba8136d60 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 >> v1:172.21.15.148:6790/0 conn(0x7f8ba8137140 legacy=0x7f8ba813bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 >> v1:172.21.15.148:6789/0 conn(0x7f8ba81375a0 legacy=0x7f8ba813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 monclient: found mon.noname-c 2024-09-16T04:11:31.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba8143b20 con 0x7f8ba8136d60 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: _finish_auth 0 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ba0001610 con 0x7f8ba8136d60 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8ba559a640 10 monclient: dump: 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:31.293 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 5 monclient: authenticate success, global_id 5995 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.281+0000 7f8bacfed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:31.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ba0001900 con 0x7f8ba8136d60 2024-09-16T04:11:31.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8ba559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:31.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8ba559a640 1 -- 172.21.15.171:0/1357581170 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ba0001d10 con 0x7f8ba8136d60 2024-09-16T04:11:31.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:31.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 4 set_mon_vals no callback set 2024-09-16T04:11:31.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8ba559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:31.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8ba559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:31.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8ba559a640 10 monclient: dump: 2024-09-16T04:11:31.296 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:31.296 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:31.296 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.296 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.296 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:31.297 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:31.297 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:31.297 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:31.297 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:31.297 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:31.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 10 monclient: shutdown 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 1 -- 172.21.15.171:0/1357581170 >> v1:172.21.15.171:6789/0 conn(0x7f8ba8136d60 legacy=0x7f8ba809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 1 -- 172.21.15.171:0/1357581170 shutdown_connections 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 1 -- 172.21.15.171:0/1357581170 >> 172.21.15.171:0/1357581170 conn(0x7f8ba80a9ae0 msgr2=0x7f8ba80a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 1 -- 172.21.15.171:0/1357581170 shutdown_connections 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.282+0000 7f8bacfed640 1 -- 172.21.15.171:0/1357581170 wait complete. 2024-09-16T04:11:31.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.283+0000 7f8bacfed640 10 monclient: build_initial_monmap 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.283+0000 7f8bacfed640 10 monclient: monmap: 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:31.299 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:31.300 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:31.300 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:31.300 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:31.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.283+0000 7f8bacfed640 1 Processor -- start 2024-09-16T04:11:31.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.283+0000 7f8bacfed640 1 -- start start 2024-09-16T04:11:31.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.283+0000 7f8bacfed640 10 monclient: init 2024-09-16T04:11:31.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:31.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:31.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba8136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:31.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba8137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:31.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba81375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:31.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba8138d20 con 0x7f8ba8137140 2024-09-16T04:11:31.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba813a590 con 0x7f8ba8136d60 2024-09-16T04:11:31.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba80a8740 con 0x7f8ba81375a0 2024-09-16T04:11:31.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:31.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient(hunting): authenticate will time out at 1066.789230s 2024-09-16T04:11:31.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8ba659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8ba8136d60 0x7f8ba8137f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34030/0 (socket says 172.21.15.171:34030) 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8ba659c640 1 -- 172.21.15.171:0/3882021127 learned_addr learned my addr 172.21.15.171:0/3882021127 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8ba6d9d640 1 -- 172.21.15.171:0/3882021127 >> v1:172.21.15.148:6790/0 conn(0x7f8ba81375a0 legacy=0x7f8ba81d3a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736022385 0 0) 0x7f8ba813a590 con 0x7f8ba8136d60 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 5998 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b88003550 con 0x7f8ba8136d60 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2137888258 0 0) 0x7f8ba8138d20 con 0x7f8ba8137140 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:31.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 64437 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba813a590 con 0x7f8ba8137140 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2321817606 0 0) 0x7f8b88003550 con 0x7f8ba8136d60 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ba8138d20 con 0x7f8ba8136d60 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ba00015d0 con 0x7f8ba8136d60 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.304 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4173853790 0 0) 0x7f8ba8138d20 con 0x7f8ba8136d60 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 >> v1:172.21.15.148:6790/0 conn(0x7f8ba81375a0 legacy=0x7f8ba81d3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:31.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 >> v1:172.21.15.148:6789/0 conn(0x7f8ba8137140 legacy=0x7f8ba8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:31.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:31.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 monclient: found mon.b 2024-09-16T04:11:31.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:31.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba813bd20 con 0x7f8ba8136d60 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 5 monclient: authenticate success, global_id 5998 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient: _renew_subs 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba813b480 con 0x7f8ba8136d60 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient: _renew_subs 2024-09-16T04:11:31.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ba813a310 con 0x7f8ba8136d60 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8ba0004cc0 con 0x7f8ba8136d60 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8ba0004e70 con 0x7f8ba8136d60 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8ba4d99640 4 set_mon_vals no callback set 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.284+0000 7f8b96ffd640 10 monclient: dump: 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.308 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.285+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8ba0005010 con 0x7f8ba8136d60 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.285+0000 7f8bacfed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.285+0000 7f8bacfed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.285+0000 7f8bacfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.285+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8c005080 con 0x7f8ba8136d60 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.285+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(190..190 src has 1..190) ==== 8569+0+0 (unknown 2317440643 0 0) 0x7f8ba0048210 con 0x7f8ba8136d60 2024-09-16T04:11:31.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.286+0000 7f8ba5d9b640 10 monclient: get_auth_request con 0x7f8b88038fc0 auth_method 0 2024-09-16T04:11:31.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.290+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8ba0015e10 con 0x7f8ba8136d60 2024-09-16T04:11:31.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.290+0000 7f8b96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:31.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.290+0000 7f8b96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:31.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.459+0000 7f8bacfed640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:31.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.459+0000 7f8bacfed640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:31.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.459+0000 7f8bacfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:31.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:31.459+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f8b8c005370 con 0x7f8ba8136d60 2024-09-16T04:11:32.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.083+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v191) ==== 166+0+0 (unknown 3727796080 0 0) 0x7f8ba0015710 con 0x7f8ba8136d60 2024-09-16T04:11:32.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.083+0000 7f8b96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:32.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.083+0000 7f8b96ffd640 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-16T04:11:32.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.186+0000 7f8bacfed640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:32.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.186+0000 7f8bacfed640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:32.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.186+0000 7f8bacfed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.186+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f8b8c005ab0 con 0x7f8ba8136d60 2024-09-16T04:11:32.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.188+0000 7f8b96ffd640 1 -- 172.21.15.171:0/3882021127 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v191) ==== 145+0+0 (unknown 973977455 0 0) 0x7f8ba001e260 con 0x7f8ba8136d60 2024-09-16T04:11:32.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.188+0000 7f8b96ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:32.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.188+0000 7f8b96ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-16T04:11:32.190 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h8' already exists 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 >> v1:172.21.15.171:6812/7327273 conn(0x7f8b88038fc0 legacy=0x7f8b8803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 10 monclient: shutdown 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 >> v1:172.21.15.171:6789/0 conn(0x7f8ba8136d60 legacy=0x7f8ba8137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 shutdown_connections 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 >> 172.21.15.171:0/3882021127 conn(0x7f8ba80a9ae0 msgr2=0x7f8ba8141460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 shutdown_connections 2024-09-16T04:11:32.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.190+0000 7f8bacfed640 1 -- 172.21.15.171:0/3882021127 wait complete. 2024-09-16T04:11:32.201 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:32.201 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-16T04:11:32.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.280+0000 7fc74939f640 10 monclient: get_monmap_and_config 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.280+0000 7fc74939f640 10 monclient: build_initial_monmap 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.281+0000 7fc74939f640 10 monclient: monmap: 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:32.281767+0000 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:32.281767+0000 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:32.283 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:32.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.281+0000 7fc74939f640 1 Processor -- start 2024-09-16T04:11:32.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.281+0000 7fc74939f640 1 -- start start 2024-09-16T04:11:32.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.281+0000 7fc74939f640 10 monclient: init 2024-09-16T04:11:32.290 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient(hunting): picked mon.noname-a con 0x7fc74409fc90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient(hunting): picked mon.noname-b con 0x7fc74406c470 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7440a0070 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74409e760 con 0x7fc74409fc90 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7440a35a0 con 0x7fc7440a0070 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74406c1f0 con 0x7fc74406c470 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc74939f640 10 monclient(hunting): authenticate will time out at 1067.787448s 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc73bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc74406c470 legacy=0x7fc7440a2e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc748b9e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc7440a0070 0x7fc74413f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34034/0 (socket says 172.21.15.171:34034) 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc748b9e640 1 -- 172.21.15.171:0/3529631048 learned_addr learned my addr 172.21.15.171:0/3529631048 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:32.291 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819053505 0 0) 0x7fc7440a35a0 con 0x7fc7440a0070 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): my global_id is 6001 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc730003520 con 0x7fc7440a0070 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742682006 0 0) 0x7fc74409e760 con 0x7fc74409fc90 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 10 monclient(hunting): my global_id is 64440 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7440a35a0 con 0x7fc74409fc90 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.282+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4068041538 0 0) 0x7fc730003520 con 0x7fc7440a0070 2024-09-16T04:11:32.292 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 >> v1:172.21.15.148:6790/0 conn(0x7fc74406c470 legacy=0x7fc7440a2e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 >> v1:172.21.15.148:6789/0 conn(0x7fc74409fc90 legacy=0x7fc7440a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 monclient: found mon.noname-c 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74406d990 con 0x7fc7440a0070 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 5 monclient: authenticate success, global_id 6001 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc73c001990 con 0x7fc7440a0070 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:32.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: dump: 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc73c001c40 con 0x7fc7440a0070 2024-09-16T04:11:32.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 4 set_mon_vals no callback set 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 1 -- 172.21.15.171:0/3529631048 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc73c002010 con 0x7fc7440a0070 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc7437fe640 10 monclient: dump: 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:32.295 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 10 monclient: shutdown 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 1 -- 172.21.15.171:0/3529631048 >> v1:172.21.15.171:6789/0 conn(0x7fc7440a0070 legacy=0x7fc74413f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 1 -- 172.21.15.171:0/3529631048 shutdown_connections 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 1 -- 172.21.15.171:0/3529631048 >> 172.21.15.171:0/3529631048 conn(0x7fc7440a74e0 msgr2=0x7fc7440a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 1 -- 172.21.15.171:0/3529631048 shutdown_connections 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.283+0000 7fc74939f640 1 -- 172.21.15.171:0/3529631048 wait complete. 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.284+0000 7fc74939f640 10 monclient: build_initial_monmap 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.284+0000 7fc74939f640 10 monclient: monmap: 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:32.296 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:32.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.284+0000 7fc74939f640 1 Processor -- start 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.284+0000 7fc74939f640 1 -- start start 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.284+0000 7fc74939f640 10 monclient: init 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient(hunting): picked mon.noname-a con 0x7fc74409fc90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7440a0070 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7441d1630 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7440a2bb0 con 0x7fc74409fc90 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7441a8d40 con 0x7fc7440a0070 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7441a9020 con 0x7fc7441d1630 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient(hunting): authenticate will time out at 1067.790142s 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc748b9e640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc7441d1630 legacy=0x7fc7441d3a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:32.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc73bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc7440a0070 0x7fc7441cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34048/0 (socket says 172.21.15.171:34048) 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc73bfff640 1 -- 172.21.15.171:0/4266656653 learned_addr learned my addr 172.21.15.171:0/4266656653 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2911964620 0 0) 0x7fc7441a8d40 con 0x7fc7440a0070 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): my global_id is 6004 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc724003520 con 0x7fc7440a0070 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031482550 0 0) 0x7fc7440a35a0 con 0x7fc74409fc90 2024-09-16T04:11:32.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): my global_id is 64443 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7441a8d40 con 0x7fc74409fc90 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3268560182 0 0) 0x7fc724003520 con 0x7fc7440a0070 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7440a35a0 con 0x7fc7440a0070 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc72c004140 con 0x7fc7440a0070 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): dump: 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:32.300 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3726178612 0 0) 0x7fc7440a35a0 con 0x7fc7440a0070 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 >> v1:172.21.15.148:6790/0 conn(0x7fc7441d1630 legacy=0x7fc7441d3a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 >> v1:172.21.15.148:6789/0 conn(0x7fc74409fc90 legacy=0x7fc7441cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: _finish_hunting 0 2024-09-16T04:11:32.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 monclient: found mon.b 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7441cbf10 con 0x7fc7440a0070 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 5 monclient: authenticate success, global_id 6004 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient: _renew_subs 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7441a7b70 con 0x7fc7440a0070 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient: _renew_subs 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc744072420 con 0x7fc7440a0070 2024-09-16T04:11:32.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc72c004460 con 0x7fc7440a0070 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc72c004830 con 0x7fc7440a0070 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc742ffd640 4 set_mon_vals no callback set 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.285+0000 7fc7417fa640 10 monclient: dump: 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:32.303 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc72c0049d0 con 0x7fc7440a0070 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc74939f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc74939f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc74939f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc728005080 con 0x7fc7440a0070 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(191..191 src has 1..191) ==== 8779+0+0 (unknown 1606940562 0 0) 0x7fc72c048180 con 0x7fc7440a0070 2024-09-16T04:11:32.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.286+0000 7fc743fff640 10 monclient: get_auth_request con 0x7fc724038c10 auth_method 0 2024-09-16T04:11:32.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.291+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc72c015c50 con 0x7fc7440a0070 2024-09-16T04:11:32.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.291+0000 7fc7417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:32.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.291+0000 7fc7417fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:32.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.458+0000 7fc74939f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:32.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.458+0000 7fc74939f640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:32.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.458+0000 7fc74939f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:32.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:32.458+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fc728005370 con 0x7fc7440a0070 2024-09-16T04:11:33.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.085+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v192) ==== 166+0+0 (unknown 3634557683 0 0) 0x7fc72c015550 con 0x7fc7440a0070 2024-09-16T04:11:33.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.085+0000 7fc7417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:33.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.085+0000 7fc7417fa640 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-16T04:11:33.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.189+0000 7fc74939f640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:33.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.189+0000 7fc74939f640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:33.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.189+0000 7fc74939f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.189+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fc728005ab0 con 0x7fc7440a0070 2024-09-16T04:11:33.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.190+0000 7fc7417fa640 1 -- 172.21.15.171:0/4266656653 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v192) ==== 145+0+0 (unknown 530611223 0 0) 0x7fc72c01e100 con 0x7fc7440a0070 2024-09-16T04:11:33.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.190+0000 7fc7417fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:33.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.190+0000 7fc7417fa640 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-16T04:11:33.192 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h9' already exists 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 >> v1:172.21.15.171:6812/7327273 conn(0x7fc724038c10 legacy=0x7fc72403b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 10 monclient: shutdown 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 >> v1:172.21.15.171:6789/0 conn(0x7fc7440a0070 legacy=0x7fc7441cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 shutdown_connections 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 >> 172.21.15.171:0/4266656653 conn(0x7fc7440a74e0 msgr2=0x7fc7441beac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 shutdown_connections 2024-09-16T04:11:33.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.192+0000 7fc74939f640 1 -- 172.21.15.171:0/4266656653 wait complete. 2024-09-16T04:11:33.203 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:33.204 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-16T04:11:33.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.283+0000 7faa2bb31640 10 monclient: get_monmap_and_config 2024-09-16T04:11:33.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.283+0000 7faa2bb31640 10 monclient: build_initial_monmap 2024-09-16T04:11:33.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.283+0000 7faa2bb31640 10 monclient: monmap: 2024-09-16T04:11:33.285 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:33.285 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:33.283996+0000 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:33.283996+0000 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.283+0000 7faa2bb31640 1 Processor -- start 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 1 -- start start 2024-09-16T04:11:33.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient: init 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient(hunting): picked mon.noname-c con 0x7faa24133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient(hunting): picked mon.noname-a con 0x7faa241341f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient(hunting): picked mon.noname-b con 0x7faa2413a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2409da90 con 0x7faa241341f0 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2409dd70 con 0x7faa24133700 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2409e050 con 0x7faa2413a350 2024-09-16T04:11:33.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2bb31640 10 monclient(hunting): authenticate will time out at 1068.789606s 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa298cd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7faa24133700 0x7faa24133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34052/0 (socket says 172.21.15.171:34052) 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa298cd640 1 -- 172.21.15.171:0/4060672578 learned_addr learned my addr 172.21.15.171:0/4060672578 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa2a0ce640 1 -- 172.21.15.171:0/4060672578 >> v1:172.21.15.148:6790/0 conn(0x7faa2413a350 legacy=0x7faa2413c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64278249 0 0) 0x7faa2409dd70 con 0x7faa24133700 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa288cb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa288cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa288cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa288cb640 10 monclient(hunting): my global_id is 6007 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.284+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa10003520 con 0x7faa24133700 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4051623851 0 0) 0x7faa10003520 con 0x7faa24133700 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 >> v1:172.21.15.148:6790/0 conn(0x7faa2413a350 legacy=0x7faa2413c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:33.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 >> v1:172.21.15.148:6789/0 conn(0x7faa241341f0 legacy=0x7faa24138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 monclient: found mon.noname-c 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2409d4f0 con 0x7faa24133700 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: _finish_auth 0 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa20001610 con 0x7faa24133700 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 5 monclient: authenticate success, global_id 6007 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:33.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: dump: 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:33.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa200018f0 con 0x7faa24133700 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 1 -- 172.21.15.171:0/4060672578 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa20001cc0 con 0x7faa24133700 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 4 set_mon_vals no callback set 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa288cb640 10 monclient: dump: 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:33.297 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 10 monclient: shutdown 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 1 -- 172.21.15.171:0/4060672578 >> v1:172.21.15.171:6789/0 conn(0x7faa24133700 legacy=0x7faa24133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 1 -- 172.21.15.171:0/4060672578 shutdown_connections 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 1 -- 172.21.15.171:0/4060672578 >> 172.21.15.171:0/4060672578 conn(0x7faa240a5450 msgr2=0x7faa240a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 1 -- 172.21.15.171:0/4060672578 shutdown_connections 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.285+0000 7faa2bb31640 1 -- 172.21.15.171:0/4060672578 wait complete. 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.286+0000 7faa2bb31640 10 monclient: build_initial_monmap 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.286+0000 7faa2bb31640 10 monclient: monmap: 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:33.298 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.286+0000 7faa2bb31640 1 Processor -- start 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.286+0000 7faa2bb31640 1 -- start start 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.286+0000 7faa2bb31640 10 monclient: init 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:33.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient(hunting): picked mon.noname-b con 0x7faa24133700 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient(hunting): picked mon.noname-c con 0x7faa241341f0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient(hunting): picked mon.noname-a con 0x7faa2413a350 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa24139250 con 0x7faa2413a350 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa24139530 con 0x7faa24133700 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa240a46b0 con 0x7faa241341f0 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient(hunting): authenticate will time out at 1068.792346s 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa290cc640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7faa241341f0 legacy=0x7faa241cc700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa298cd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7faa24133700 0x7faa241c7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34058/0 (socket says 172.21.15.171:34058) 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa298cd640 1 -- 172.21.15.171:0/1252806777 learned_addr learned my addr 172.21.15.171:0/1252806777 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:33.300 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1754391530 0 0) 0x7faa24139530 con 0x7faa24133700 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): my global_id is 6010 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa08003520 con 0x7faa24133700 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3053379587 0 0) 0x7faa08003520 con 0x7faa24133700 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa24139530 con 0x7faa24133700 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa20001510 con 0x7faa24133700 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): dump: 2024-09-16T04:11:33.301 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490180833 0 0) 0x7faa24139250 con 0x7faa2413a350 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:33.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient(hunting): my global_id is 64449 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa08003520 con 0x7faa2413a350 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4172614113 0 0) 0x7faa24139530 con 0x7faa24133700 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 >> v1:172.21.15.148:6790/0 conn(0x7faa241341f0 legacy=0x7faa241cc700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 >> v1:172.21.15.148:6789/0 conn(0x7faa2413a350 legacy=0x7faa241cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 monclient: found mon.b 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24138f70 con 0x7faa24133700 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:33.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 5 monclient: authenticate success, global_id 6010 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient: _renew_subs 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.287+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa24137b20 con 0x7faa24133700 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa2bb31640 10 monclient: _renew_subs 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa2bb31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faa241365b0 con 0x7faa24133700 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa200018f0 con 0x7faa24133700 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa20004e60 con 0x7faa24133700 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:33.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1bfff640 4 set_mon_vals no callback set 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 10 monclient: dump: 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7faa20005000 con 0x7faa24133700 2024-09-16T04:11:33.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa2bb31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.288+0000 7faa2bb31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa2bb31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0c005080 con 0x7faa24133700 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(192..192 src has 1..192) ==== 8989+0+0 (unknown 1614585108 0 0) 0x7faa200483c0 con 0x7faa24133700 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa290cc640 10 monclient: get_auth_request con 0x7faa08038fc0 auth_method 0 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa290cc640 1 -- 172.21.15.171:0/1252806777 >> v1:172.21.15.171:6812/7327273 conn(0x7faa08038fc0 legacy=0x7faa0803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa290cc640 1 -- 172.21.15.171:0/1252806777 >> v1:172.21.15.171:6812/7327273 conn(0x7faa08038fc0 legacy=0x7faa0803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa290cc640 1 --1- 172.21.15.171:0/1252806777 >> v1:172.21.15.171:6812/7327273 conn(0x7faa08038fc0 0x7faa0803b480 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa290cc640 1 --1- 172.21.15.171:0/1252806777 >> v1:172.21.15.171:6812/7327273 conn(0x7faa08038fc0 0x7faa0803b480 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T04:11:33.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.289+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 >> v1:172.21.15.171:6812/7327273 conn(0x7faa08038fc0 legacy=0x7faa0803b480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:33.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.293+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7faa20015dc0 con 0x7faa24133700 2024-09-16T04:11:33.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.293+0000 7faa1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:33.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.293+0000 7faa1a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:33.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.462+0000 7faa2bb31640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:33.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.462+0000 7faa2bb31640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:33.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.462+0000 7faa2bb31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:33.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:33.462+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7faa0c005370 con 0x7faa24133700 2024-09-16T04:11:34.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.099+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v193) ==== 168+0+0 (unknown 3245680003 0 0) 0x7faa2001e270 con 0x7faa24133700 2024-09-16T04:11:34.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.099+0000 7faa1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:34.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.099+0000 7faa1a7fc640 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-16T04:11:34.203 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.201+0000 7faa2bb31640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:34.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.201+0000 7faa2bb31640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:34.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.201+0000 7faa2bb31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.201+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7faa0c005bf0 con 0x7faa24133700 2024-09-16T04:11:34.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.203+0000 7faa1a7fc640 1 -- 172.21.15.171:0/1252806777 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v193) ==== 147+0+0 (unknown 1370988209 0 0) 0x7faa20044ca0 con 0x7faa24133700 2024-09-16T04:11:34.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.203+0000 7faa1a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:34.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.203+0000 7faa1a7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-16T04:11:34.205 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h10' already exists 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 10 monclient: shutdown 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 >> v1:172.21.15.171:6789/0 conn(0x7faa24133700 legacy=0x7faa241c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 shutdown_connections 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 >> 172.21.15.171:0/1252806777 conn(0x7faa240a5450 msgr2=0x7faa240a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 shutdown_connections 2024-09-16T04:11:34.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.205+0000 7faa2bb31640 1 -- 172.21.15.171:0/1252806777 wait complete. 2024-09-16T04:11:34.216 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:34.216 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-16T04:11:34.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.294+0000 7f5e985c4640 10 monclient: get_monmap_and_config 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.294+0000 7f5e985c4640 10 monclient: build_initial_monmap 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.294+0000 7f5e985c4640 10 monclient: monmap: 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:34.295723+0000 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:34.295723+0000 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:34.297 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.295+0000 7f5e985c4640 1 Processor -- start 2024-09-16T04:11:34.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.295+0000 7f5e985c4640 1 -- start start 2024-09-16T04:11:34.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.295+0000 7f5e985c4640 10 monclient: init 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient(hunting): picked mon.noname-a con 0x7f5e90133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient(hunting): picked mon.noname-c con 0x7f5e901341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient(hunting): picked mon.noname-b con 0x7f5e9013a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e9009da90 con 0x7f5e90133700 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e9009dd70 con 0x7f5e901341f0 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e9009e050 con 0x7f5e9013a350 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 10 monclient(hunting): authenticate will time out at 1069.801266s 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e95b5f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5e901341f0 0x7f5e90138c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34074/0 (socket says 172.21.15.171:34074) 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e95b5f640 1 -- 172.21.15.171:0/2249389124 learned_addr learned my addr 172.21.15.171:0/2249389124 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:34.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e96b61640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f5e9013a350 legacy=0x7f5e9013c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1880966971 0 0) 0x7f5e9009dd70 con 0x7f5e901341f0 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): _negotiate 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): my global_id is 6013 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e7c003520 con 0x7f5e901341f0 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119275519 0 0) 0x7f5e9009da90 con 0x7f5e90133700 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): _negotiate 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:34.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient(hunting): my global_id is 64452 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e9009dd70 con 0x7f5e90133700 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2151870536 0 0) 0x7f5e7c003520 con 0x7f5e901341f0 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 >> v1:172.21.15.148:6790/0 conn(0x7f5e9013a350 legacy=0x7f5e9013c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 >> v1:172.21.15.148:6789/0 conn(0x7f5e90133700 legacy=0x7f5e90133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: _finish_hunting 0 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 monclient: found mon.noname-c 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9009d4f0 con 0x7f5e901341f0 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: _finish_auth 0 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: _check_auth_tickets 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:34.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 5 monclient: authenticate success, global_id 6013 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e80001610 con 0x7f5e901341f0 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e9535e640 10 monclient: dump: 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:34.311 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.296+0000 7f5e985c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e80001930 con 0x7f5e901341f0 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e9535e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e9535e640 1 -- 172.21.15.171:0/2249389124 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e80001d00 con 0x7f5e901341f0 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 4 set_mon_vals no callback set 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e9535e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e9535e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e9535e640 10 monclient: dump: 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:34.312 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 10 monclient: shutdown 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 1 -- 172.21.15.171:0/2249389124 >> v1:172.21.15.171:6789/0 conn(0x7f5e901341f0 legacy=0x7f5e90138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 1 -- 172.21.15.171:0/2249389124 shutdown_connections 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 1 -- 172.21.15.171:0/2249389124 >> 172.21.15.171:0/2249389124 conn(0x7f5e900a5450 msgr2=0x7f5e900a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 1 -- 172.21.15.171:0/2249389124 shutdown_connections 2024-09-16T04:11:34.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.297+0000 7f5e985c4640 1 -- 172.21.15.171:0/2249389124 wait complete. 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient: build_initial_monmap 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient: monmap: 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 1 Processor -- start 2024-09-16T04:11:34.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 1 -- start start 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient: init 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient(hunting): picked mon.noname-a con 0x7f5e90133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient(hunting): picked mon.noname-b con 0x7f5e901341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient(hunting): picked mon.noname-c con 0x7f5e9013a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e9006fe80 con 0x7f5e90133700 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e90120c20 con 0x7f5e901341f0 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e900a3ed0 con 0x7f5e9013a350 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e985c4640 10 monclient(hunting): authenticate will time out at 1069.803947s 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.298+0000 7f5e96b61640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f5e9013a350 legacy=0x7f5e901d3970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e95b5f640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5e901341f0 0x7f5e9006f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34078/0 (socket says 172.21.15.171:34078) 2024-09-16T04:11:34.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e95b5f640 1 -- 172.21.15.171:0/2584601891 learned_addr learned my addr 172.21.15.171:0/2584601891 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758812507 0 0) 0x7f5e90120c20 con 0x7f5e901341f0 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): my global_id is 6016 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e74003520 con 0x7f5e901341f0 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202663175 0 0) 0x7f5e9006fe80 con 0x7f5e90133700 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): my global_id is 64455 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e90120c20 con 0x7f5e90133700 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1854030394 0 0) 0x7f5e74003520 con 0x7f5e901341f0 2024-09-16T04:11:34.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e9006fe80 con 0x7f5e901341f0 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e800015d0 con 0x7f5e901341f0 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:34.317 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3829912667 0 0) 0x7f5e90120c20 con 0x7f5e90133700 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e74003520 con 0x7f5e90133700 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e8c0018b0 con 0x7f5e90133700 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.318 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 394044926 0 0) 0x7f5e9006fe80 con 0x7f5e901341f0 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 >> v1:172.21.15.148:6790/0 conn(0x7f5e9013a350 legacy=0x7f5e901d3970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 >> v1:172.21.15.148:6789/0 conn(0x7f5e90133700 legacy=0x7f5e90132210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 monclient: found mon.b 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e901333f0 con 0x7f5e901341f0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 5 monclient: authenticate success, global_id 6016 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 10 monclient: _renew_subs 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5e90120940 con 0x7f5e901341f0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 10 monclient: _renew_subs 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5e90132920 con 0x7f5e901341f0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e80001930 con 0x7f5e901341f0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e80004e60 con 0x7f5e901341f0 2024-09-16T04:11:34.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e94b5d640 4 set_mon_vals no callback set 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.299+0000 7f5e86ffd640 10 monclient: dump: 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:34.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5e80005000 con 0x7f5e901341f0 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e985c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e985c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e985c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e78005080 con 0x7f5e901341f0 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(193..193 src has 1..193) ==== 9202+0+0 (unknown 3877999198 0 0) 0x7f5e800484c0 con 0x7f5e901341f0 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.300+0000 7f5e96360640 10 monclient: get_auth_request con 0x7f5e74038f10 auth_method 0 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.305+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5e80015e90 con 0x7f5e901341f0 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.305+0000 7f5e86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:34.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.305+0000 7f5e86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:34.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.471+0000 7f5e985c4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:34.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.471+0000 7f5e985c4640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:34.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.471+0000 7f5e985c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:34.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:34.471+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f5e78005370 con 0x7f5e901341f0 2024-09-16T04:11:35.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.098+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v194) ==== 168+0+0 (unknown 223484147 0 0) 0x7f5e80015790 con 0x7f5e901341f0 2024-09-16T04:11:35.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.098+0000 7f5e86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.098+0000 7f5e86ffd640 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-16T04:11:35.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.199+0000 7f5e985c4640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.199+0000 7f5e985c4640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.199+0000 7f5e985c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.199+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f5e78005bd0 con 0x7f5e901341f0 2024-09-16T04:11:35.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.200+0000 7f5e86ffd640 1 -- 172.21.15.171:0/2584601891 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v194) ==== 147+0+0 (unknown 1934786741 0 0) 0x7f5e80013af0 con 0x7f5e901341f0 2024-09-16T04:11:35.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.200+0000 7f5e86ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.202 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.200+0000 7f5e86ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-16T04:11:35.202 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h11' already exists 2024-09-16T04:11:35.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 >> v1:172.21.15.171:6812/7327273 conn(0x7f5e74038f10 legacy=0x7f5e7403b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 10 monclient: shutdown 2024-09-16T04:11:35.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:35.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 >> v1:172.21.15.171:6789/0 conn(0x7f5e901341f0 legacy=0x7f5e9006f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 shutdown_connections 2024-09-16T04:11:35.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 >> 172.21.15.171:0/2584601891 conn(0x7f5e900a5450 msgr2=0x7f5e9009f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:35.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 shutdown_connections 2024-09-16T04:11:35.205 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.202+0000 7f5e985c4640 1 -- 172.21.15.171:0/2584601891 wait complete. 2024-09-16T04:11:35.213 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:35.213 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.293+0000 7f9375586640 10 monclient: get_monmap_and_config 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.293+0000 7f9375586640 10 monclient: build_initial_monmap 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.293+0000 7f9375586640 10 monclient: monmap: 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:35.294154+0000 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:35.294154+0000 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:35.295 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:35.296 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:35.296 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:35.296 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:35.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.293+0000 7f9375586640 1 Processor -- start 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 1 -- start start 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient: init 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient(hunting): picked mon.noname-c con 0x7f9370136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient(hunting): picked mon.noname-a con 0x7f9370137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient(hunting): picked mon.noname-b con 0x7f93701375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937009b2c0 con 0x7f9370137140 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937009b5a0 con 0x7f9370136d60 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93700a6640 con 0x7f93701375a0 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:35.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f9375586640 10 monclient(hunting): authenticate will time out at 1070.799622s 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936f7fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f93701375a0 legacy=0x7f937013f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9370136d60 0x7f937009bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34092/0 (socket says 172.21.15.171:34092) 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936effd640 1 -- 172.21.15.171:0/1350652644 learned_addr learned my addr 172.21.15.171:0/1350652644 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473031786 0 0) 0x7f937009b5a0 con 0x7f9370136d60 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936dffb640 10 monclient(hunting): my global_id is 6019 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.294+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9358003520 con 0x7f9370136d60 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3093204301 0 0) 0x7f937009b2c0 con 0x7f9370137140 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:35.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient(hunting): my global_id is 64458 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f937009b5a0 con 0x7f9370137140 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 571455618 0 0) 0x7f9358003520 con 0x7f9370136d60 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 >> v1:172.21.15.148:6790/0 conn(0x7f93701375a0 legacy=0x7f937013f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 >> v1:172.21.15.148:6789/0 conn(0x7f9370137140 legacy=0x7f937013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:35.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 monclient: found mon.noname-c 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9370143b20 con 0x7f9370136d60 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9360001610 con 0x7f9370136d60 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 5 monclient: authenticate success, global_id 6019 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:35.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: dump: 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.308 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93600018c0 con 0x7f9370136d60 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 1 -- 172.21.15.171:0/1350652644 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9360001c90 con 0x7f9370136d60 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:35.309 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 4 set_mon_vals no callback set 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f936dffb640 10 monclient: dump: 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:35.310 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 10 monclient: shutdown 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 1 -- 172.21.15.171:0/1350652644 >> v1:172.21.15.171:6789/0 conn(0x7f9370136d60 legacy=0x7f937009bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 1 -- 172.21.15.171:0/1350652644 shutdown_connections 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 1 -- 172.21.15.171:0/1350652644 >> 172.21.15.171:0/1350652644 conn(0x7f93700a9ae0 msgr2=0x7f93700a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 1 -- 172.21.15.171:0/1350652644 shutdown_connections 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.295+0000 7f9375586640 1 -- 172.21.15.171:0/1350652644 wait complete. 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.296+0000 7f9375586640 10 monclient: build_initial_monmap 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.296+0000 7f9375586640 10 monclient: monmap: 2024-09-16T04:11:35.311 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.296+0000 7f9375586640 1 Processor -- start 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.296+0000 7f9375586640 1 -- start start 2024-09-16T04:11:35.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.296+0000 7f9375586640 10 monclient: init 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient(hunting): picked mon.noname-b con 0x7f9370136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient(hunting): picked mon.noname-a con 0x7f9370137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient(hunting): picked mon.noname-c con 0x7f93701375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93700a03b0 con 0x7f9370137140 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937012e660 con 0x7f9370136d60 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f937012e940 con 0x7f93701375a0 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient(hunting): authenticate will time out at 1070.802343s 2024-09-16T04:11:35.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f936effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9370136d60 0x7f937012b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34102/0 (socket says 172.21.15.171:34102) 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f936effd640 1 -- 172.21.15.171:0/1475622542 learned_addr learned my addr 172.21.15.171:0/1475622542 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f936f7fe640 1 -- 172.21.15.171:0/1475622542 >> v1:172.21.15.148:6790/0 conn(0x7f93701375a0 legacy=0x7f93701d3a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3658172675 0 0) 0x7f937012e660 con 0x7f9370136d60 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): my global_id is 6022 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f935c003520 con 0x7f9370136d60 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267654042 0 0) 0x7f93700a03b0 con 0x7f9370137140 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:35.314 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): my global_id is 64461 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f937012e660 con 0x7f9370137140 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1106831808 0 0) 0x7f935c003520 con 0x7f9370136d60 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93700a03b0 con 0x7f9370136d60 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9360001c10 con 0x7f9370136d60 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): dump: 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:35.315 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3775668301 0 0) 0x7f93700a03b0 con 0x7f9370136d60 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 >> v1:172.21.15.148:6790/0 conn(0x7f93701375a0 legacy=0x7f93701d3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 >> v1:172.21.15.148:6789/0 conn(0x7f9370137140 legacy=0x7f937012b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 monclient: found mon.b 2024-09-16T04:11:35.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9370136730 con 0x7f9370136d60 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f93577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 5 monclient: authenticate success, global_id 6022 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient: _renew_subs 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.297+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f937012ca50 con 0x7f9370136d60 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f9375586640 10 monclient: _renew_subs 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f9375586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f937012bfd0 con 0x7f9370136d60 2024-09-16T04:11:35.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9360004c40 con 0x7f9370136d60 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9360004df0 con 0x7f9370136d60 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f936d7fa640 4 set_mon_vals no callback set 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 10 monclient: dump: 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:35.318 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.298+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9360010220 con 0x7f9370136d60 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.299+0000 7f9375586640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.299+0000 7f9375586640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.299+0000 7f9375586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.299+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9350005080 con 0x7f9370136d60 2024-09-16T04:11:35.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.299+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(194..194 src has 1..194) ==== 9415+0+0 (unknown 2092433982 0 0) 0x7f9360048570 con 0x7f9370136d60 2024-09-16T04:11:35.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.299+0000 7f936e7fc640 10 monclient: get_auth_request con 0x7f935c039200 auth_method 0 2024-09-16T04:11:35.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.303+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9360015e90 con 0x7f9370136d60 2024-09-16T04:11:35.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.303+0000 7f93577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:35.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.303+0000 7f93577fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:35.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.476+0000 7f9375586640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.476+0000 7f9375586640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.476+0000 7f9375586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.478 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.476+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9350005370 con 0x7f9370136d60 2024-09-16T04:11:35.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.824+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v195) ==== 168+0+0 (unknown 961556089 0 0) 0x7f936001d1d0 con 0x7f9370136d60 2024-09-16T04:11:35.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.824+0000 7f93577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.826 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.824+0000 7f93577fe640 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-16T04:11:35.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.927+0000 7f9375586640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.927+0000 7f9375586640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.927+0000 7f9375586640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:35.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.927+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f9350005ab0 con 0x7f9370136d60 2024-09-16T04:11:35.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.929+0000 7f93577fe640 1 -- 172.21.15.171:0/1475622542 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v195) ==== 147+0+0 (unknown 1188801899 0 0) 0x7f936001d410 con 0x7f9370136d60 2024-09-16T04:11:35.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.929+0000 7f93577fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:35.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.929+0000 7f93577fe640 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-16T04:11:35.931 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h12' already exists 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 >> v1:172.21.15.171:6812/7327273 conn(0x7f935c039200 legacy=0x7f935c03b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 10 monclient: shutdown 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 >> v1:172.21.15.171:6789/0 conn(0x7f9370136d60 legacy=0x7f937012b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 shutdown_connections 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 >> 172.21.15.171:0/1475622542 conn(0x7f93700a9ae0 msgr2=0x7f9370072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 shutdown_connections 2024-09-16T04:11:35.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:35.931+0000 7f9375586640 1 -- 172.21.15.171:0/1475622542 wait complete. 2024-09-16T04:11:35.942 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:35.942 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-16T04:11:36.024 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.022+0000 7fec00966640 10 monclient: get_monmap_and_config 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.022+0000 7fec00966640 10 monclient: build_initial_monmap 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.022+0000 7fec00966640 10 monclient: monmap: 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:36.023690+0000 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:36.023690+0000 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:36.025 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.023+0000 7fec00966640 1 Processor -- start 2024-09-16T04:11:36.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.023+0000 7fec00966640 1 -- start start 2024-09-16T04:11:36.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.023+0000 7fec00966640 10 monclient: init 2024-09-16T04:11:36.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient(hunting): picked mon.noname-b con 0x7febfc09b920 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient(hunting): picked mon.noname-a con 0x7febfc136da0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient(hunting): picked mon.noname-c con 0x7febfc13d320 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febfc073a20 con 0x7febfc136da0 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febfc099920 con 0x7febfc13d320 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febfc099c00 con 0x7febfc09b920 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient(hunting): authenticate will time out at 1071.529213s 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febfa59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7febfc09b920 legacy=0x7febfc09bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febfad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7febfc13d320 0x7febfc13f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34118/0 (socket says 172.21.15.171:34118) 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febfad9d640 1 -- 172.21.15.171:0/796089156 learned_addr learned my addr 172.21.15.171:0/796089156 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:36.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150711070 0 0) 0x7febfc099920 con 0x7febfc13d320 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): my global_id is 6025 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe8003520 con 0x7febfc13d320 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 664575178 0 0) 0x7febfc073a20 con 0x7febfc136da0 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient(hunting): my global_id is 64464 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febfc099920 con 0x7febfc136da0 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1517808739 0 0) 0x7febe8003520 con 0x7febfc13d320 2024-09-16T04:11:36.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 >> v1:172.21.15.148:6790/0 conn(0x7febfc09b920 legacy=0x7febfc09bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 >> v1:172.21.15.148:6789/0 conn(0x7febfc136da0 legacy=0x7febfc13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 monclient: found mon.noname-c 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc099ee0 con 0x7febfc13d320 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: _finish_auth 0 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febf4001610 con 0x7febfc13d320 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:36.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: dump: 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 5 monclient: authenticate success, global_id 6025 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:36.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febf40018c0 con 0x7febfc13d320 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 1 -- 172.21.15.171:0/796089156 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febf4001c90 con 0x7febfc13d320 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7fec00966640 4 set_mon_vals no callback set 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.024+0000 7febf959a640 10 monclient: dump: 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:36.037 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 10 monclient: shutdown 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 1 -- 172.21.15.171:0/796089156 >> v1:172.21.15.171:6789/0 conn(0x7febfc13d320 legacy=0x7febfc13f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 1 -- 172.21.15.171:0/796089156 shutdown_connections 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 1 -- 172.21.15.171:0/796089156 >> 172.21.15.171:0/796089156 conn(0x7febfc0a2150 msgr2=0x7febfc0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 1 -- 172.21.15.171:0/796089156 shutdown_connections 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.025+0000 7fec00966640 1 -- 172.21.15.171:0/796089156 wait complete. 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.026+0000 7fec00966640 10 monclient: build_initial_monmap 2024-09-16T04:11:36.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.026+0000 7fec00966640 10 monclient: monmap: 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.026+0000 7fec00966640 1 Processor -- start 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.026+0000 7fec00966640 1 -- start start 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.026+0000 7fec00966640 10 monclient: init 2024-09-16T04:11:36.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient(hunting): picked mon.noname-c con 0x7febfc09b920 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient(hunting): picked mon.noname-b con 0x7febfc136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient(hunting): picked mon.noname-a con 0x7febfc13d320 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febfc13b910 con 0x7febfc13d320 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febfc13c4b0 con 0x7febfc136da0 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febfc12c370 con 0x7febfc09b920 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febfa59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7febfc09b920 legacy=0x7febfc13aaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:36.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient(hunting): authenticate will time out at 1071.532158s 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf9d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7febfc136da0 0x7febfc13b200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34120/0 (socket says 172.21.15.171:34120) 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf9d9b640 1 -- 172.21.15.171:0/1892517719 learned_addr learned my addr 172.21.15.171:0/1892517719 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204416339 0 0) 0x7febfc13c4b0 con 0x7febfc136da0 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): my global_id is 6028 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febdc003520 con 0x7febfc136da0 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2016115908 0 0) 0x7febfc13b910 con 0x7febfc13d320 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:36.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): my global_id is 64467 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febfc13c4b0 con 0x7febfc13d320 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 408056592 0 0) 0x7febdc003520 con 0x7febfc136da0 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febfc13b910 con 0x7febfc136da0 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febe4001910 con 0x7febfc136da0 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.042 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4059358823 0 0) 0x7febfc13c4b0 con 0x7febfc13d320 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febdc003520 con 0x7febfc13d320 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febf4001a70 con 0x7febfc13d320 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1787073105 0 0) 0x7febfc13b910 con 0x7febfc136da0 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 >> v1:172.21.15.148:6790/0 conn(0x7febfc09b920 legacy=0x7febfc13aaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 >> v1:172.21.15.148:6789/0 conn(0x7febfc13d320 legacy=0x7febfc1d3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 monclient: found mon.b 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:36.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc12c090 con 0x7febfc136da0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 5 monclient: authenticate success, global_id 6028 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient: _renew_subs 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7febfc1d5070 con 0x7febfc136da0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient: _renew_subs 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7febfc1d5420 con 0x7febfc136da0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.027+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7febe4001ce0 con 0x7febfc136da0 2024-09-16T04:11:36.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7febe4003a60 con 0x7febfc136da0 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf8d99640 4 set_mon_vals no callback set 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf2ffd640 10 monclient: dump: 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:36.046 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7febe4003c00 con 0x7febfc136da0 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7fec00966640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7fec00966640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7fec00966640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.028+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe0005080 con 0x7febfc136da0 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.029+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(195..195 src has 1..195) ==== 9628+0+0 (unknown 3154340709 0 0) 0x7febe4048590 con 0x7febfc136da0 2024-09-16T04:11:36.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.029+0000 7febfa59c640 10 monclient: get_auth_request con 0x7febdc038ea0 auth_method 0 2024-09-16T04:11:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.033+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7febe4015c70 con 0x7febfc136da0 2024-09-16T04:11:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.033+0000 7febf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:36.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.033+0000 7febf2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:36.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.204+0000 7fec00966640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:36.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.204+0000 7fec00966640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:36.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.204+0000 7fec00966640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:36.206 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:36.204+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7febe0005370 con 0x7febfc136da0 2024-09-16T04:11:37.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.104+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v196) ==== 168+0+0 (unknown 2431653090 0 0) 0x7febe4019920 con 0x7febfc136da0 2024-09-16T04:11:37.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.104+0000 7febf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:37.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.104+0000 7febf2ffd640 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-16T04:11:37.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.210+0000 7fec00966640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.210+0000 7fec00966640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.210+0000 7fec00966640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.210+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7febe0005ab0 con 0x7febfc136da0 2024-09-16T04:11:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.212+0000 7febf2ffd640 1 -- 172.21.15.171:0/1892517719 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v196) ==== 147+0+0 (unknown 915979453 0 0) 0x7febe401e1e0 con 0x7febfc136da0 2024-09-16T04:11:37.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.212+0000 7febf2ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.212+0000 7febf2ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-16T04:11:37.214 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h13' already exists 2024-09-16T04:11:37.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 >> v1:172.21.15.171:6812/7327273 conn(0x7febdc038ea0 legacy=0x7febdc03b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:37.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 10 monclient: shutdown 2024-09-16T04:11:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 >> v1:172.21.15.171:6789/0 conn(0x7febfc136da0 legacy=0x7febfc13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 shutdown_connections 2024-09-16T04:11:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 >> 172.21.15.171:0/1892517719 conn(0x7febfc0a2150 msgr2=0x7febfc0a0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 shutdown_connections 2024-09-16T04:11:37.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.214+0000 7fec00966640 1 -- 172.21.15.171:0/1892517719 wait complete. 2024-09-16T04:11:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:37.225 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-16T04:11:37.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.305+0000 7f26ab047640 10 monclient: get_monmap_and_config 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.305+0000 7f26ab047640 10 monclient: build_initial_monmap 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.305+0000 7f26ab047640 10 monclient: monmap: 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:37.305941+0000 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:37.305941+0000 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:37.307 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:37.308 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.305+0000 7f26ab047640 1 Processor -- start 2024-09-16T04:11:37.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.305+0000 7f26ab047640 1 -- start start 2024-09-16T04:11:37.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.305+0000 7f26ab047640 10 monclient: init 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient(hunting): picked mon.noname-b con 0x7f26a4136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient(hunting): picked mon.noname-c con 0x7f26a4137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient(hunting): picked mon.noname-a con 0x7f26a41375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a409b2c0 con 0x7f26a41375a0 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a409b5a0 con 0x7f26a4137140 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a40a6640 con 0x7f26a4136d60 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26ab047640 10 monclient(hunting): authenticate will time out at 1072.811413s 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f26a4137140 0x7f26a413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36130/0 (socket says 172.21.15.171:36130) 2024-09-16T04:11:37.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a3fff640 1 -- 172.21.15.171:0/585505699 learned_addr learned my addr 172.21.15.171:0/585505699 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a8de3640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f26a4136d60 legacy=0x7f26a409bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421468760 0 0) 0x7f26a409b5a0 con 0x7f26a4137140 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): my global_id is 6031 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f268c003520 con 0x7f26a4137140 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232497984 0 0) 0x7f26a409b2c0 con 0x7f26a41375a0 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:37.317 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 10 monclient(hunting): my global_id is 64470 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26a409b5a0 con 0x7f26a41375a0 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.306+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3405589911 0 0) 0x7f268c003520 con 0x7f26a4137140 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 >> v1:172.21.15.148:6790/0 conn(0x7f26a4136d60 legacy=0x7f26a409bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 >> v1:172.21.15.148:6789/0 conn(0x7f26a41375a0 legacy=0x7f26a413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 monclient: found mon.noname-c 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a4143b20 con 0x7f26a4137140 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:37.318 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2698001610 con 0x7f26a4137140 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: dump: 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:37.319 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 5 monclient: authenticate success, global_id 6031 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26980018c0 con 0x7f26a4137140 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 1 -- 172.21.15.171:0/585505699 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2698001c90 con 0x7f26a4137140 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 4 set_mon_vals no callback set 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26a37fe640 10 monclient: dump: 2024-09-16T04:11:37.320 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 10 monclient: shutdown 2024-09-16T04:11:37.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 1 -- 172.21.15.171:0/585505699 >> v1:172.21.15.171:6789/0 conn(0x7f26a4137140 legacy=0x7f26a413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 1 -- 172.21.15.171:0/585505699 shutdown_connections 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 1 -- 172.21.15.171:0/585505699 >> 172.21.15.171:0/585505699 conn(0x7f26a40a9ae0 msgr2=0x7f26a40a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 1 -- 172.21.15.171:0/585505699 shutdown_connections 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.307+0000 7f26ab047640 1 -- 172.21.15.171:0/585505699 wait complete. 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.308+0000 7f26ab047640 10 monclient: build_initial_monmap 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.308+0000 7f26ab047640 10 monclient: monmap: 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:37.322 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.308+0000 7f26ab047640 1 Processor -- start 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.308+0000 7f26ab047640 1 -- start start 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.308+0000 7f26ab047640 10 monclient: init 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient(hunting): picked mon.noname-a con 0x7f26a4136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:37.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient(hunting): picked mon.noname-c con 0x7f26a4137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient(hunting): picked mon.noname-b con 0x7f26a41375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a41cc850 con 0x7f26a4136d60 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a4130a90 con 0x7f26a41375a0 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a4131640 con 0x7f26a4137140 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient(hunting): authenticate will time out at 1072.814205s 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a3fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f26a4137140 legacy=0x7f26a41cc140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a95e4640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f26a41375a0 0x7f26a41d3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36140/0 (socket says 172.21.15.171:36140) 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a95e4640 1 -- 172.21.15.171:0/3830831497 learned_addr learned my addr 172.21.15.171:0/3830831497 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113465537 0 0) 0x7f26a4130a90 con 0x7f26a41375a0 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:37.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): my global_id is 6034 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2690003520 con 0x7f26a41375a0 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367444413 0 0) 0x7f26a41cc850 con 0x7f26a4136d60 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): my global_id is 64473 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26a4130a90 con 0x7f26a4136d60 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 106786808 0 0) 0x7f2690003520 con 0x7f26a41375a0 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26a41cc850 con 0x7f26a41375a0 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f269c001c30 con 0x7f26a41375a0 2024-09-16T04:11:37.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): dump: 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:37.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2733722962 0 0) 0x7f26a4130a90 con 0x7f26a4136d60 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2690003520 con 0x7f26a4136d60 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26940040f0 con 0x7f26a4136d60 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient(hunting): dump: 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.327 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2959620385 0 0) 0x7f26a41cc850 con 0x7f26a41375a0 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 >> v1:172.21.15.148:6790/0 conn(0x7f26a4137140 legacy=0x7f26a41cc140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 >> v1:172.21.15.148:6789/0 conn(0x7f26a4136d60 legacy=0x7f26a41cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 monclient: found mon.b 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a4131920 con 0x7f26a41375a0 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:37.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 5 monclient: authenticate success, global_id 6034 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient: _renew_subs 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26a412fcf0 con 0x7f26a41375a0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient: _renew_subs 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.309+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f26a412f200 con 0x7f26a41375a0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f269c001fe0 con 0x7f26a41375a0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f269c003de0 con 0x7f26a41375a0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:37.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a2ffd640 4 set_mon_vals no callback set 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 10 monclient: dump: 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:37.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f269c004040 con 0x7f26a41375a0 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26ab047640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26ab047640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26ab047640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.310+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2684005080 con 0x7f26a41375a0 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.311+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(196..196 src has 1..196) ==== 9841+0+0 (unknown 705010232 0 0) 0x7f269c048980 con 0x7f26a41375a0 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.311+0000 7f26a8de3640 10 monclient: get_auth_request con 0x7f2690038ea0 auth_method 0 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.315+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f269c015f90 con 0x7f26a41375a0 2024-09-16T04:11:37.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.315+0000 7f26a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:37.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.315+0000 7f26a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:37.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.485+0000 7f26ab047640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:37.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.485+0000 7f26ab047640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:37.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.485+0000 7f26ab047640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:37.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:37.485+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f2684005370 con 0x7f26a41375a0 2024-09-16T04:11:38.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.117+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v197) ==== 168+0+0 (unknown 886260870 0 0) 0x7f269c019c40 con 0x7f26a41375a0 2024-09-16T04:11:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.117+0000 7f26a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:38.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.117+0000 7f26a17fa640 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-16T04:11:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.223+0000 7f26ab047640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.223+0000 7f26ab047640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.223+0000 7f26ab047640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.223+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f2684005bd0 con 0x7f26a41375a0 2024-09-16T04:11:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.224+0000 7f26a17fa640 1 -- 172.21.15.171:0/3830831497 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v197) ==== 147+0+0 (unknown 2138044677 0 0) 0x7f269c01e500 con 0x7f26a41375a0 2024-09-16T04:11:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.224+0000 7f26a17fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.224+0000 7f26a17fa640 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-16T04:11:38.226 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h14' already exists 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 >> v1:172.21.15.171:6812/7327273 conn(0x7f2690038ea0 legacy=0x7f269003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 10 monclient: shutdown 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 >> v1:172.21.15.171:6789/0 conn(0x7f26a41375a0 legacy=0x7f26a41d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 shutdown_connections 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 >> 172.21.15.171:0/3830831497 conn(0x7f26a40a9ae0 msgr2=0x7f26a4072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:38.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 shutdown_connections 2024-09-16T04:11:38.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.226+0000 7f26ab047640 1 -- 172.21.15.171:0/3830831497 wait complete. 2024-09-16T04:11:38.238 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:38.238 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.318+0000 7fbd41001640 10 monclient: get_monmap_and_config 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.318+0000 7fbd41001640 10 monclient: build_initial_monmap 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.318+0000 7fbd41001640 10 monclient: monmap: 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:38.319624+0000 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:38.319624+0000 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:38.321 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.319+0000 7fbd41001640 1 Processor -- start 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.319+0000 7fbd41001640 1 -- start start 2024-09-16T04:11:38.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.319+0000 7fbd41001640 10 monclient: init 2024-09-16T04:11:38.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.319+0000 7fbd41001640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:38.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.319+0000 7fbd41001640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:38.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.319+0000 7fbd41001640 10 monclient(hunting): picked mon.noname-c con 0x7fbd3c09ff20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 10 monclient(hunting): picked mon.noname-a con 0x7fbd3c09e6f0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 10 monclient(hunting): picked mon.noname-b con 0x7fbd3c13aa40 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3c09e4d0 con 0x7fbd3c09e6f0 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3c0a0a10 con 0x7fbd3c09ff20 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3c0a5b70 con 0x7fbd3c13aa40 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 10 monclient(hunting): authenticate will time out at 1073.825137s 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd3ad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbd3c09ff20 0x7fbd3c09f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36146/0 (socket says 172.21.15.171:36146) 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd3ad9d640 1 -- 172.21.15.171:0/1123685301 learned_addr learned my addr 172.21.15.171:0/1123685301 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd3b59e640 1 -- 172.21.15.171:0/1123685301 >> v1:172.21.15.148:6790/0 conn(0x7fbd3c13aa40 legacy=0x7fbd3c13ce30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:38.330 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450110762 0 0) 0x7fbd3c0a0a10 con 0x7fbd3c09ff20 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): my global_id is 6037 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd24003520 con 0x7fbd3c09ff20 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035286152 0 0) 0x7fbd30001110 con 0x7fbd3c09e6f0 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:38.331 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient(hunting): my global_id is 64476 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd3c0a0a10 con 0x7fbd3c09e6f0 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1064002880 0 0) 0x7fbd24003520 con 0x7fbd3c09ff20 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 >> v1:172.21.15.148:6790/0 conn(0x7fbd3c13aa40 legacy=0x7fbd3c13ce30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 >> v1:172.21.15.148:6789/0 conn(0x7fbd3c09e6f0 legacy=0x7fbd3c0a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 monclient: found mon.noname-c 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3c141580 con 0x7fbd3c09ff20 2024-09-16T04:11:38.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 5 monclient: authenticate success, global_id 6037 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd2c001610 con 0x7fbd3c09ff20 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: dump: 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.333 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd2c001870 con 0x7fbd3c09ff20 2024-09-16T04:11:38.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 1 -- 172.21.15.171:0/1123685301 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd2c001c40 con 0x7fbd3c09ff20 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd41001640 4 set_mon_vals no callback set 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.320+0000 7fbd39d9b640 10 monclient: dump: 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.337 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 10 monclient: shutdown 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 1 -- 172.21.15.171:0/1123685301 >> v1:172.21.15.171:6789/0 conn(0x7fbd3c09ff20 legacy=0x7fbd3c09f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 1 -- 172.21.15.171:0/1123685301 shutdown_connections 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 1 -- 172.21.15.171:0/1123685301 >> 172.21.15.171:0/1123685301 conn(0x7fbd3c0a5450 msgr2=0x7fbd3c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 1 -- 172.21.15.171:0/1123685301 shutdown_connections 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 1 -- 172.21.15.171:0/1123685301 wait complete. 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.321+0000 7fbd41001640 10 monclient: build_initial_monmap 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient: monmap: 2024-09-16T04:11:38.338 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 1 Processor -- start 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 1 -- start start 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient: init 2024-09-16T04:11:38.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient(hunting): picked mon.noname-c con 0x7fbd3c09ff20 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient(hunting): picked mon.noname-b con 0x7fbd3c13aa40 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 10 monclient(hunting): picked mon.noname-a con 0x7fbd3c13a240 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.322+0000 7fbd41001640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3c09b6c0 con 0x7fbd3c13a240 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3c09dc40 con 0x7fbd3c13aa40 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd3c136e70 con 0x7fbd3c09ff20 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 10 monclient(hunting): authenticate will time out at 1073.828031s 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd3ad9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fbd3c09ff20 legacy=0x7fbd3c139b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd3a59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbd3c13aa40 0x7fbd3c1d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36162/0 (socket says 172.21.15.171:36162) 2024-09-16T04:11:38.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd3a59c640 1 -- 172.21.15.171:0/2951278105 learned_addr learned my addr 172.21.15.171:0/2951278105 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432872484 0 0) 0x7fbd3c09dc40 con 0x7fbd3c13aa40 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): my global_id is 6040 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd28003520 con 0x7fbd3c13aa40 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024182990 0 0) 0x7fbd3c09b6c0 con 0x7fbd3c13a240 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): my global_id is 64479 2024-09-16T04:11:38.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd3c09dc40 con 0x7fbd3c13a240 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2106955984 0 0) 0x7fbd28003520 con 0x7fbd3c13aa40 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd3c09b6c0 con 0x7fbd3c13aa40 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd30001a80 con 0x7fbd3c13aa40 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): dump: 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.342 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3083131001 0 0) 0x7fbd3c09dc40 con 0x7fbd3c13a240 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd28003520 con 0x7fbd3c13a240 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd34004320 con 0x7fbd3c13a240 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient(hunting): dump: 2024-09-16T04:11:38.343 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1199894456 0 0) 0x7fbd3c09b6c0 con 0x7fbd3c13aa40 2024-09-16T04:11:38.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 >> v1:172.21.15.148:6790/0 conn(0x7fbd3c09ff20 legacy=0x7fbd3c139b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 >> v1:172.21.15.148:6789/0 conn(0x7fbd3c13a240 legacy=0x7fbd3c1d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 monclient: found mon.b 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3c137d60 con 0x7fbd3c13aa40 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:38.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 5 monclient: authenticate success, global_id 6040 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 10 monclient: _renew_subs 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3c09d580 con 0x7fbd3c13aa40 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 10 monclient: _renew_subs 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd3c1cb770 con 0x7fbd3c13aa40 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbd300038c0 con 0x7fbd3c13aa40 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbd30003c90 con 0x7fbd3c13aa40 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:38.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd3959a640 4 set_mon_vals no callback set 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.323+0000 7fbd237fe640 10 monclient: dump: 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbd30003e30 con 0x7fbd3c13aa40 2024-09-16T04:11:38.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd41001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd41001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd41001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd1c005080 con 0x7fbd3c13aa40 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(197..197 src has 1..197) ==== 10054+0+0 (unknown 3945451726 0 0) 0x7fbd30048760 con 0x7fbd3c13aa40 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.324+0000 7fbd3ad9d640 10 monclient: get_auth_request con 0x7fbd28038ef0 auth_method 0 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.329+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbd3c09e4d0 con 0x7fbd3c13aa40 2024-09-16T04:11:38.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.329+0000 7fbd237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:38.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.329+0000 7fbd237fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:38.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.500+0000 7fbd41001640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:38.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.500+0000 7fbd41001640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:38.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.500+0000 7fbd41001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:38.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:38.500+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fbd1c005370 con 0x7fbd3c13aa40 2024-09-16T04:11:39.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.123+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v198) ==== 168+0+0 (unknown 4176771574 0 0) 0x7fbd30015d20 con 0x7fbd3c13aa40 2024-09-16T04:11:39.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.123+0000 7fbd237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:39.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.123+0000 7fbd237fe640 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-16T04:11:39.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.228+0000 7fbd41001640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:39.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.228+0000 7fbd41001640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:39.233 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.228+0000 7fbd41001640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.228+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fbd1c005ab0 con 0x7fbd3c13aa40 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.229+0000 7fbd237fe640 1 -- 172.21.15.171:0/2951278105 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v198) ==== 147+0+0 (unknown 1569384193 0 0) 0x7fbd30015620 con 0x7fbd3c13aa40 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.229+0000 7fbd237fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.229+0000 7fbd237fe640 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h15' already exists 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 >> v1:172.21.15.171:6812/7327273 conn(0x7fbd28038ef0 legacy=0x7fbd2803b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 10 monclient: shutdown 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 >> v1:172.21.15.171:6789/0 conn(0x7fbd3c13aa40 legacy=0x7fbd3c1d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 shutdown_connections 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 >> 172.21.15.171:0/2951278105 conn(0x7fbd3c0a5450 msgr2=0x7fbd3c13eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 shutdown_connections 2024-09-16T04:11:39.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.231+0000 7fbd41001640 1 -- 172.21.15.171:0/2951278105 wait complete. 2024-09-16T04:11:39.242 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:39.242 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-16T04:11:39.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.321+0000 7f8150cb7640 10 monclient: get_monmap_and_config 2024-09-16T04:11:39.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.321+0000 7f8150cb7640 10 monclient: build_initial_monmap 2024-09-16T04:11:39.324 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.321+0000 7f8150cb7640 10 monclient: monmap: 2024-09-16T04:11:39.324 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:39.324 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:39.324 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:39.322384+0000 2024-09-16T04:11:39.324 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:39.322384+0000 2024-09-16T04:11:39.325 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:39.325 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:39.325 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:39.325 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:39.325 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:39.326 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 1 Processor -- start 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 1 -- start start 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient: init 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient(hunting): picked mon.noname-a con 0x7f814c09ff90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient(hunting): picked mon.noname-b con 0x7f814c0a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient(hunting): picked mon.noname-c con 0x7f814c0a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814c0a0f20 con 0x7f814c09ff90 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814c0a1200 con 0x7f814c0a0800 2024-09-16T04:11:39.334 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814c09bca0 con 0x7f814c0a0370 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8150cb7640 10 monclient(hunting): authenticate will time out at 1074.827917s 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.322+0000 7f8143fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f814c0a0370 legacy=0x7f814c139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f814ad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f814c0a0800 0x7f814c13d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36170/0 (socket says 172.21.15.171:36170) 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f814ad9d640 1 -- 172.21.15.171:0/4125698975 learned_addr learned my addr 172.21.15.171:0/4125698975 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044974229 0 0) 0x7f814c0a1200 con 0x7f814c0a0800 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): my global_id is 6043 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8138003520 con 0x7f814c0a0800 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187762920 0 0) 0x7f814c0a0f20 con 0x7f814c09ff90 2024-09-16T04:11:39.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient(hunting): my global_id is 64482 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f814c0a1200 con 0x7f814c09ff90 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 98179069 0 0) 0x7f8138003520 con 0x7f814c0a0800 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 >> v1:172.21.15.148:6790/0 conn(0x7f814c0a0370 legacy=0x7f814c139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 >> v1:172.21.15.148:6789/0 conn(0x7f814c09ff90 legacy=0x7f814c09f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 monclient: found mon.noname-c 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814c09e830 con 0x7f814c0a0800 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8144001990 con 0x7f814c0a0800 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8150cb7640 5 monclient: authenticate success, global_id 6043 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8150cb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: dump: 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.337 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8150cb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:39.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8144001cb0 con 0x7f814c0a0800 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 1 -- 172.21.15.171:0/4125698975 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8144002080 con 0x7f814c0a0800 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8150cb7640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8150cb7640 4 set_mon_vals no callback set 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.323+0000 7f8149d9b640 10 monclient: dump: 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:39.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 10 monclient: shutdown 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 1 -- 172.21.15.171:0/4125698975 >> v1:172.21.15.171:6789/0 conn(0x7f814c0a0800 legacy=0x7f814c13d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 1 -- 172.21.15.171:0/4125698975 shutdown_connections 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 1 -- 172.21.15.171:0/4125698975 >> 172.21.15.171:0/4125698975 conn(0x7f814c0a5450 msgr2=0x7f814c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:39.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 1 -- 172.21.15.171:0/4125698975 shutdown_connections 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 1 -- 172.21.15.171:0/4125698975 wait complete. 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 10 monclient: build_initial_monmap 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.324+0000 7f8150cb7640 10 monclient: monmap: 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:39.341 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 1 Processor -- start 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 1 -- start start 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient: init 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient(hunting): picked mon.noname-b con 0x7f814c09ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient(hunting): picked mon.noname-c con 0x7f814c0a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient(hunting): picked mon.noname-a con 0x7f814c0a0800 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814c11b660 con 0x7f814c0a0800 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814c13d330 con 0x7f814c09ff90 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f814c06e9d0 con 0x7f814c0a0370 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8150cb7640 10 monclient(hunting): authenticate will time out at 1074.830805s 2024-09-16T04:11:39.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f814a59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f814c09ff90 0x7f814c133fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36182/0 (socket says 172.21.15.171:36182) 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f814a59c640 1 -- 172.21.15.171:0/3665038398 learned_addr learned my addr 172.21.15.171:0/3665038398 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8143fff640 1 -- 172.21.15.171:0/3665038398 >> v1:172.21.15.148:6790/0 conn(0x7f814c0a0370 legacy=0x7f814c1cd910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704477496 0 0) 0x7f814c13d330 con 0x7f814c09ff90 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8142ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8142ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8142ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.325+0000 7f8142ffd640 10 monclient(hunting): my global_id is 6046 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f812c003520 con 0x7f814c09ff90 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672341382 0 0) 0x7f814c11b660 con 0x7f814c0a0800 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:39.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): my global_id is 64485 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f814c13d330 con 0x7f814c0a0800 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2008086188 0 0) 0x7f812c003520 con 0x7f814c09ff90 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f814c11b660 con 0x7f814c09ff90 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f813c001970 con 0x7f814c09ff90 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.344 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2786814826 0 0) 0x7f814c11b660 con 0x7f814c09ff90 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 >> v1:172.21.15.148:6790/0 conn(0x7f814c0a0370 legacy=0x7f814c1cd910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 >> v1:172.21.15.148:6789/0 conn(0x7f814c0a0800 legacy=0x7f814c1d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 monclient: found mon.b 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814c09c0b0 con 0x7f814c09ff90 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 5 monclient: authenticate success, global_id 6046 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 10 monclient: _renew_subs 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f814c13e500 con 0x7f814c09ff90 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 10 monclient: _renew_subs 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f814c1d2760 con 0x7f814c09ff90 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f813c001d20 con 0x7f814c09ff90 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:39.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f813c003a60 con 0x7f814c09ff90 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f814959a640 4 set_mon_vals no callback set 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.326+0000 7f8142ffd640 10 monclient: dump: 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:39.347 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f813c003c00 con 0x7f814c09ff90 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8150cb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8150cb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8150cb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8130005080 con 0x7f814c09ff90 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(198..198 src has 1..198) ==== 10267+0+0 (unknown 1010844747 0 0) 0x7f813c048840 con 0x7f814c09ff90 2024-09-16T04:11:39.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.327+0000 7f8143fff640 10 monclient: get_auth_request con 0x7f812c038be0 auth_method 0 2024-09-16T04:11:39.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.331+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f813c015ce0 con 0x7f814c09ff90 2024-09-16T04:11:39.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.331+0000 7f8142ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:39.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.331+0000 7f8142ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:39.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.499+0000 7f8150cb7640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:39.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.499+0000 7f8150cb7640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:39.501 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.499+0000 7f8150cb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:39.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:39.499+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f8130005370 con 0x7f814c09ff90 2024-09-16T04:11:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.129+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v199) ==== 168+0+0 (unknown 3438694268 0 0) 0x7f813c0155e0 con 0x7f814c09ff90 2024-09-16T04:11:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.129+0000 7f8142ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:40.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.129+0000 7f8142ffd640 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-16T04:11:40.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.232+0000 7f8150cb7640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:40.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.232+0000 7f8150cb7640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:40.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.232+0000 7f8150cb7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.234 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.232+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f8130005ab0 con 0x7f814c09ff90 2024-09-16T04:11:40.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.233+0000 7f8142ffd640 1 -- 172.21.15.171:0/3665038398 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v199) ==== 147+0+0 (unknown 1745079007 0 0) 0x7f813c019930 con 0x7f814c09ff90 2024-09-16T04:11:40.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.233+0000 7f8142ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:40.235 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.233+0000 7f8142ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-16T04:11:40.235 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h16' already exists 2024-09-16T04:11:40.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 >> v1:172.21.15.171:6812/7327273 conn(0x7f812c038be0 legacy=0x7f812c03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:40.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 10 monclient: shutdown 2024-09-16T04:11:40.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:40.237 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 >> v1:172.21.15.171:6789/0 conn(0x7f814c09ff90 legacy=0x7f814c133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:40.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 shutdown_connections 2024-09-16T04:11:40.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 >> 172.21.15.171:0/3665038398 conn(0x7f814c0a5450 msgr2=0x7f814c09f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:40.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 shutdown_connections 2024-09-16T04:11:40.238 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.235+0000 7f8150cb7640 1 -- 172.21.15.171:0/3665038398 wait complete. 2024-09-16T04:11:40.246 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:40.246 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-16T04:11:40.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.325+0000 7f709c931640 10 monclient: get_monmap_and_config 2024-09-16T04:11:40.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.325+0000 7f709c931640 10 monclient: build_initial_monmap 2024-09-16T04:11:40.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.325+0000 7f709c931640 10 monclient: monmap: 2024-09-16T04:11:40.327 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:40.327 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:40.327 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:40.326419+0000 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:40.326419+0000 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:40.328 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 1 Processor -- start 2024-09-16T04:11:40.335 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 1 -- start start 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient: init 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient(hunting): picked mon.noname-b con 0x7f709809b920 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient(hunting): picked mon.noname-a con 0x7f7098136da0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient(hunting): picked mon.noname-c con 0x7f709813d320 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7098073a20 con 0x7f7098136da0 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7098099920 con 0x7f709813d320 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7098099c00 con 0x7f709809b920 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709c931640 10 monclient(hunting): authenticate will time out at 1075.831727s 2024-09-16T04:11:40.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f709659c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f709809b920 legacy=0x7f709809bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f7096d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f709813d320 0x7f709813f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36198/0 (socket says 172.21.15.171:36198) 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.326+0000 7f7096d9d640 1 -- 172.21.15.171:0/1372117115 learned_addr learned my addr 172.21.15.171:0/1372117115 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1012171595 0 0) 0x7f7098099920 con 0x7f709813d320 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient(hunting): my global_id is 6049 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7084003520 con 0x7f709813d320 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1779675101 0 0) 0x7f7084003520 con 0x7f709813d320 2024-09-16T04:11:40.337 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 >> v1:172.21.15.148:6790/0 conn(0x7f709809b920 legacy=0x7f709809bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 >> v1:172.21.15.148:6789/0 conn(0x7f7098136da0 legacy=0x7f709813bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 monclient: found mon.noname-c 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7098099ee0 con 0x7f709813d320 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7090001990 con 0x7f709813d320 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 5 monclient: authenticate success, global_id 6049 2024-09-16T04:11:40.338 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: dump: 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:40.339 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7090001c80 con 0x7f709813d320 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 4 set_mon_vals no callback set 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 1 -- 172.21.15.171:0/1372117115 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7090002090 con 0x7f709813d320 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f7095d9b640 10 monclient: dump: 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.340 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 10 monclient: shutdown 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 1 -- 172.21.15.171:0/1372117115 >> v1:172.21.15.171:6789/0 conn(0x7f709813d320 legacy=0x7f709813f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 1 -- 172.21.15.171:0/1372117115 shutdown_connections 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 1 -- 172.21.15.171:0/1372117115 >> 172.21.15.171:0/1372117115 conn(0x7f70980a2150 msgr2=0x7f70980a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.327+0000 7f709c931640 1 -- 172.21.15.171:0/1372117115 shutdown_connections 2024-09-16T04:11:40.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.328+0000 7f709c931640 1 -- 172.21.15.171:0/1372117115 wait complete. 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.328+0000 7f709c931640 10 monclient: build_initial_monmap 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.328+0000 7f709c931640 10 monclient: monmap: 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 1 Processor -- start 2024-09-16T04:11:40.342 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 1 -- start start 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient: init 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient(hunting): picked mon.noname-c con 0x7f709809b920 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient(hunting): picked mon.noname-b con 0x7f7098136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient(hunting): picked mon.noname-a con 0x7f709813d320 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709813cb60 con 0x7f709813d320 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709813d090 con 0x7f7098136da0 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f709812bd10 con 0x7f709809b920 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709659c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f709809b920 legacy=0x7f70980987a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:40.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f709c931640 10 monclient(hunting): authenticate will time out at 1075.834725s 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f708dd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7098136da0 0x7f70981d0750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36200/0 (socket says 172.21.15.171:36200) 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.329+0000 7f708dd9b640 1 -- 172.21.15.171:0/2652706078 learned_addr learned my addr 172.21.15.171:0/2652706078 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2552271211 0 0) 0x7f709813d090 con 0x7f7098136da0 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): my global_id is 6052 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7078003550 con 0x7f7098136da0 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591149691 0 0) 0x7f7098099ee0 con 0x7f709813d320 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:40.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): my global_id is 64491 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f709813d090 con 0x7f709813d320 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3646228360 0 0) 0x7f7078003550 con 0x7f7098136da0 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7098099ee0 con 0x7f7098136da0 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7080001930 con 0x7f7098136da0 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:40.345 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2501001682 0 0) 0x7f709813d090 con 0x7f709813d320 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7078003550 con 0x7f709813d320 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f70900017b0 con 0x7f709813d320 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:40.346 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 800582748 0 0) 0x7f7098099ee0 con 0x7f7098136da0 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 >> v1:172.21.15.148:6790/0 conn(0x7f709809b920 legacy=0x7f70980987a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 >> v1:172.21.15.148:6789/0 conn(0x7f709813d320 legacy=0x7f70981d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:40.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 monclient: found mon.b 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709812bff0 con 0x7f7098136da0 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 5 monclient: authenticate success, global_id 6052 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 10 monclient: _renew_subs 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7098098eb0 con 0x7f7098136da0 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 10 monclient: _renew_subs 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f709813b860 con 0x7f7098136da0 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7080001d00 con 0x7f7098136da0 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7080003a60 con 0x7f7098136da0 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f708f7fe640 10 monclient: dump: 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:40.349 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.330+0000 7f709559a640 4 set_mon_vals no callback set 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7080003c00 con 0x7f7098136da0 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f709c931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f709c931640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f709c931640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f707c005080 con 0x7f7098136da0 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(199..199 src has 1..199) ==== 10736+0+0 (unknown 3710437420 0 0) 0x7f7080048a80 con 0x7f7098136da0 2024-09-16T04:11:40.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.331+0000 7f709659c640 10 monclient: get_auth_request con 0x7f7078038ed0 auth_method 0 2024-09-16T04:11:40.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.336+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7080015d10 con 0x7f7098136da0 2024-09-16T04:11:40.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.336+0000 7f708f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:40.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.336+0000 7f708f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:40.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.505+0000 7f709c931640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:40.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.505+0000 7f709c931640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:40.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.505+0000 7f709c931640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:40.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:40.505+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f707c005370 con 0x7f7098136da0 2024-09-16T04:11:41.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.129+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v200) ==== 168+0+0 (unknown 2925034033 0 0) 0x7f70800199c0 con 0x7f7098136da0 2024-09-16T04:11:41.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.129+0000 7f708f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:41.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.129+0000 7f708f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-16T04:11:41.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.238+0000 7f709c931640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:41.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.238+0000 7f709c931640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:41.240 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.238+0000 7f709c931640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.238+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f707c005ab0 con 0x7f7098136da0 2024-09-16T04:11:41.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.239+0000 7f708f7fe640 1 -- 172.21.15.171:0/2652706078 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v200) ==== 147+0+0 (unknown 3171730605 0 0) 0x7f708001e280 con 0x7f7098136da0 2024-09-16T04:11:41.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.239+0000 7f708f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:41.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.239+0000 7f708f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-16T04:11:41.241 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h17' already exists 2024-09-16T04:11:41.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 >> v1:172.21.15.171:6812/7327273 conn(0x7f7078038ed0 legacy=0x7f707803b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.243 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 10 monclient: shutdown 2024-09-16T04:11:41.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:41.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 >> v1:172.21.15.171:6789/0 conn(0x7f7098136da0 legacy=0x7f70981d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 shutdown_connections 2024-09-16T04:11:41.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 >> 172.21.15.171:0/2652706078 conn(0x7f70980a2150 msgr2=0x7f70980a4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:41.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 shutdown_connections 2024-09-16T04:11:41.244 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.241+0000 7f709c931640 1 -- 172.21.15.171:0/2652706078 wait complete. 2024-09-16T04:11:41.252 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:41.252 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-16T04:11:41.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.331+0000 7f0241a23640 10 monclient: get_monmap_and_config 2024-09-16T04:11:41.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.331+0000 7f0241a23640 10 monclient: build_initial_monmap 2024-09-16T04:11:41.333 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.331+0000 7f0241a23640 10 monclient: monmap: 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:41.332656+0000 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:41.332656+0000 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:41.334 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.332+0000 7f0241a23640 1 Processor -- start 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.332+0000 7f0241a23640 1 -- start start 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.332+0000 7f0241a23640 10 monclient: init 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.332+0000 7f0241a23640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient(hunting): picked mon.noname-b con 0x7f023c0a0ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient(hunting): picked mon.noname-c con 0x7f023c09f320 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:41.343 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient(hunting): picked mon.noname-a con 0x7f023c13d2c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c0737b0 con 0x7f023c13d2c0 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c06c6d0 con 0x7f023c09f320 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c06d980 con 0x7f023c0a0ae0 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient(hunting): authenticate will time out at 1076.838103s 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f023a7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f023c09f320 0x7f023c13bba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36210/0 (socket says 172.21.15.171:36210) 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f023affd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f023c0a0ae0 legacy=0x7f023c0a0ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f023a7fc640 1 -- 172.21.15.171:0/2038126500 learned_addr learned my addr 172.21.15.171:0/2038126500 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1029378864 0 0) 0x7f023c06c6d0 con 0x7f023c09f320 2024-09-16T04:11:41.344 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): my global_id is 6055 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0224003520 con 0x7f023c09f320 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436770384 0 0) 0x7f023c0737b0 con 0x7f023c13d2c0 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient(hunting): my global_id is 64494 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f023c06c6d0 con 0x7f023c13d2c0 2024-09-16T04:11:41.345 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 611617462 0 0) 0x7f0224003520 con 0x7f023c09f320 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 >> v1:172.21.15.148:6790/0 conn(0x7f023c0a0ae0 legacy=0x7f023c0a0ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 >> v1:172.21.15.148:6789/0 conn(0x7f023c13d2c0 legacy=0x7f023c13f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 monclient: found mon.noname-c 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c06c950 con 0x7f023c09f320 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:41.346 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0230001610 con 0x7f023c09f320 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 5 monclient: authenticate success, global_id 6055 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: dump: 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:41.347 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0230001930 con 0x7f023c09f320 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 1 -- 172.21.15.171:0/2038126500 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0230001d00 con 0x7f023c09f320 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0241a23640 4 set_mon_vals no callback set 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.333+0000 7f0239ffb640 10 monclient: dump: 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:41.348 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 10 monclient: shutdown 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 1 -- 172.21.15.171:0/2038126500 >> v1:172.21.15.171:6789/0 conn(0x7f023c09f320 legacy=0x7f023c13bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 1 -- 172.21.15.171:0/2038126500 shutdown_connections 2024-09-16T04:11:41.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 1 -- 172.21.15.171:0/2038126500 >> 172.21.15.171:0/2038126500 conn(0x7f023c0a77f0 msgr2=0x7f023c0a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 1 -- 172.21.15.171:0/2038126500 shutdown_connections 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 1 -- 172.21.15.171:0/2038126500 wait complete. 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 10 monclient: build_initial_monmap 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.334+0000 7f0241a23640 10 monclient: monmap: 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:41.350 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 1 Processor -- start 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 1 -- start start 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient: init 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient(hunting): picked mon.noname-c con 0x7f023c09f320 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient(hunting): picked mon.noname-a con 0x7f023c0a0ae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient(hunting): picked mon.noname-b con 0x7f023c13d2c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c127150 con 0x7f023c0a0ae0 2024-09-16T04:11:41.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c127430 con 0x7f023c13d2c0 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c0a2140 con 0x7f023c09f320 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f0241a23640 10 monclient(hunting): authenticate will time out at 1076.840843s 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f023affd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f023c09f320 legacy=0x7f023c1cb890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f023b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f023c13d2c0 0x7f023c1d4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36212/0 (socket says 172.21.15.171:36212) 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.335+0000 7f023b7fe640 1 -- 172.21.15.171:0/2519220501 learned_addr learned my addr 172.21.15.171:0/2519220501 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138349096 0 0) 0x7f023c127430 con 0x7f023c13d2c0 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): my global_id is 6058 2024-09-16T04:11:41.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0228003520 con 0x7f023c13d2c0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026564982 0 0) 0x7f023c127150 con 0x7f023c0a0ae0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): my global_id is 64497 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f023c127430 con 0x7f023c0a0ae0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3971332546 0 0) 0x7f0228003520 con 0x7f023c13d2c0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f023c127150 con 0x7f023c13d2c0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0234001c50 con 0x7f023c13d2c0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): dump: 2024-09-16T04:11:41.353 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2585994049 0 0) 0x7f023c127150 con 0x7f023c13d2c0 2024-09-16T04:11:41.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 >> v1:172.21.15.148:6790/0 conn(0x7f023c09f320 legacy=0x7f023c1cb890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 >> v1:172.21.15.148:6789/0 conn(0x7f023c0a0ae0 legacy=0x7f023c1d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 monclient: found mon.b 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c0a1c30 con 0x7f023c13d2c0 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 5 monclient: authenticate success, global_id 6058 2024-09-16T04:11:41.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 10 monclient: _renew_subs 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f023c09e1b0 con 0x7f023c13d2c0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 10 monclient: _renew_subs 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f023c125a20 con 0x7f023c13d2c0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0234002020 con 0x7f023c13d2c0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0234003de0 con 0x7f023c13d2c0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f02397fa640 4 set_mon_vals no callback set 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.336+0000 7f0223fff640 10 monclient: dump: 2024-09-16T04:11:41.356 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0234004000 con 0x7f023c13d2c0 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f0241a23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f0241a23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:41.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f0241a23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021c005080 con 0x7f023c13d2c0 2024-09-16T04:11:41.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(200..200 src has 1..200) ==== 10949+0+0 (unknown 99053967 0 0) 0x7f0234048eb0 con 0x7f023c13d2c0 2024-09-16T04:11:41.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.337+0000 7f023affd640 10 monclient: get_auth_request con 0x7f0228038b90 auth_method 0 2024-09-16T04:11:41.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.342+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f02340160b0 con 0x7f023c13d2c0 2024-09-16T04:11:41.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.342+0000 7f0223fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:41.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.342+0000 7f0223fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:41.511 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.509+0000 7f0241a23640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:41.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.509+0000 7f0241a23640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:41.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.509+0000 7f0241a23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:41.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:41.509+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f021c005370 con 0x7f023c13d2c0 2024-09-16T04:11:42.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.139+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v201) ==== 168+0+0 (unknown 802525560 0 0) 0x7f02340159b0 con 0x7f023c13d2c0 2024-09-16T04:11:42.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.139+0000 7f0223fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:42.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.139+0000 7f0223fff640 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-16T04:11:42.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.245+0000 7f0241a23640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:42.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.245+0000 7f0241a23640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:42.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.245+0000 7f0241a23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.247 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.245+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f021c005ab0 con 0x7f023c13d2c0 2024-09-16T04:11:42.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.246+0000 7f0223fff640 1 -- 172.21.15.171:0/2519220501 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v201) ==== 147+0+0 (unknown 201806297 0 0) 0x7f0234019d00 con 0x7f023c13d2c0 2024-09-16T04:11:42.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.246+0000 7f0223fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:42.248 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.246+0000 7f0223fff640 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-16T04:11:42.248 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h18' already exists 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 >> v1:172.21.15.171:6812/7327273 conn(0x7f0228038b90 legacy=0x7f022803b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 10 monclient: shutdown 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 >> v1:172.21.15.171:6789/0 conn(0x7f023c13d2c0 legacy=0x7f023c1d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 shutdown_connections 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 >> 172.21.15.171:0/2519220501 conn(0x7f023c0a77f0 msgr2=0x7f023c072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 shutdown_connections 2024-09-16T04:11:42.250 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.248+0000 7f0241a23640 1 -- 172.21.15.171:0/2519220501 wait complete. 2024-09-16T04:11:42.259 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:42.259 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-16T04:11:42.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.337+0000 7f88bd480640 10 monclient: get_monmap_and_config 2024-09-16T04:11:42.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.337+0000 7f88bd480640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.339 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.337+0000 7f88bd480640 10 monclient: monmap: 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:42.338591+0000 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:42.338591+0000 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:42.340 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:42.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.338+0000 7f88bd480640 1 Processor -- start 2024-09-16T04:11:42.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.338+0000 7f88bd480640 1 -- start start 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.338+0000 7f88bd480640 10 monclient: init 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.338+0000 7f88bd480640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.338+0000 7f88bd480640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.338+0000 7f88bd480640 10 monclient(hunting): picked mon.noname-b con 0x7f88b8136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 10 monclient(hunting): picked mon.noname-c con 0x7f88b8137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 10 monclient(hunting): picked mon.noname-a con 0x7f88b81375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b809b2c0 con 0x7f88b81375a0 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b809b5a0 con 0x7f88b8137140 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b80a6640 con 0x7f88b8136d60 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 10 monclient(hunting): authenticate will time out at 1077.844057s 2024-09-16T04:11:42.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88affff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f88b8137140 0x7f88b813bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36228/0 (socket says 172.21.15.171:36228) 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b6ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f88b8136d60 legacy=0x7f88b809bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88affff640 1 -- 172.21.15.171:0/2783319098 learned_addr learned my addr 172.21.15.171:0/2783319098 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 468342932 0 0) 0x7f88b809b5a0 con 0x7f88b8137140 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient(hunting): my global_id is 6061 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88a4003520 con 0x7f88b8137140 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 406569342 0 0) 0x7f88a4003520 con 0x7f88b8137140 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 >> v1:172.21.15.148:6790/0 conn(0x7f88b8136d60 legacy=0x7f88b809bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 >> v1:172.21.15.148:6789/0 conn(0x7f88b81375a0 legacy=0x7f88b813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 monclient: found mon.noname-c 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b8143b20 con 0x7f88b8137140 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88a0001610 con 0x7f88b8137140 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:42.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: dump: 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 5 monclient: authenticate success, global_id 6061 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:42.351 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88a0001900 con 0x7f88b8137140 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 1 -- 172.21.15.171:0/2783319098 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88a0001d10 con 0x7f88b8137140 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88bd480640 4 set_mon_vals no callback set 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.339+0000 7f88b67fc640 10 monclient: dump: 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.352 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 10 monclient: shutdown 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 1 -- 172.21.15.171:0/2783319098 >> v1:172.21.15.171:6789/0 conn(0x7f88b8137140 legacy=0x7f88b813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 1 -- 172.21.15.171:0/2783319098 shutdown_connections 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 1 -- 172.21.15.171:0/2783319098 >> 172.21.15.171:0/2783319098 conn(0x7f88b80a9ae0 msgr2=0x7f88b80a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 1 -- 172.21.15.171:0/2783319098 shutdown_connections 2024-09-16T04:11:42.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 1 -- 172.21.15.171:0/2783319098 wait complete. 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.340+0000 7f88bd480640 10 monclient: monmap: 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 1 Processor -- start 2024-09-16T04:11:42.354 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 1 -- start start 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient: init 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient(hunting): picked mon.noname-c con 0x7f88b8136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient(hunting): picked mon.noname-a con 0x7f88b8137140 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient(hunting): picked mon.noname-b con 0x7f88b81375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b8135d30 con 0x7f88b8137140 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b8131cf0 con 0x7f88b81375a0 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88b8131fd0 con 0x7f88b8136d60 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88bd480640 10 monclient(hunting): authenticate will time out at 1077.846804s 2024-09-16T04:11:42.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88b6ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f88b8136d60 legacy=0x7f88b812e460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88b77fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f88b81375a0 0x7f88b81d3a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36238/0 (socket says 172.21.15.171:36238) 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.341+0000 7f88b77fe640 1 -- 172.21.15.171:0/2801754524 learned_addr learned my addr 172.21.15.171:0/2801754524 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1987595553 0 0) 0x7f88b8131cf0 con 0x7f88b81375a0 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): my global_id is 6064 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8898003520 con 0x7f88b81375a0 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150574942 0 0) 0x7f88b8135d30 con 0x7f88b8137140 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): my global_id is 64503 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88b8131cf0 con 0x7f88b8137140 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3722890600 0 0) 0x7f8898003520 con 0x7f88b81375a0 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88b8135d30 con 0x7f88b81375a0 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88b0001c70 con 0x7f88b81375a0 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.357 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1681247114 0 0) 0x7f88b8135d30 con 0x7f88b81375a0 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 >> v1:172.21.15.148:6790/0 conn(0x7f88b8136d60 legacy=0x7f88b812e460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 >> v1:172.21.15.148:6789/0 conn(0x7f88b8137140 legacy=0x7f88b812eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 monclient: found mon.b 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b8136680 con 0x7f88b81375a0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 5 monclient: authenticate success, global_id 6064 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 10 monclient: _renew_subs 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88b812f280 con 0x7f88b81375a0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 10 monclient: _renew_subs 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f88b81d51a0 con 0x7f88b81375a0 2024-09-16T04:11:42.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88b0002040 con 0x7f88b81375a0 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88b0003de0 con 0x7f88b81375a0 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88af7fe640 10 monclient: dump: 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:42.360 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.342+0000 7f88b5ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f88b0004000 con 0x7f88b81375a0 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88bd480640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88bd480640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88bd480640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889c005080 con 0x7f88b81375a0 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(201..201 src has 1..201) ==== 11162+0+0 (unknown 4045219656 0 0) 0x7f88b0048fa0 con 0x7f88b81375a0 2024-09-16T04:11:42.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.343+0000 7f88b6ffd640 10 monclient: get_auth_request con 0x7f8898038b90 auth_method 0 2024-09-16T04:11:42.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.348+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f88b0016140 con 0x7f88b81375a0 2024-09-16T04:11:42.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.348+0000 7f88af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.348+0000 7f88af7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:42.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.514+0000 7f88bd480640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:42.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.514+0000 7f88bd480640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:42.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.514+0000 7f88bd480640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:42.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:42.514+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f889c005370 con 0x7f88b81375a0 2024-09-16T04:11:43.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.147+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v202) ==== 168+0+0 (unknown 3824331784 0 0) 0x7f88b0015a40 con 0x7f88b81375a0 2024-09-16T04:11:43.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.147+0000 7f88af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:43.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.147+0000 7f88af7fe640 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-16T04:11:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.251+0000 7f88bd480640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.251+0000 7f88bd480640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.251+0000 7f88bd480640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.251+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f889c005bd0 con 0x7f88b81375a0 2024-09-16T04:11:43.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.252+0000 7f88af7fe640 1 -- 172.21.15.171:0/2801754524 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v202) ==== 147+0+0 (unknown 786583517 0 0) 0x7f88b001e5d0 con 0x7f88b81375a0 2024-09-16T04:11:43.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.252+0000 7f88af7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:43.254 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.252+0000 7f88af7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-16T04:11:43.254 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h19' already exists 2024-09-16T04:11:43.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 >> v1:172.21.15.171:6812/7327273 conn(0x7f8898038b90 legacy=0x7f889803b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 10 monclient: shutdown 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 >> v1:172.21.15.171:6789/0 conn(0x7f88b81375a0 legacy=0x7f88b81d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 shutdown_connections 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 >> 172.21.15.171:0/2801754524 conn(0x7f88b80a9ae0 msgr2=0x7f88b8072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 shutdown_connections 2024-09-16T04:11:43.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.254+0000 7f88bd480640 1 -- 172.21.15.171:0/2801754524 wait complete. 2024-09-16T04:11:43.266 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:43.266 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-16T04:11:43.347 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.345+0000 7fe9e7023640 10 monclient: get_monmap_and_config 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.345+0000 7fe9e7023640 10 monclient: build_initial_monmap 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.345+0000 7fe9e7023640 10 monclient: monmap: 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:43.346702+0000 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:43.346702+0000 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:43.348 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:43.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.346+0000 7fe9e7023640 1 Processor -- start 2024-09-16T04:11:43.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.346+0000 7fe9e7023640 1 -- start start 2024-09-16T04:11:43.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.346+0000 7fe9e7023640 10 monclient: init 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e009ff90 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e00a0370 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e00a0800 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e00a0f20 con 0x7fe9e00a0370 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e00a1200 con 0x7fe9e009ff90 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e009bca0 con 0x7fe9e00a0800 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient(hunting): authenticate will time out at 1078.852256s 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e55c0640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe9e00a0800 legacy=0x7fe9e013d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:43.356 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e4dbf640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe9e009ff90 0x7fe9e009f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36246/0 (socket says 172.21.15.171:36246) 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e4dbf640 1 -- 172.21.15.171:0/1439407437 learned_addr learned my addr 172.21.15.171:0/1439407437 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3321718556 0 0) 0x7fe9e00a1200 con 0x7fe9e009ff90 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): my global_id is 6067 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9c8003520 con 0x7fe9e009ff90 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614732388 0 0) 0x7fe9e00a0f20 con 0x7fe9e00a0370 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient(hunting): my global_id is 64506 2024-09-16T04:11:43.357 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9e00a1200 con 0x7fe9e00a0370 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1721042994 0 0) 0x7fe9c8003520 con 0x7fe9e009ff90 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 >> v1:172.21.15.148:6790/0 conn(0x7fe9e00a0800 legacy=0x7fe9e013d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 >> v1:172.21.15.148:6789/0 conn(0x7fe9e00a0370 legacy=0x7fe9e0139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 monclient: found mon.noname-c 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e009e830 con 0x7fe9e009ff90 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9d0001590 con 0x7fe9e009ff90 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 5 monclient: authenticate success, global_id 6067 2024-09-16T04:11:43.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: dump: 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:43.359 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9d00017f0 con 0x7fe9e009ff90 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 1 -- 172.21.15.171:0/1439407437 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9d0001bc0 con 0x7fe9e009ff90 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9e7023640 4 set_mon_vals no callback set 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.347+0000 7fe9df7fe640 10 monclient: dump: 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.360 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 10 monclient: shutdown 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 1 -- 172.21.15.171:0/1439407437 >> v1:172.21.15.171:6789/0 conn(0x7fe9e009ff90 legacy=0x7fe9e009f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 1 -- 172.21.15.171:0/1439407437 shutdown_connections 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 1 -- 172.21.15.171:0/1439407437 >> 172.21.15.171:0/1439407437 conn(0x7fe9e00a5450 msgr2=0x7fe9e00a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 1 -- 172.21.15.171:0/1439407437 shutdown_connections 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 1 -- 172.21.15.171:0/1439407437 wait complete. 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 10 monclient: build_initial_monmap 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.348+0000 7fe9e7023640 10 monclient: monmap: 2024-09-16T04:11:43.361 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 1 Processor -- start 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 1 -- start start 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient: init 2024-09-16T04:11:43.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient(hunting): picked mon.noname-a con 0x7fe9e009ff90 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient(hunting): picked mon.noname-c con 0x7fe9e00a0370 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient(hunting): picked mon.noname-b con 0x7fe9e00a0800 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e011fda0 con 0x7fe9e009ff90 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e013cf30 con 0x7fe9e00a0800 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e013ccb0 con 0x7fe9e00a0370 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.349+0000 7fe9e7023640 10 monclient(hunting): authenticate will time out at 1078.854978s 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe9e00a0370 legacy=0x7fe9e01d0b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e55c0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe9e00a0800 0x7fe9e01d4280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36252/0 (socket says 172.21.15.171:36252) 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e55c0640 1 -- 172.21.15.171:0/1488423816 learned_addr learned my addr 172.21.15.171:0/1488423816 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:43.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085518349 0 0) 0x7fe9e013cf30 con 0x7fe9e00a0800 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): my global_id is 6070 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9cc003520 con 0x7fe9e00a0800 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569806365 0 0) 0x7fe9e011fda0 con 0x7fe9e009ff90 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): my global_id is 64509 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9e013cf30 con 0x7fe9e009ff90 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2720376273 0 0) 0x7fe9cc003520 con 0x7fe9e00a0800 2024-09-16T04:11:43.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9e011fda0 con 0x7fe9e00a0800 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9d80044d0 con 0x7fe9e00a0800 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): dump: 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:43.365 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2481353459 0 0) 0x7fe9e011fda0 con 0x7fe9e00a0800 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 >> v1:172.21.15.148:6790/0 conn(0x7fe9e00a0370 legacy=0x7fe9e01d0b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 >> v1:172.21.15.148:6789/0 conn(0x7fe9e009ff90 legacy=0x7fe9e01cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 monclient: found mon.b 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e009c830 con 0x7fe9e00a0800 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:43.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 5 monclient: authenticate success, global_id 6070 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 10 monclient: _renew_subs 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e011fac0 con 0x7fe9e00a0800 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 10 monclient: _renew_subs 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9e013b740 con 0x7fe9e00a0800 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9d8004780 con 0x7fe9e00a0800 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9d8004b50 con 0x7fe9e00a0800 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9deffd640 4 set_mon_vals no callback set 2024-09-16T04:11:43.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.350+0000 7fe9dd7fa640 10 monclient: dump: 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe9d8004cf0 con 0x7fe9e00a0800 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9e7023640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9e7023640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:43.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9e7023640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c0005080 con 0x7fe9e00a0800 2024-09-16T04:11:43.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(202..202 src has 1..202) ==== 11375+0+0 (unknown 2549810853 0 0) 0x7fe9d8048f60 con 0x7fe9e00a0800 2024-09-16T04:11:43.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.351+0000 7fe9e4dbf640 10 monclient: get_auth_request con 0x7fe9cc038fb0 auth_method 0 2024-09-16T04:11:43.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.356+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe9d8015fb0 con 0x7fe9e00a0800 2024-09-16T04:11:43.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.356+0000 7fe9dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:43.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.356+0000 7fe9dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:43.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.524+0000 7fe9e7023640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:43.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.524+0000 7fe9e7023640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:43.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.524+0000 7fe9e7023640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:43.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:43.524+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fe9c0005370 con 0x7fe9e00a0800 2024-09-16T04:11:44.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.152+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v203) ==== 168+0+0 (unknown 3476003295 0 0) 0x7fe9d80158b0 con 0x7fe9e00a0800 2024-09-16T04:11:44.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.152+0000 7fe9dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:44.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.152+0000 7fe9dd7fa640 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-16T04:11:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.254+0000 7fe9e7023640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.254+0000 7fe9e7023640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.254+0000 7fe9e7023640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.256 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.254+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fe9c0005ab0 con 0x7fe9e00a0800 2024-09-16T04:11:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.256+0000 7fe9dd7fa640 1 -- 172.21.15.171:0/1488423816 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v203) ==== 147+0+0 (unknown 1830156877 0 0) 0x7fe9d8019c00 con 0x7fe9e00a0800 2024-09-16T04:11:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.256+0000 7fe9dd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:44.257 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.256+0000 7fe9dd7fa640 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-16T04:11:44.258 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h20' already exists 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 >> v1:172.21.15.171:6812/7327273 conn(0x7fe9cc038fb0 legacy=0x7fe9cc03b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 10 monclient: shutdown 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 >> v1:172.21.15.171:6789/0 conn(0x7fe9e00a0800 legacy=0x7fe9e01d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 shutdown_connections 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 >> 172.21.15.171:0/1488423816 conn(0x7fe9e00a5450 msgr2=0x7fe9e0138f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 shutdown_connections 2024-09-16T04:11:44.260 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.258+0000 7fe9e7023640 1 -- 172.21.15.171:0/1488423816 wait complete. 2024-09-16T04:11:44.269 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:44.269 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.348+0000 7fab94cd0640 10 monclient: get_monmap_and_config 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.348+0000 7fab94cd0640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.348+0000 7fab94cd0640 10 monclient: monmap: 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:44.349394+0000 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:44.349394+0000 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:44.361 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 1 Processor -- start 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 1 -- start start 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient: init 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient(hunting): picked mon.noname-b con 0x7fab9009f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient(hunting): picked mon.noname-a con 0x7fab9009fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient(hunting): picked mon.noname-c con 0x7fab901395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab900a1490 con 0x7fab9009fba0 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab9009d1b0 con 0x7fab901395d0 2024-09-16T04:11:44.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab9009d490 con 0x7fab9009f7c0 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.349+0000 7fab94cd0640 10 monclient(hunting): authenticate will time out at 1079.854942s 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8e59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fab9009f7c0 legacy=0x7fab9009de70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8ed9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab901395d0 0x7fab9013b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36254/0 (socket says 172.21.15.171:36254) 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8ed9d640 1 -- 172.21.15.171:0/3237872521 learned_addr learned my addr 172.21.15.171:0/3237872521 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554865405 0 0) 0x7fab9009d1b0 con 0x7fab901395d0 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): my global_id is 6073 2024-09-16T04:11:44.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab7c003520 con 0x7fab901395d0 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089221501 0 0) 0x7fab900a1490 con 0x7fab9009fba0 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient(hunting): my global_id is 64512 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab9009d1b0 con 0x7fab9009fba0 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4205428003 0 0) 0x7fab7c003520 con 0x7fab901395d0 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 >> v1:172.21.15.148:6790/0 conn(0x7fab9009f7c0 legacy=0x7fab9009de70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 >> v1:172.21.15.148:6789/0 conn(0x7fab9009fba0 legacy=0x7fab900a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 monclient: found mon.noname-c 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9009bca0 con 0x7fab901395d0 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab88001910 con 0x7fab901395d0 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: dump: 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:44.365 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab94cd0640 5 monclient: authenticate success, global_id 6073 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab94cd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab88001b70 con 0x7fab901395d0 2024-09-16T04:11:44.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 1 -- 172.21.15.171:0/3237872521 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab88001f40 con 0x7fab901395d0 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab94cd0640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab94cd0640 4 set_mon_vals no callback set 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.350+0000 7fab8dd9b640 10 monclient: dump: 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:44.367 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 10 monclient: shutdown 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 1 -- 172.21.15.171:0/3237872521 >> v1:172.21.15.171:6789/0 conn(0x7fab901395d0 legacy=0x7fab9013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 1 -- 172.21.15.171:0/3237872521 shutdown_connections 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 1 -- 172.21.15.171:0/3237872521 >> 172.21.15.171:0/3237872521 conn(0x7fab900a5450 msgr2=0x7fab900a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 1 -- 172.21.15.171:0/3237872521 shutdown_connections 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 1 -- 172.21.15.171:0/3237872521 wait complete. 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.351+0000 7fab94cd0640 10 monclient: monmap: 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:44.368 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 1 Processor -- start 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 1 -- start start 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient: init 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.369 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient(hunting): picked mon.noname-b con 0x7fab9009f7c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient(hunting): picked mon.noname-a con 0x7fab9009fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient(hunting): picked mon.noname-c con 0x7fab901395d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab90120140 con 0x7fab9009fba0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab90121b50 con 0x7fab9009f7c0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab900a4290 con 0x7fab901395d0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab94cd0640 10 monclient(hunting): authenticate will time out at 1079.857695s 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab8e59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fab9009f7c0 0x7fab9011de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36258/0 (socket says 172.21.15.171:36258) 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab8ed9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fab901395d0 legacy=0x7fab901d17c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab8e59c640 1 -- 172.21.15.171:0/2555148035 learned_addr learned my addr 172.21.15.171:0/2555148035 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:44.370 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919713669 0 0) 0x7fab90121b50 con 0x7fab9009f7c0 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 10 monclient(hunting): my global_id is 6076 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab70003520 con 0x7fab9009f7c0 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab87fff640 1 -- 172.21.15.171:0/2555148035 >> v1:172.21.15.148:6789/0 conn(0x7fab9009fba0 legacy=0x7fab9011e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab87fff640 1 -- 172.21.15.171:0/2555148035 >> v1:172.21.15.148:6789/0 conn(0x7fab9009fba0 legacy=0x7fab9011e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab87fff640 1 --1- 172.21.15.171:0/2555148035 >> v1:172.21.15.148:6789/0 conn(0x7fab9009fba0 0x7fab9011e540 :-1 s=OPENED pgs=201 cs=1 l=1).handle_message read tag failed 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab87fff640 1 --1- 172.21.15.171:0/2555148035 >> v1:172.21.15.148:6789/0 conn(0x7fab9009fba0 0x7fab9011e540 :-1 s=OPENED pgs=201 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.352+0000 7fab86ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1273395701 0 0) 0x7fab70003520 con 0x7fab9009f7c0 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fab90121b50 con 0x7fab9009f7c0 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab800041d0 con 0x7fab9009f7c0 2024-09-16T04:11:44.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:44.372 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2495690055 0 0) 0x7fab90121b50 con 0x7fab9009f7c0 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 >> v1:172.21.15.148:6790/0 conn(0x7fab901395d0 legacy=0x7fab901d17c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 >> v1:172.21.15.148:6789/0 conn(0x7fab9009fba0 legacy=0x7fab9011e540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 monclient: found mon.b 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab900a3d40 con 0x7fab9009f7c0 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 5 monclient: authenticate success, global_id 6076 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 10 monclient: _renew_subs 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fab9009c0b0 con 0x7fab9009f7c0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 10 monclient: _renew_subs 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fab90121870 con 0x7fab9009f7c0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fab800044f0 con 0x7fab9009f7c0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fab800048c0 con 0x7fab9009f7c0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab86ffd640 10 monclient: dump: 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:44.375 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.353+0000 7fab8d59a640 4 set_mon_vals no callback set 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fab80004a60 con 0x7fab9009f7c0 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab94cd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab94cd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab94cd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab74005080 con 0x7fab9009f7c0 2024-09-16T04:11:44.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(203..203 src has 1..203) ==== 11588+0+0 (unknown 4246494721 0 0) 0x7fab80048cc0 con 0x7fab9009f7c0 2024-09-16T04:11:44.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.354+0000 7fab87fff640 10 monclient: get_auth_request con 0x7fab70038e70 auth_method 0 2024-09-16T04:11:44.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.358+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fab80015d60 con 0x7fab9009f7c0 2024-09-16T04:11:44.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.358+0000 7fab86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.358+0000 7fab86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:44.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.526+0000 7fab94cd0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:44.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.526+0000 7fab94cd0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:44.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.526+0000 7fab94cd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:44.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:44.526+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fab74005370 con 0x7fab9009f7c0 2024-09-16T04:11:45.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.156+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v204) ==== 168+0+0 (unknown 1720689476 0 0) 0x7fab80015660 con 0x7fab9009f7c0 2024-09-16T04:11:45.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.156+0000 7fab86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.156+0000 7fab86ffd640 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-16T04:11:45.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.259+0000 7fab94cd0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.261 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.259+0000 7fab94cd0640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.259+0000 7fab94cd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.259+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fab74005bd0 con 0x7fab9009f7c0 2024-09-16T04:11:45.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.260+0000 7fab86ffd640 1 -- 172.21.15.171:0/2555148035 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v204) ==== 147+0+0 (unknown 492235675 0 0) 0x7fab8001e210 con 0x7fab9009f7c0 2024-09-16T04:11:45.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.260+0000 7fab86ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.262 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.260+0000 7fab86ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-16T04:11:45.263 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h21' already exists 2024-09-16T04:11:45.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 >> v1:172.21.15.171:6812/7327273 conn(0x7fab70038e70 legacy=0x7fab7003b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 10 monclient: shutdown 2024-09-16T04:11:45.264 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:45.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 >> v1:172.21.15.171:6789/0 conn(0x7fab9009f7c0 legacy=0x7fab9011de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 shutdown_connections 2024-09-16T04:11:45.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 >> 172.21.15.171:0/2555148035 conn(0x7fab900a5450 msgr2=0x7fab901378e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:45.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 shutdown_connections 2024-09-16T04:11:45.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.262+0000 7fab94cd0640 1 -- 172.21.15.171:0/2555148035 wait complete. 2024-09-16T04:11:45.273 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:45.273 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-16T04:11:45.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.353+0000 7fcc344ed640 10 monclient: get_monmap_and_config 2024-09-16T04:11:45.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.353+0000 7fcc344ed640 10 monclient: build_initial_monmap 2024-09-16T04:11:45.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.353+0000 7fcc344ed640 10 monclient: monmap: 2024-09-16T04:11:45.358 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:45.354190+0000 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:45.354190+0000 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:45.359 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.353+0000 7fcc344ed640 1 Processor -- start 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 1 -- start start 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient: init 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2c072aa0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:45.360 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2c0702c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2c13b0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c0a0590 con 0x7fcc2c13b0e0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c09f000 con 0x7fcc2c072aa0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c0a02b0 con 0x7fcc2c0702c0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc344ed640 10 monclient(hunting): authenticate will time out at 1080.859770s 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc32289640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcc2c072aa0 0x7fcc2c072e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36264/0 (socket says 172.21.15.171:36264) 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc32289640 1 -- 172.21.15.171:0/1620612539 learned_addr learned my addr 172.21.15.171:0/1620612539 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.354+0000 7fcc31a88640 1 -- 172.21.15.171:0/1620612539 >> v1:172.21.15.148:6790/0 conn(0x7fcc2c0702c0 legacy=0x7fcc2c1399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3079455171 0 0) 0x7fcc2c09f000 con 0x7fcc2c072aa0 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient(hunting): _negotiate 2024-09-16T04:11:45.361 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient(hunting): my global_id is 6079 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc18003520 con 0x7fcc2c072aa0 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1365796279 0 0) 0x7fcc18003520 con 0x7fcc2c072aa0 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 >> v1:172.21.15.148:6790/0 conn(0x7fcc2c0702c0 legacy=0x7fcc2c1399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 >> v1:172.21.15.148:6789/0 conn(0x7fcc2c13b0e0 legacy=0x7fcc2c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: _finish_hunting 0 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 monclient: found mon.noname-c 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c09f280 con 0x7fcc2c072aa0 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: _finish_auth 0 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: _check_auth_tickets 2024-09-16T04:11:45.362 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcc28001610 con 0x7fcc2c072aa0 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 5 monclient: authenticate success, global_id 6079 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: dump: 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:45.363 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc28001970 con 0x7fcc2c072aa0 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 1 -- 172.21.15.171:0/1620612539 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcc28001d40 con 0x7fcc2c072aa0 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 4 set_mon_vals no callback set 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc31287640 10 monclient: dump: 2024-09-16T04:11:45.364 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 10 monclient: shutdown 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.355+0000 7fcc344ed640 1 -- 172.21.15.171:0/1620612539 >> v1:172.21.15.171:6789/0 conn(0x7fcc2c072aa0 legacy=0x7fcc2c072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 1 -- 172.21.15.171:0/1620612539 shutdown_connections 2024-09-16T04:11:45.365 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 1 -- 172.21.15.171:0/1620612539 >> 172.21.15.171:0/1620612539 conn(0x7fcc2c0a77f0 msgr2=0x7fcc2c0a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 1 -- 172.21.15.171:0/1620612539 shutdown_connections 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 1 -- 172.21.15.171:0/1620612539 wait complete. 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 10 monclient: build_initial_monmap 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 10 monclient: monmap: 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:45.366 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:45.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 1 Processor -- start 2024-09-16T04:11:45.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 1 -- start start 2024-09-16T04:11:45.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.356+0000 7fcc344ed640 10 monclient: init 2024-09-16T04:11:45.373 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2c0702c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2c072aa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2c13b0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c11f1b0 con 0x7fcc2c13b0e0 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c120760 con 0x7fcc2c0702c0 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c120490 con 0x7fcc2c072aa0 2024-09-16T04:11:45.374 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc344ed640 10 monclient(hunting): authenticate will time out at 1080.862495s 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc32289640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fcc2c0702c0 0x7fcc2c135140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36270/0 (socket says 172.21.15.171:36270) 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc32289640 1 -- 172.21.15.171:0/2374969475 learned_addr learned my addr 172.21.15.171:0/2374969475 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc31a88640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6790/0 conn(0x7fcc2c072aa0 legacy=0x7fcc2c1ce6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1656504619 0 0) 0x7fcc2c120760 con 0x7fcc2c0702c0 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): my global_id is 6082 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc10003520 con 0x7fcc2c0702c0 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc32a8a640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6789/0 conn(0x7fcc2c13b0e0 legacy=0x7fcc2c1d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc32a8a640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6789/0 conn(0x7fcc2c13b0e0 legacy=0x7fcc2c1d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:11:45.375 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc32a8a640 1 --1- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6789/0 conn(0x7fcc2c13b0e0 0x7fcc2c1d1de0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc32a8a640 1 --1- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6789/0 conn(0x7fcc2c13b0e0 0x7fcc2c1d1de0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3785000641 0 0) 0x7fcc10003520 con 0x7fcc2c0702c0 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc2c120760 con 0x7fcc2c0702c0 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcc280015d0 con 0x7fcc2c0702c0 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): dump: 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.376 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.357+0000 7fcc1effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 436699720 0 0) 0x7fcc2c120760 con 0x7fcc2c0702c0 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6790/0 conn(0x7fcc2c072aa0 legacy=0x7fcc2c1ce6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.148:6789/0 conn(0x7fcc2c13b0e0 legacy=0x7fcc2c1d1de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 monclient: found mon.b 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:45.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c11f870 con 0x7fcc2c0702c0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: _finish_auth 0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 5 monclient: authenticate success, global_id 6082 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 10 monclient: _renew_subs 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c120d20 con 0x7fcc2c0702c0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 10 monclient: _renew_subs 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc2c11e620 con 0x7fcc2c0702c0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc28004cc0 con 0x7fcc2c0702c0 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:45.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcc28005010 con 0x7fcc2c0702c0 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 10 monclient: dump: 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:45.379 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc30a86640 4 set_mon_vals no callback set 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcc28010540 con 0x7fcc2c0702c0 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.358+0000 7fcc344ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:45.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.359+0000 7fcc344ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.359+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc14005080 con 0x7fcc2c0702c0 2024-09-16T04:11:45.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.359+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(204..204 src has 1..204) ==== 11801+0+0 (unknown 2811072544 0 0) 0x7fcc28048fb0 con 0x7fcc2c0702c0 2024-09-16T04:11:45.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.359+0000 7fcc31a88640 10 monclient: get_auth_request con 0x7fcc10038e70 auth_method 0 2024-09-16T04:11:45.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.363+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fcc28015f50 con 0x7fcc2c0702c0 2024-09-16T04:11:45.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.363+0000 7fcc1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:45.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.363+0000 7fcc1effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:45.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.532+0000 7fcc344ed640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.532+0000 7fcc344ed640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.532+0000 7fcc344ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.532+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fcc14005370 con 0x7fcc2c0702c0 2024-09-16T04:11:45.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.825+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v205) ==== 168+0+0 (unknown 1385010638 0 0) 0x7fcc28015850 con 0x7fcc2c0702c0 2024-09-16T04:11:45.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.825+0000 7fcc1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.827 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.825+0000 7fcc1effd640 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-16T04:11:45.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.928+0000 7fcc344ed640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.928+0000 7fcc344ed640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.928+0000 7fcc344ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:45.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.928+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fcc14005bd0 con 0x7fcc2c0702c0 2024-09-16T04:11:45.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.929+0000 7fcc1effd640 1 -- 172.21.15.171:0/2374969475 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v205) ==== 147+0+0 (unknown 685753925 0 0) 0x7fcc28044d70 con 0x7fcc2c0702c0 2024-09-16T04:11:45.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.929+0000 7fcc1effd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:45.931 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.929+0000 7fcc1effd640 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-16T04:11:45.931 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h22' already exists 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.171:6812/7327273 conn(0x7fcc10038e70 legacy=0x7fcc1003b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 10 monclient: shutdown 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 >> v1:172.21.15.171:6789/0 conn(0x7fcc2c0702c0 legacy=0x7fcc2c135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 shutdown_connections 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 >> 172.21.15.171:0/2374969475 conn(0x7fcc2c0a77f0 msgr2=0x7fcc2c0a4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 shutdown_connections 2024-09-16T04:11:45.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:45.931+0000 7fcc344ed640 1 -- 172.21.15.171:0/2374969475 wait complete. 2024-09-16T04:11:45.941 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:45.942 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-16T04:11:46.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.020+0000 7f33cf27b640 10 monclient: get_monmap_and_config 2024-09-16T04:11:46.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.020+0000 7f33cf27b640 10 monclient: build_initial_monmap 2024-09-16T04:11:46.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.020+0000 7f33cf27b640 10 monclient: monmap: 2024-09-16T04:11:46.023 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:46.023 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:46.023 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:46.021401+0000 2024-09-16T04:11:46.024 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:46.021401+0000 2024-09-16T04:11:46.024 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:46.024 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:46.024 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:46.024 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:46.025 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:46.025 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:46.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 1 Processor -- start 2024-09-16T04:11:46.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 1 -- start start 2024-09-16T04:11:46.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient: init 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient(hunting): picked mon.noname-a con 0x7f33c8133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient(hunting): picked mon.noname-b con 0x7f33c81341f0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient(hunting): picked mon.noname-c con 0x7f33c813a350 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c809da90 con 0x7f33c8133700 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c809dd70 con 0x7f33c813a350 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c809e050 con 0x7f33c81341f0 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:46.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cf27b640 10 monclient(hunting): authenticate will time out at 1081.526887s 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.021+0000 7f33cc816640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f33c81341f0 legacy=0x7f33c8138c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cd818640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f33c813a350 0x7f33c813c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36274/0 (socket says 172.21.15.171:36274) 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cd818640 1 -- 172.21.15.171:0/3039342670 learned_addr learned my addr 172.21.15.171:0/3039342670 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652046925 0 0) 0x7f33c809dd70 con 0x7f33c813a350 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient(hunting): my global_id is 6085 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b4003520 con 0x7f33c813a350 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3562545037 0 0) 0x7f33b4003520 con 0x7f33c813a350 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 >> v1:172.21.15.148:6790/0 conn(0x7f33c81341f0 legacy=0x7f33c8138c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 >> v1:172.21.15.148:6789/0 conn(0x7f33c8133700 legacy=0x7f33c8133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:46.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 monclient: found mon.noname-c 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c809d4f0 con 0x7f33c813a350 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: _finish_auth 0 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cf27b640 5 monclient: authenticate success, global_id 6085 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33c0001990 con 0x7f33c813a350 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cf27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:46.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: dump: 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:46.040 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cf27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33c0001cf0 con 0x7f33c813a350 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cf27b640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33cf27b640 4 set_mon_vals no callback set 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 1 -- 172.21.15.171:0/3039342670 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33c0004c20 con 0x7f33c813a350 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.022+0000 7f33bffff640 10 monclient: dump: 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.041 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 10 monclient: shutdown 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 1 -- 172.21.15.171:0/3039342670 >> v1:172.21.15.171:6789/0 conn(0x7f33c813a350 legacy=0x7f33c813c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 1 -- 172.21.15.171:0/3039342670 shutdown_connections 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 1 -- 172.21.15.171:0/3039342670 >> 172.21.15.171:0/3039342670 conn(0x7f33c80a5450 msgr2=0x7f33c80a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:46.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 1 -- 172.21.15.171:0/3039342670 shutdown_connections 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 1 -- 172.21.15.171:0/3039342670 wait complete. 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 10 monclient: build_initial_monmap 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.023+0000 7f33cf27b640 10 monclient: monmap: 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:46.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 1 Processor -- start 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 1 -- start start 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient: init 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient(hunting): picked mon.noname-a con 0x7f33c8133700 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient(hunting): picked mon.noname-b con 0x7f33c81341f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient(hunting): picked mon.noname-c con 0x7f33c813a350 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c811c2c0 con 0x7f33c8133700 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c811c5a0 con 0x7f33c81341f0 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c8139c60 con 0x7f33c813a350 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:46.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cf27b640 10 monclient(hunting): authenticate will time out at 1081.529555s 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cd818640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f33c813a350 legacy=0x7f33c8071540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cc816640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f33c81341f0 0x7f33c8070e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36286/0 (socket says 172.21.15.171:36286) 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33cc816640 1 -- 172.21.15.171:0/3968524097 learned_addr learned my addr 172.21.15.171:0/3968524097 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380886757 0 0) 0x7f33c811c5a0 con 0x7f33c81341f0 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): my global_id is 6088 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33ac003520 con 0x7f33c81341f0 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775349702 0 0) 0x7f33c811c2c0 con 0x7f33c8133700 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:46.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 10 monclient(hunting): my global_id is 64521 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.024+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33c811c5a0 con 0x7f33c8133700 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 957501823 0 0) 0x7f33ac003520 con 0x7f33c81341f0 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33c811c2c0 con 0x7f33c81341f0 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33b8004180 con 0x7f33c81341f0 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient(hunting): dump: 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.046 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 950504726 0 0) 0x7f33c811c2c0 con 0x7f33c81341f0 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 >> v1:172.21.15.148:6790/0 conn(0x7f33c813a350 legacy=0x7f33c8071540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 >> v1:172.21.15.148:6789/0 conn(0x7f33c8133700 legacy=0x7f33c8070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 monclient: found mon.b 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:46.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c811bfe0 con 0x7f33c81341f0 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 5 monclient: authenticate success, global_id 6088 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 10 monclient: _renew_subs 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f33c81d0520 con 0x7f33c81341f0 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33b8004430 con 0x7f33c81341f0 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:46.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33b8004800 con 0x7f33c81341f0 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bdffb640 10 monclient: dump: 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:46.049 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33bf7fe640 4 set_mon_vals no callback set 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 10 monclient: _renew_subs 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.025+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f33c81d08d0 con 0x7f33c81341f0 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f33b8004a20 con 0x7f33c81341f0 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(205..205 src has 1..205) ==== 12014+0+0 (unknown 2130990634 0 0) 0x7f33b8048e00 con 0x7f33c81341f0 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33cd017640 10 monclient: get_auth_request con 0x7f33ac038bc0 auth_method 0 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33cf27b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:46.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33cf27b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:46.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33cf27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:46.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.026+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c809c0b0 con 0x7f33c81341f0 2024-09-16T04:11:46.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.031+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f33b8015cf0 con 0x7f33c81341f0 2024-09-16T04:11:46.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.031+0000 7f33bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:46.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.031+0000 7f33bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:46.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.205+0000 7f33cf27b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:46.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.205+0000 7f33cf27b640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:46.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.205+0000 7f33cf27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:46.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:46.205+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f33c806bd60 con 0x7f33c81341f0 2024-09-16T04:11:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.167+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v206) ==== 168+0+0 (unknown 2661984446 0 0) 0x7f33b80155f0 con 0x7f33c81341f0 2024-09-16T04:11:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.167+0000 7f33bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:47.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.167+0000 7f33bdffb640 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-16T04:11:47.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.273+0000 7f33cf27b640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:47.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.273+0000 7f33cf27b640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:47.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.273+0000 7f33cf27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.276 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.273+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f33c809bca0 con 0x7f33c81341f0 2024-09-16T04:11:47.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.275+0000 7f33bdffb640 1 -- 172.21.15.171:0/3968524097 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v206) ==== 147+0+0 (unknown 171628609 0 0) 0x7f33b8019940 con 0x7f33c81341f0 2024-09-16T04:11:47.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.275+0000 7f33bdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:47.277 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.275+0000 7f33bdffb640 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-16T04:11:47.277 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h23' already exists 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 >> v1:172.21.15.171:6812/7327273 conn(0x7f33ac038bc0 legacy=0x7f33ac03b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 10 monclient: shutdown 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 >> v1:172.21.15.171:6789/0 conn(0x7f33c81341f0 legacy=0x7f33c8070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 shutdown_connections 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 >> 172.21.15.171:0/3968524097 conn(0x7f33c80a5450 msgr2=0x7f33c80a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 shutdown_connections 2024-09-16T04:11:47.279 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.277+0000 7f33cf27b640 1 -- 172.21.15.171:0/3968524097 wait complete. 2024-09-16T04:11:47.288 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:47.288 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-16T04:11:47.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.369+0000 7fbee06d0640 10 monclient: get_monmap_and_config 2024-09-16T04:11:47.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.369+0000 7fbee06d0640 10 monclient: build_initial_monmap 2024-09-16T04:11:47.371 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.369+0000 7fbee06d0640 10 monclient: monmap: 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:47.370644+0000 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:47.370644+0000 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:47.372 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.370+0000 7fbee06d0640 1 Processor -- start 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.370+0000 7fbee06d0640 1 -- start start 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.370+0000 7fbee06d0640 10 monclient: init 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient(hunting): picked mon.noname-a con 0x7fbed809f7c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:47.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient(hunting): picked mon.noname-c con 0x7fbed809fba0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient(hunting): picked mon.noname-b con 0x7fbed81395d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed80a1490 con 0x7fbed809f7c0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed809d1b0 con 0x7fbed809fba0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed809d490 con 0x7fbed81395d0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient(hunting): authenticate will time out at 1082.876143s 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbeddc6b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbed809fba0 0x7fbed80a0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58004/0 (socket says 172.21.15.171:58004) 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbeddc6b640 1 -- 172.21.15.171:0/1328942320 learned_addr learned my addr 172.21.15.171:0/1328942320 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedec6d640 1 -- 172.21.15.171:0/1328942320 >> v1:172.21.15.148:6790/0 conn(0x7fbed81395d0 legacy=0x7fbed813b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660954546 0 0) 0x7fbed809d1b0 con 0x7fbed809fba0 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:47.380 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient(hunting): my global_id is 6091 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec0003440 con 0x7fbed809fba0 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2711881490 0 0) 0x7fbec0003440 con 0x7fbed809fba0 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 >> v1:172.21.15.148:6790/0 conn(0x7fbed81395d0 legacy=0x7fbed813b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 >> v1:172.21.15.148:6789/0 conn(0x7fbed809f7c0 legacy=0x7fbed809de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 monclient: found mon.noname-c 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed809bca0 con 0x7fbed809fba0 2024-09-16T04:11:47.381 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: _finish_auth 0 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbecc001610 con 0x7fbed809fba0 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 5 monclient: authenticate success, global_id 6091 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: dump: 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.382 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbecc001970 con 0x7fbed809fba0 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 1 -- 172.21.15.171:0/1328942320 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbecc0048a0 con 0x7fbed809fba0 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:47.383 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbee06d0640 4 set_mon_vals no callback set 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.371+0000 7fbedd46a640 10 monclient: dump: 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:47.384 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 10 monclient: shutdown 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 1 -- 172.21.15.171:0/1328942320 >> v1:172.21.15.171:6789/0 conn(0x7fbed809fba0 legacy=0x7fbed80a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 1 -- 172.21.15.171:0/1328942320 shutdown_connections 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 1 -- 172.21.15.171:0/1328942320 >> 172.21.15.171:0/1328942320 conn(0x7fbed80a5450 msgr2=0x7fbed80a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 1 -- 172.21.15.171:0/1328942320 shutdown_connections 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 1 -- 172.21.15.171:0/1328942320 wait complete. 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 10 monclient: build_initial_monmap 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.372+0000 7fbee06d0640 10 monclient: monmap: 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:47.385 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 1 Processor -- start 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 1 -- start start 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient: init 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient(hunting): picked mon.noname-c con 0x7fbed809f7c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient(hunting): picked mon.noname-a con 0x7fbed809fba0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:47.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient(hunting): picked mon.noname-b con 0x7fbed81395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed81beb80 con 0x7fbed809fba0 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed80a3e20 con 0x7fbed81395d0 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed8120990 con 0x7fbed809f7c0 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbede46c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fbed809f7c0 legacy=0x7fbed81317e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbee06d0640 10 monclient(hunting): authenticate will time out at 1082.878797s 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbedec6d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fbed81395d0 0x7fbed81d4170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58008/0 (socket says 172.21.15.171:58008) 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.373+0000 7fbedec6d640 1 -- 172.21.15.171:0/700426989 learned_addr learned my addr 172.21.15.171:0/700426989 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165628847 0 0) 0x7fbec8003d70 con 0x7fbed81395d0 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:47.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): my global_id is 6094 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4003520 con 0x7fbed81395d0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1784042102 0 0) 0x7fbed809bca0 con 0x7fbed809fba0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): my global_id is 64527 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec8003d70 con 0x7fbed809fba0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2973475888 0 0) 0x7fbec4003520 con 0x7fbed81395d0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbed809bca0 con 0x7fbed81395d0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbec8004580 con 0x7fbed81395d0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:47.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:47.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1252580056 0 0) 0x7fbed809bca0 con 0x7fbed81395d0 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 >> v1:172.21.15.148:6790/0 conn(0x7fbed809f7c0 legacy=0x7fbed81317e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 >> v1:172.21.15.148:6789/0 conn(0x7fbed809fba0 legacy=0x7fbed81d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 monclient: found mon.b 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed8120c70 con 0x7fbed81395d0 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 5 monclient: authenticate success, global_id 6094 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 10 monclient: _renew_subs 2024-09-16T04:11:47.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbed8131ef0 con 0x7fbed81395d0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 10 monclient: _renew_subs 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbed811f540 con 0x7fbed81395d0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbec80047e0 con 0x7fbed81395d0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbec8004bb0 con 0x7fbed81395d0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbedcc69640 4 set_mon_vals no callback set 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.374+0000 7fbed6ffd640 10 monclient: dump: 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:47.391 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbec8004d50 con 0x7fbed81395d0 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbee06d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbee06d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbee06d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb8005080 con 0x7fbed81395d0 2024-09-16T04:11:47.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(206..206 src has 1..206) ==== 12227+0+0 (unknown 2847570335 0 0) 0x7fbec80492f0 con 0x7fbed81395d0 2024-09-16T04:11:47.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.375+0000 7fbede46c640 10 monclient: get_auth_request con 0x7fbec4039000 auth_method 0 2024-09-16T04:11:47.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.380+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fbed80a3e20 con 0x7fbed81395d0 2024-09-16T04:11:47.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.380+0000 7fbed6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:47.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.380+0000 7fbed6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:47.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.549+0000 7fbee06d0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:47.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.549+0000 7fbee06d0640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:47.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.549+0000 7fbee06d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:47.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:47.549+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fbeb8005370 con 0x7fbed81395d0 2024-09-16T04:11:48.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.176+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v207) ==== 168+0+0 (unknown 982120666 0 0) 0x7fbec8016050 con 0x7fbed81395d0 2024-09-16T04:11:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.176+0000 7fbed6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:48.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.176+0000 7fbed6ffd640 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-16T04:11:48.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.280+0000 7fbee06d0640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:48.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.280+0000 7fbee06d0640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:48.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.280+0000 7fbee06d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.280+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fbeb8005ab0 con 0x7fbed81395d0 2024-09-16T04:11:48.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.282+0000 7fbed6ffd640 1 -- 172.21.15.171:0/700426989 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v207) ==== 147+0+0 (unknown 1137548793 0 0) 0x7fbec8015950 con 0x7fbed81395d0 2024-09-16T04:11:48.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.282+0000 7fbed6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:48.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.282+0000 7fbed6ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-16T04:11:48.284 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h24' already exists 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 >> v1:172.21.15.171:6812/7327273 conn(0x7fbec4039000 legacy=0x7fbec403b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 10 monclient: shutdown 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 >> v1:172.21.15.171:6789/0 conn(0x7fbed81395d0 legacy=0x7fbed81d4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 shutdown_connections 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 >> 172.21.15.171:0/700426989 conn(0x7fbed80a5450 msgr2=0x7fbed8138d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 shutdown_connections 2024-09-16T04:11:48.286 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.284+0000 7fbee06d0640 1 -- 172.21.15.171:0/700426989 wait complete. 2024-09-16T04:11:48.295 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:48.295 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-16T04:11:48.377 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.376+0000 7f24d7845640 10 monclient: get_monmap_and_config 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.376+0000 7f24d7845640 10 monclient: build_initial_monmap 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.376+0000 7f24d7845640 10 monclient: monmap: 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:48.376875+0000 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:48.376875+0000 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:48.378 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:48.379 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:48.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.376+0000 7f24d7845640 1 Processor -- start 2024-09-16T04:11:48.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.376+0000 7f24d7845640 1 -- start start 2024-09-16T04:11:48.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.376+0000 7f24d7845640 10 monclient: init 2024-09-16T04:11:48.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:48.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:48.385 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient(hunting): picked mon.noname-b con 0x7f24d0136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient(hunting): picked mon.noname-c con 0x7f24d0137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient(hunting): picked mon.noname-a con 0x7f24d01375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d009b2c0 con 0x7f24d01375a0 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d009b5a0 con 0x7f24d0137140 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d00a6640 con 0x7f24d0136d60 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d7845640 10 monclient(hunting): authenticate will time out at 1083.882435s 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d4de0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f24d0137140 0x7f24d013bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58014/0 (socket says 172.21.15.171:58014) 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d4de0640 1 -- 172.21.15.171:0/3033784234 learned_addr learned my addr 172.21.15.171:0/3033784234 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24d55e1640 1 -- 172.21.15.171:0/3033784234 >> v1:172.21.15.148:6790/0 conn(0x7f24d0136d60 legacy=0x7f24d009bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:48.386 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4083941394 0 0) 0x7f24d009b5a0 con 0x7f24d0137140 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 10 monclient(hunting): my global_id is 6097 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24b8003520 con 0x7f24d0137140 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2039467369 0 0) 0x7f24b8003520 con 0x7f24d0137140 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 >> v1:172.21.15.148:6790/0 conn(0x7f24d0136d60 legacy=0x7f24d009bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 >> v1:172.21.15.148:6789/0 conn(0x7f24d01375a0 legacy=0x7f24d013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.377+0000 7f24cffff640 1 monclient: found mon.noname-c 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d0143b20 con 0x7f24d0137140 2024-09-16T04:11:48.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: _finish_auth 0 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f24c4001610 con 0x7f24d0137140 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 5 monclient: authenticate success, global_id 6097 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: dump: 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.388 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24c4001900 con 0x7f24d0137140 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 1 -- 172.21.15.171:0/3033784234 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f24c4001cd0 con 0x7f24d0137140 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:48.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24cffff640 10 monclient: dump: 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:48.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 4 set_mon_vals no callback set 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 10 monclient: shutdown 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 1 -- 172.21.15.171:0/3033784234 >> v1:172.21.15.171:6789/0 conn(0x7f24d0137140 legacy=0x7f24d013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 1 -- 172.21.15.171:0/3033784234 shutdown_connections 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 1 -- 172.21.15.171:0/3033784234 >> 172.21.15.171:0/3033784234 conn(0x7f24d00a9ae0 msgr2=0x7f24d00a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 1 -- 172.21.15.171:0/3033784234 shutdown_connections 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.378+0000 7f24d7845640 1 -- 172.21.15.171:0/3033784234 wait complete. 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.379+0000 7f24d7845640 10 monclient: build_initial_monmap 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.379+0000 7f24d7845640 10 monclient: monmap: 2024-09-16T04:11:48.391 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.379+0000 7f24d7845640 1 Processor -- start 2024-09-16T04:11:48.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.379+0000 7f24d7845640 1 -- start start 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.379+0000 7f24d7845640 10 monclient: init 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient(hunting): picked mon.noname-b con 0x7f24d0136d60 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient(hunting): picked mon.noname-c con 0x7f24d0137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient(hunting): picked mon.noname-a con 0x7f24d01375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d00700c0 con 0x7f24d01375a0 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d4de0640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f24d0137140 legacy=0x7f24d0135630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d0071920 con 0x7f24d0136d60 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d55e1640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f24d0136d60 0x7f24d0134f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58026/0 (socket says 172.21.15.171:58026) 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d55e1640 1 -- 172.21.15.171:0/2293834982 learned_addr learned my addr 172.21.15.171:0/2293834982 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:48.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d0135d40 con 0x7f24d0137140 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient(hunting): authenticate will time out at 1083.885241s 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599754744 0 0) 0x7f24d0071920 con 0x7f24d0136d60 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): my global_id is 6100 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24bc003520 con 0x7f24d0136d60 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596579412 0 0) 0x7f24d00700c0 con 0x7f24d01375a0 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:48.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): my global_id is 64533 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24d0071920 con 0x7f24d01375a0 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 91856137 0 0) 0x7f24bc003520 con 0x7f24d0136d60 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f24d00700c0 con 0x7f24d0136d60 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f24c0004500 con 0x7f24d0136d60 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): dump: 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:48.395 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 958795295 0 0) 0x7f24d00700c0 con 0x7f24d0136d60 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 >> v1:172.21.15.148:6790/0 conn(0x7f24d0137140 legacy=0x7f24d0135630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 >> v1:172.21.15.148:6789/0 conn(0x7f24d01375a0 legacy=0x7f24d01cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:48.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 monclient: found mon.b 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d0137b80 con 0x7f24d0136d60 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 5 monclient: authenticate success, global_id 6100 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient: _renew_subs 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f24d00a8460 con 0x7f24d0136d60 2024-09-16T04:11:48.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f24c00047b0 con 0x7f24d0136d60 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f24c0004b80 con 0x7f24d0136d60 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cdffb640 10 monclient: dump: 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:48.398 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 10 monclient: _renew_subs 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f24d00704a0 con 0x7f24d0136d60 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.380+0000 7f24cf7fe640 4 set_mon_vals no callback set 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f24c0004d20 con 0x7f24d0136d60 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(207..207 src has 1..207) ==== 12440+0+0 (unknown 3545162834 0 0) 0x7f24c00493b0 con 0x7f24d0136d60 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:48.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d0071f90 con 0x7f24d0136d60 2024-09-16T04:11:48.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.381+0000 7f24d4de0640 10 monclient: get_auth_request con 0x7f24bc038b90 auth_method 0 2024-09-16T04:11:48.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.386+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f24c0015fe0 con 0x7f24d0136d60 2024-09-16T04:11:48.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.386+0000 7f24cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:48.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.386+0000 7f24cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:48.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.551+0000 7f24d7845640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:48.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.551+0000 7f24d7845640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:48.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.551+0000 7f24d7845640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:48.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:48.551+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f24d006bd60 con 0x7f24d0136d60 2024-09-16T04:11:49.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.179+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v208) ==== 168+0+0 (unknown 3419907274 0 0) 0x7f24c00158e0 con 0x7f24d0136d60 2024-09-16T04:11:49.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.179+0000 7f24cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:49.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.179+0000 7f24cdffb640 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-16T04:11:49.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.280+0000 7f24d7845640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:49.282 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.280+0000 7f24d7845640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:49.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.280+0000 7f24d7845640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.280+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f24d01407e0 con 0x7f24d0136d60 2024-09-16T04:11:49.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.282+0000 7f24cdffb640 1 -- 172.21.15.171:0/2293834982 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v208) ==== 147+0+0 (unknown 3652639282 0 0) 0x7f24c0015670 con 0x7f24d0136d60 2024-09-16T04:11:49.283 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.282+0000 7f24cdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:49.284 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.282+0000 7f24cdffb640 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-16T04:11:49.284 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h25' already exists 2024-09-16T04:11:49.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 >> v1:172.21.15.171:6812/7327273 conn(0x7f24bc038b90 legacy=0x7f24bc03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:49.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 10 monclient: shutdown 2024-09-16T04:11:49.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:49.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 >> v1:172.21.15.171:6789/0 conn(0x7f24d0136d60 legacy=0x7f24d0134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:49.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 shutdown_connections 2024-09-16T04:11:49.288 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 >> 172.21.15.171:0/2293834982 conn(0x7f24d00a9ae0 msgr2=0x7f24d0072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:49.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 shutdown_connections 2024-09-16T04:11:49.289 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.284+0000 7f24d7845640 1 -- 172.21.15.171:0/2293834982 wait complete. 2024-09-16T04:11:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:49.295 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.375+0000 7f0a5d531640 10 monclient: get_monmap_and_config 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.375+0000 7f0a5d531640 10 monclient: build_initial_monmap 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.375+0000 7f0a5d531640 10 monclient: monmap: 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:49.376677+0000 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:49.376677+0000 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:49.393 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.376+0000 7f0a5d531640 1 Processor -- start 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.376+0000 7f0a5d531640 1 -- start start 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.376+0000 7f0a5d531640 10 monclient: init 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.376+0000 7f0a5d531640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.376+0000 7f0a5d531640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.376+0000 7f0a5d531640 10 monclient(hunting): picked mon.noname-c con 0x7f0a580a0ae0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:49.394 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 10 monclient(hunting): picked mon.noname-b con 0x7f0a5809f320 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 10 monclient(hunting): picked mon.noname-a con 0x7f0a5813d2c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a580737b0 con 0x7f0a5813d2c0 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a5806c6d0 con 0x7f0a580a0ae0 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a5806d980 con 0x7f0a5809f320 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 10 monclient(hunting): authenticate will time out at 1084.882056s 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a56ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0a580a0ae0 0x7f0a580a0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58038/0 (socket says 172.21.15.171:58038) 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a56ffd640 1 -- 172.21.15.171:0/4018399064 learned_addr learned my addr 172.21.15.171:0/4018399064 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a567fc640 1 -- 172.21.15.171:0/4018399064 >> v1:172.21.15.148:6790/0 conn(0x7f0a5809f320 legacy=0x7f0a5813bba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663133481 0 0) 0x7f0a5806c6d0 con 0x7f0a580a0ae0 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:49.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient(hunting): my global_id is 6103 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a40003520 con 0x7f0a580a0ae0 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 918605962 0 0) 0x7f0a40003520 con 0x7f0a580a0ae0 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 >> v1:172.21.15.148:6790/0 conn(0x7f0a5809f320 legacy=0x7f0a5813bba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 >> v1:172.21.15.148:6789/0 conn(0x7f0a5813d2c0 legacy=0x7f0a5813f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:49.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 monclient: found mon.noname-c 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5806c950 con 0x7f0a580a0ae0 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 5 monclient: authenticate success, global_id 6103 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0a48001610 con 0x7f0a580a0ae0 2024-09-16T04:11:49.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: dump: 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:49.398 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a48001970 con 0x7f0a580a0ae0 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 1 -- 172.21.15.171:0/4018399064 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0a48001d40 con 0x7f0a580a0ae0 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a5d531640 4 set_mon_vals no callback set 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.377+0000 7f0a55ffb640 10 monclient: dump: 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.399 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 10 monclient: shutdown 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 1 -- 172.21.15.171:0/4018399064 >> v1:172.21.15.171:6789/0 conn(0x7f0a580a0ae0 legacy=0x7f0a580a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 1 -- 172.21.15.171:0/4018399064 shutdown_connections 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 1 -- 172.21.15.171:0/4018399064 >> 172.21.15.171:0/4018399064 conn(0x7f0a580a77f0 msgr2=0x7f0a580a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:49.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 1 -- 172.21.15.171:0/4018399064 shutdown_connections 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 1 -- 172.21.15.171:0/4018399064 wait complete. 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 10 monclient: build_initial_monmap 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.378+0000 7f0a5d531640 10 monclient: monmap: 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:49.401 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 1 Processor -- start 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 1 -- start start 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient: init 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient(hunting): picked mon.noname-a con 0x7f0a5809f320 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient(hunting): picked mon.noname-c con 0x7f0a580a0ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient(hunting): picked mon.noname-b con 0x7f0a5813d2c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:49.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a58127630 con 0x7f0a5809f320 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a58127910 con 0x7f0a5813d2c0 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a580a1c70 con 0x7f0a580a0ae0 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a5d531640 10 monclient(hunting): authenticate will time out at 1084.884938s 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.379+0000 7f0a567fc640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0a580a0ae0 legacy=0x7f0a581d08d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a577fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0a5813d2c0 0x7f0a581d4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58044/0 (socket says 172.21.15.171:58044) 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a577fe640 1 -- 172.21.15.171:0/2946894729 learned_addr learned my addr 172.21.15.171:0/2946894729 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281806345 0 0) 0x7f0a58127910 con 0x7f0a5813d2c0 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): my global_id is 6106 2024-09-16T04:11:49.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a44003520 con 0x7f0a5813d2c0 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 263615441 0 0) 0x7f0a58127630 con 0x7f0a5809f320 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): my global_id is 64539 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a58127910 con 0x7f0a5809f320 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1077716738 0 0) 0x7f0a44003520 con 0x7f0a5813d2c0 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0a58127630 con 0x7f0a5813d2c0 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0a50001ce0 con 0x7f0a5813d2c0 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:49.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4294491581 0 0) 0x7f0a58127630 con 0x7f0a5813d2c0 2024-09-16T04:11:49.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 >> v1:172.21.15.148:6790/0 conn(0x7f0a580a0ae0 legacy=0x7f0a581d08d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 >> v1:172.21.15.148:6789/0 conn(0x7f0a5809f320 legacy=0x7f0a581cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 monclient: found mon.b 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a580a04c0 con 0x7f0a5813d2c0 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 5 monclient: authenticate success, global_id 6106 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 10 monclient: _renew_subs 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5809e1b0 con 0x7f0a5813d2c0 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 10 monclient: _renew_subs 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0a58125f00 con 0x7f0a5813d2c0 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0a50003c40 con 0x7f0a5813d2c0 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0a50004010 con 0x7f0a5813d2c0 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a557fa640 4 set_mon_vals no callback set 2024-09-16T04:11:49.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.380+0000 7f0a3f7fe640 10 monclient: dump: 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a5d531640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a5d531640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:49.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a5d531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0a50004230 con 0x7f0a5813d2c0 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a38005080 con 0x7f0a5813d2c0 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(208..208 src has 1..208) ==== 12653+0+0 (unknown 758147489 0 0) 0x7f0a500495f0 con 0x7f0a5813d2c0 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.381+0000 7f0a56ffd640 10 monclient: get_auth_request con 0x7f0a44038f60 auth_method 0 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.386+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0a50016120 con 0x7f0a5813d2c0 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.386+0000 7f0a3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:49.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.386+0000 7f0a3f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:49.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.555+0000 7f0a5d531640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:49.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.555+0000 7f0a5d531640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:49.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.555+0000 7f0a5d531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:49.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:49.555+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0a38005370 con 0x7f0a5813d2c0 2024-09-16T04:11:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.186+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v209) ==== 168+0+0 (unknown 4292186688 0 0) 0x7f0a50015a20 con 0x7f0a5813d2c0 2024-09-16T04:11:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.186+0000 7f0a3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:50.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.186+0000 7f0a3f7fe640 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-16T04:11:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.291+0000 7f0a5d531640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.291+0000 7f0a5d531640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.291+0000 7f0a5d531640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.291+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f0a38005bd0 con 0x7f0a5813d2c0 2024-09-16T04:11:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.292+0000 7f0a3f7fe640 1 -- 172.21.15.171:0/2946894729 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v209) ==== 147+0+0 (unknown 3963615212 0 0) 0x7f0a50019d70 con 0x7f0a5813d2c0 2024-09-16T04:11:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.292+0000 7f0a3f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.292+0000 7f0a3f7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-16T04:11:50.294 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h26' already exists 2024-09-16T04:11:50.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 >> v1:172.21.15.171:6812/7327273 conn(0x7f0a44038f60 legacy=0x7f0a4403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 10 monclient: shutdown 2024-09-16T04:11:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 >> v1:172.21.15.171:6789/0 conn(0x7f0a5813d2c0 legacy=0x7f0a581d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:50.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 shutdown_connections 2024-09-16T04:11:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 >> 172.21.15.171:0/2946894729 conn(0x7f0a580a77f0 msgr2=0x7f0a580a31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 shutdown_connections 2024-09-16T04:11:50.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.294+0000 7f0a5d531640 1 -- 172.21.15.171:0/2946894729 wait complete. 2024-09-16T04:11:50.305 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:50.305 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.385+0000 7f1984417640 10 monclient: get_monmap_and_config 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.385+0000 7f1984417640 10 monclient: build_initial_monmap 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.385+0000 7f1984417640 10 monclient: monmap: 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:50.386167+0000 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:50.386167+0000 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.387 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:50.388 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:50.388 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:50.388 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.385+0000 7f1984417640 1 Processor -- start 2024-09-16T04:11:50.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.385+0000 7f1984417640 1 -- start start 2024-09-16T04:11:50.395 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.385+0000 7f1984417640 10 monclient: init 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient(hunting): picked mon.noname-a con 0x7f197c09b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient(hunting): picked mon.noname-c con 0x7f197c136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient(hunting): picked mon.noname-b con 0x7f197c13d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c073a20 con 0x7f197c09b920 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c099920 con 0x7f197c136da0 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c099c00 con 0x7f197c13d320 2024-09-16T04:11:50.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f1984417640 10 monclient(hunting): authenticate will time out at 1085.891640s 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19819b2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f197c136da0 0x7f197c13bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58052/0 (socket says 172.21.15.171:58052) 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19819b2640 1 -- 172.21.15.171:0/1021518299 learned_addr learned my addr 172.21.15.171:0/1021518299 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19829b4640 1 -- 172.21.15.171:0/1021518299 >> v1:172.21.15.148:6790/0 conn(0x7f197c13d320 legacy=0x7f197c13f710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164761284 0 0) 0x7f197c099920 con 0x7f197c136da0 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19811b1640 10 monclient(hunting): _negotiate 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19811b1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19811b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19811b1640 10 monclient(hunting): my global_id is 6109 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.386+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1968003520 con 0x7f197c136da0 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1769885431 0 0) 0x7f197c073a20 con 0x7f197c09b920 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient(hunting): _negotiate 2024-09-16T04:11:50.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient(hunting): my global_id is 64542 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f197c099920 con 0x7f197c09b920 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 231937361 0 0) 0x7f1968003520 con 0x7f197c136da0 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 >> v1:172.21.15.148:6790/0 conn(0x7f197c13d320 legacy=0x7f197c13f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 >> v1:172.21.15.148:6789/0 conn(0x7f197c09b920 legacy=0x7f197c09bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: _finish_hunting 0 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 monclient: found mon.noname-c 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c099ee0 con 0x7f197c136da0 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: _finish_auth 0 2024-09-16T04:11:50.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: _check_auth_tickets 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 5 monclient: authenticate success, global_id 6109 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f196c001610 con 0x7f197c136da0 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: dump: 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.399 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f196c0018c0 con 0x7f197c136da0 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 1 -- 172.21.15.171:0/1021518299 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f196c001c90 con 0x7f197c136da0 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:50.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 4 set_mon_vals no callback set 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f19811b1640 10 monclient: dump: 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 10 monclient: shutdown 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 1 -- 172.21.15.171:0/1021518299 >> v1:172.21.15.171:6789/0 conn(0x7f197c136da0 legacy=0x7f197c13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 1 -- 172.21.15.171:0/1021518299 shutdown_connections 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 1 -- 172.21.15.171:0/1021518299 >> 172.21.15.171:0/1021518299 conn(0x7f197c0a2150 msgr2=0x7f197c0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 1 -- 172.21.15.171:0/1021518299 shutdown_connections 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.387+0000 7f1984417640 1 -- 172.21.15.171:0/1021518299 wait complete. 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.388+0000 7f1984417640 10 monclient: build_initial_monmap 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.388+0000 7f1984417640 10 monclient: monmap: 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:50.402 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.388+0000 7f1984417640 1 Processor -- start 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.388+0000 7f1984417640 1 -- start start 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.388+0000 7f1984417640 10 monclient: init 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient(hunting): picked mon.noname-a con 0x7f197c09b920 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient(hunting): picked mon.noname-c con 0x7f197c136da0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:50.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient(hunting): picked mon.noname-b con 0x7f197c13d320 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c13ccd0 con 0x7f197c09b920 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c12b7f0 con 0x7f197c13d320 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197c12bad0 con 0x7f197c136da0 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient(hunting): authenticate will time out at 1085.894237s 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f19819b2640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f197c136da0 legacy=0x7f197c1d08f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f19829b4640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f197c13d320 0x7f197c1d4020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58060/0 (socket says 172.21.15.171:58060) 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f19829b4640 1 -- 172.21.15.171:0/4102521325 learned_addr learned my addr 172.21.15.171:0/4102521325 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 806484421 0 0) 0x7f197c12b7f0 con 0x7f197c13d320 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:50.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): my global_id is 6112 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1960003520 con 0x7f197c13d320 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608258804 0 0) 0x7f197c13ccd0 con 0x7f197c09b920 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): my global_id is 64545 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f197c12b7f0 con 0x7f197c09b920 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2504850480 0 0) 0x7f1960003520 con 0x7f197c13d320 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f197c13ccd0 con 0x7f197c13d320 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19740043b0 con 0x7f197c13d320 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:50.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:50.406 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3945535290 0 0) 0x7f197c12b7f0 con 0x7f197c09b920 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1960003520 con 0x7f197c09b920 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f19780018b0 con 0x7f197c09b920 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:50.407 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 13260186 0 0) 0x7f197c13ccd0 con 0x7f197c13d320 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 >> v1:172.21.15.148:6790/0 conn(0x7f197c136da0 legacy=0x7f197c1d08f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 >> v1:172.21.15.148:6789/0 conn(0x7f197c09b920 legacy=0x7f197c0982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:50.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 monclient: found mon.b 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c12c9e0 con 0x7f197c13d320 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 5 monclient: authenticate success, global_id 6112 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient: _renew_subs 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f197c12a900 con 0x7f197c13d320 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient: _renew_subs 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.389+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f197c1343e0 con 0x7f197c13d320 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1974004660 con 0x7f197c13d320 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1974004a70 con 0x7f197c13d320 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f19809b0640 4 set_mon_vals no callback set 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:50.410 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1974004d10 con 0x7f197c13d320 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1984417640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1984417640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1984417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.390+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1964005080 con 0x7f197c13d320 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.391+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(209..209 src has 1..209) ==== 12866+0+0 (unknown 4012520591 0 0) 0x7f19740494a0 con 0x7f197c13d320 2024-09-16T04:11:50.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.391+0000 7f19821b3640 10 monclient: get_auth_request con 0x7f1960038ea0 auth_method 0 2024-09-16T04:11:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.395+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1974015f20 con 0x7f197c13d320 2024-09-16T04:11:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.395+0000 7f1972ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:50.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.395+0000 7f1972ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:50.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.561+0000 7f1984417640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:50.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.561+0000 7f1984417640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:50.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.561+0000 7f1984417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:50.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:50.561+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f1964005370 con 0x7f197c13d320 2024-09-16T04:11:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.189+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v210) ==== 168+0+0 (unknown 871542576 0 0) 0x7f1974015820 con 0x7f197c13d320 2024-09-16T04:11:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.189+0000 7f1972ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:51.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.189+0000 7f1972ffd640 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-16T04:11:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.291+0000 7f1984417640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.291+0000 7f1984417640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:51.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.291+0000 7f1984417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.291+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f1964005ab0 con 0x7f197c13d320 2024-09-16T04:11:51.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.292+0000 7f1972ffd640 1 -- 172.21.15.171:0/4102521325 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v210) ==== 147+0+0 (unknown 3470453224 0 0) 0x7f1974019b70 con 0x7f197c13d320 2024-09-16T04:11:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.292+0000 7f1972ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.292+0000 7f1972ffd640 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-16T04:11:51.295 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h27' already exists 2024-09-16T04:11:51.296 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 >> v1:172.21.15.171:6812/7327273 conn(0x7f1960038ea0 legacy=0x7f196003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 10 monclient: shutdown 2024-09-16T04:11:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 >> v1:172.21.15.171:6789/0 conn(0x7f197c13d320 legacy=0x7f197c1d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 shutdown_connections 2024-09-16T04:11:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 >> 172.21.15.171:0/4102521325 conn(0x7f197c0a2150 msgr2=0x7f197c0a0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:51.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 shutdown_connections 2024-09-16T04:11:51.298 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.294+0000 7f1984417640 1 -- 172.21.15.171:0/4102521325 wait complete. 2024-09-16T04:11:51.306 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:51.306 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-16T04:11:51.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.385+0000 7fd6038f8640 10 monclient: get_monmap_and_config 2024-09-16T04:11:51.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.385+0000 7fd6038f8640 10 monclient: build_initial_monmap 2024-09-16T04:11:51.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.385+0000 7fd6038f8640 10 monclient: monmap: 2024-09-16T04:11:51.388 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:51.386220+0000 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:51.386220+0000 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:51.389 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:51.390 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.385+0000 7fd6038f8640 1 Processor -- start 2024-09-16T04:11:51.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 1 -- start start 2024-09-16T04:11:51.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient: init 2024-09-16T04:11:51.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:51.400 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:51.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5fc0a1fd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:51.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5fc0a23b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:51.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient(hunting): picked mon.noname-b con 0x7fd5fc139de0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:51.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc0a1490 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc09d1b0 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.401 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc09d490 con 0x7fd5fc139de0 2024-09-16T04:11:51.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:51.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd6038f8640 10 monclient(hunting): authenticate will time out at 1086.891695s 2024-09-16T04:11:51.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd601e95640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fd5fc139de0 legacy=0x7fd5fc13c1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:51.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd600e93640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd5fc0a23b0 0x7fd5fc1386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58076/0 (socket says 172.21.15.171:58076) 2024-09-16T04:11:51.402 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.386+0000 7fd600e93640 1 -- 172.21.15.171:0/1137631066 learned_addr learned my addr 172.21.15.171:0/1137631066 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:51.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115349084 0 0) 0x7fd5fc09d1b0 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:51.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:51.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:51.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): my global_id is 6115 2024-09-16T04:11:51.403 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5e8003520 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 359449391 0 0) 0x7fd5fc0a1490 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:51.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:51.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:51.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient(hunting): my global_id is 64548 2024-09-16T04:11:51.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5fc09d1b0 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 608689332 0 0) 0x7fd5e8003520 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 >> v1:172.21.15.148:6790/0 conn(0x7fd5fc139de0 legacy=0x7fd5fc13c1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:51.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 >> v1:172.21.15.148:6789/0 conn(0x7fd5fc0a1fd0 legacy=0x7fd5fc09de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:51.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:51.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 monclient: found mon.noname-c 2024-09-16T04:11:51.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:51.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc09bca0 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:51.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:51.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:51.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd5ec001610 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 5 monclient: authenticate success, global_id 6115 2024-09-16T04:11:51.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:51.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:51.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:51.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: dump: 2024-09-16T04:11:51.408 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:51.408 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:51.408 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.409 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.409 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:51.409 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.409 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:51.409 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:51.409 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:51.410 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:51.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5ec001870 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:51.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 1 -- 172.21.15.171:0/1137631066 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd5ec001c40 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:51.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 4 set_mon_vals no callback set 2024-09-16T04:11:51.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:51.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:51.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd5f3fff640 10 monclient: dump: 2024-09-16T04:11:51.411 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:51.411 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:51.412 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.412 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.412 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:51.412 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.413 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:51.413 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:51.413 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:51.414 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 10 monclient: shutdown 2024-09-16T04:11:51.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:51.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 1 -- 172.21.15.171:0/1137631066 >> v1:172.21.15.171:6789/0 conn(0x7fd5fc0a23b0 legacy=0x7fd5fc1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:51.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 1 -- 172.21.15.171:0/1137631066 shutdown_connections 2024-09-16T04:11:51.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 1 -- 172.21.15.171:0/1137631066 >> 172.21.15.171:0/1137631066 conn(0x7fd5fc0a5450 msgr2=0x7fd5fc0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:51.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.387+0000 7fd6038f8640 1 -- 172.21.15.171:0/1137631066 shutdown_connections 2024-09-16T04:11:51.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.388+0000 7fd6038f8640 1 -- 172.21.15.171:0/1137631066 wait complete. 2024-09-16T04:11:51.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.388+0000 7fd6038f8640 10 monclient: build_initial_monmap 2024-09-16T04:11:51.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.388+0000 7fd6038f8640 10 monclient: monmap: 2024-09-16T04:11:51.416 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:51.416 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:51.416 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:51.416 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:51.416 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:51.417 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.417 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:51.417 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:51.417 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:51.418 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.388+0000 7fd6038f8640 1 Processor -- start 2024-09-16T04:11:51.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.388+0000 7fd6038f8640 1 -- start start 2024-09-16T04:11:51.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.388+0000 7fd6038f8640 10 monclient: init 2024-09-16T04:11:51.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:51.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:51.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5fc0a1fd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:51.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5fc0a23b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:51.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient(hunting): picked mon.noname-b con 0x7fd5fc139de0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:51.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc0a37b0 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc11cb40 con 0x7fd5fc139de0 2024-09-16T04:11:51.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5fc11ce20 con 0x7fd5fc0a23b0 2024-09-16T04:11:51.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:51.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd6038f8640 10 monclient(hunting): authenticate will time out at 1086.894362s 2024-09-16T04:11:51.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd600e93640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fd5fc0a23b0 legacy=0x7fd5fc1cc190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:51.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd601e95640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd5fc139de0 0x7fd5fc1cf8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58086/0 (socket says 172.21.15.171:58086) 2024-09-16T04:11:51.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd601e95640 1 -- 172.21.15.171:0/3977892564 learned_addr learned my addr 172.21.15.171:0/3977892564 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:51.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011598599 0 0) 0x7fd5fc11cb40 con 0x7fd5fc139de0 2024-09-16T04:11:51.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:51.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:51.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:51.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): my global_id is 6118 2024-09-16T04:11:51.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5e0003520 con 0x7fd5fc139de0 2024-09-16T04:11:51.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420616598 0 0) 0x7fd5fc09d1b0 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:51.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:51.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:51.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): my global_id is 64551 2024-09-16T04:11:51.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5fc11cb40 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2260013107 0 0) 0x7fd5e0003520 con 0x7fd5fc139de0 2024-09-16T04:11:51.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5fc09d1b0 con 0x7fd5fc139de0 2024-09-16T04:11:51.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd5f4004440 con 0x7fd5fc139de0 2024-09-16T04:11:51.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): dump: 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.424 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:51.425 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.425 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:51.425 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:51.425 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:51.425 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 567184193 0 0) 0x7fd5fc11cb40 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd5e0003520 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd5f80018b0 con 0x7fd5fc0a1fd0 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 10 monclient(hunting): dump: 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:51.426 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:51.427 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.389+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2352660685 0 0) 0x7fd5fc09d1b0 con 0x7fd5fc139de0 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 >> v1:172.21.15.148:6790/0 conn(0x7fd5fc0a23b0 legacy=0x7fd5fc1cc190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 >> v1:172.21.15.148:6789/0 conn(0x7fd5fc0a1fd0 legacy=0x7fd5fc132350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 monclient: found mon.b 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc0a31c0 con 0x7fd5fc139de0 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:51.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 5 monclient: authenticate success, global_id 6118 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 10 monclient: _renew_subs 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc11c860 con 0x7fd5fc139de0 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd5f40046a0 con 0x7fd5fc139de0 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd5f40049f0 con 0x7fd5fc139de0 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:51.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 10 monclient: dump: 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:51.430 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:51.431 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:51.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f37fe640 4 set_mon_vals no callback set 2024-09-16T04:11:51.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 10 monclient: _renew_subs 2024-09-16T04:11:51.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd5fc138450 con 0x7fd5fc139de0 2024-09-16T04:11:51.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.390+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd5f4004c10 con 0x7fd5fc139de0 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.391+0000 7fd6038f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.391+0000 7fd6038f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.391+0000 7fd6038f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.391+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc0008d0 con 0x7fd5fc139de0 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.391+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(210..210 src has 1..210) ==== 13079+0+0 (unknown 3827316470 0 0) 0x7fd5f40495b0 con 0x7fd5fc139de0 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.391+0000 7fd601694640 10 monclient: get_auth_request con 0x7fd5e0038f40 auth_method 0 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.395+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd5f4015f60 con 0x7fd5fc139de0 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.395+0000 7fd5f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:51.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.395+0000 7fd5f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:51.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.562+0000 7fd6038f8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:51.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.562+0000 7fd6038f8640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:51.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.562+0000 7fd6038f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:51.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:51.562+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fd5fc138740 con 0x7fd5fc139de0 2024-09-16T04:11:52.199 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.197+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v211) ==== 168+0+0 (unknown 2994685049 0 0) 0x7fd5f4019bb0 con 0x7fd5fc139de0 2024-09-16T04:11:52.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.197+0000 7fd5f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:52.200 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.197+0000 7fd5f1ffb640 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-16T04:11:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.302+0000 7fd6038f8640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.302+0000 7fd6038f8640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.302+0000 7fd6038f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.302+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fd5fc1d0fe0 con 0x7fd5fc139de0 2024-09-16T04:11:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.303+0000 7fd5f1ffb640 1 -- 172.21.15.171:0/3977892564 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v211) ==== 147+0+0 (unknown 2144435356 0 0) 0x7fd5f4015220 con 0x7fd5fc139de0 2024-09-16T04:11:52.304 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.303+0000 7fd5f1ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.303+0000 7fd5f1ffb640 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-16T04:11:52.305 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h28' already exists 2024-09-16T04:11:52.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 >> v1:172.21.15.171:6812/7327273 conn(0x7fd5e0038f40 legacy=0x7fd5e003b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 10 monclient: shutdown 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 >> v1:172.21.15.171:6789/0 conn(0x7fd5fc139de0 legacy=0x7fd5fc1cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 shutdown_connections 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 >> 172.21.15.171:0/3977892564 conn(0x7fd5fc0a5450 msgr2=0x7fd5fc06e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 shutdown_connections 2024-09-16T04:11:52.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.305+0000 7fd6038f8640 1 -- 172.21.15.171:0/3977892564 wait complete. 2024-09-16T04:11:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-16T04:11:52.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.395+0000 7fa999366640 10 monclient: get_monmap_and_config 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.396+0000 7fa999366640 10 monclient: build_initial_monmap 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.396+0000 7fa999366640 10 monclient: monmap: 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:52.396809+0000 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:52.396809+0000 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:52.398 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.396+0000 7fa999366640 1 Processor -- start 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.396+0000 7fa999366640 1 -- start start 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.396+0000 7fa999366640 10 monclient: init 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient(hunting): picked mon.noname-a con 0x7fa994136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:52.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient(hunting): picked mon.noname-b con 0x7fa994137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient(hunting): picked mon.noname-c con 0x7fa9941375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99409b2c0 con 0x7fa994136d60 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99409b5a0 con 0x7fa9941375a0 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9940a6640 con 0x7fa994137140 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient(hunting): authenticate will time out at 1087.902164s 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa9927fc640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa994137140 legacy=0x7fa99413bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa9937fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa9941375a0 0x7fa99413f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58100/0 (socket says 172.21.15.171:58100) 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa9937fe640 1 -- 172.21.15.171:0/825230074 learned_addr learned my addr 172.21.15.171:0/825230074 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3920619662 0 0) 0x7fa99409b5a0 con 0x7fa9941375a0 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:52.406 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient(hunting): my global_id is 6121 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa97c003520 con 0x7fa9941375a0 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 831001532 0 0) 0x7fa97c003520 con 0x7fa9941375a0 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 >> v1:172.21.15.148:6790/0 conn(0x7fa994137140 legacy=0x7fa99413bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 >> v1:172.21.15.148:6789/0 conn(0x7fa994136d60 legacy=0x7fa99409bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 monclient: found mon.noname-c 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa994143b20 con 0x7fa9941375a0 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:52.407 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 5 monclient: authenticate success, global_id 6121 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa984001990 con 0x7fa9941375a0 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: dump: 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:52.408 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa984001c80 con 0x7fa9941375a0 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 1 -- 172.21.15.171:0/825230074 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa984002050 con 0x7fa9941375a0 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa999366640 4 set_mon_vals no callback set 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.397+0000 7fa991ffb640 10 monclient: dump: 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:52.409 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 10 monclient: shutdown 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 1 -- 172.21.15.171:0/825230074 >> v1:172.21.15.171:6789/0 conn(0x7fa9941375a0 legacy=0x7fa99413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 1 -- 172.21.15.171:0/825230074 shutdown_connections 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 1 -- 172.21.15.171:0/825230074 >> 172.21.15.171:0/825230074 conn(0x7fa9940a9ae0 msgr2=0x7fa9940a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 1 -- 172.21.15.171:0/825230074 shutdown_connections 2024-09-16T04:11:52.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 1 -- 172.21.15.171:0/825230074 wait complete. 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 10 monclient: build_initial_monmap 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.398+0000 7fa999366640 10 monclient: monmap: 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 1 Processor -- start 2024-09-16T04:11:52.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 1 -- start start 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient: init 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient(hunting): picked mon.noname-a con 0x7fa994136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient(hunting): picked mon.noname-c con 0x7fa994137140 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient(hunting): picked mon.noname-b con 0x7fa9941375a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99412e6d0 con 0x7fa994136d60 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9940a3320 con 0x7fa9941375a0 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9940a2430 con 0x7fa994137140 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa999366640 10 monclient(hunting): authenticate will time out at 1087.904927s 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa9927fc640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa994137140 legacy=0x7fa9940a1d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:52.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa9937fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa9941375a0 0x7fa9941d3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58102/0 (socket says 172.21.15.171:58102) 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.399+0000 7fa9937fe640 1 -- 172.21.15.171:0/3469735042 learned_addr learned my addr 172.21.15.171:0/3469735042 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784097745 0 0) 0x7fa9940a3320 con 0x7fa9941375a0 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): my global_id is 6124 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa980003520 con 0x7fa9941375a0 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423646357 0 0) 0x7fa99409b2c0 con 0x7fa994136d60 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): my global_id is 64557 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9940a3320 con 0x7fa994136d60 2024-09-16T04:11:52.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 513762772 0 0) 0x7fa980003520 con 0x7fa9941375a0 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa99409b2c0 con 0x7fa9941375a0 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9840017b0 con 0x7fa9941375a0 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:52.414 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 315047044 0 0) 0x7fa99409b2c0 con 0x7fa9941375a0 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 >> v1:172.21.15.148:6790/0 conn(0x7fa994137140 legacy=0x7fa9940a1d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 >> v1:172.21.15.148:6789/0 conn(0x7fa994136d60 legacy=0x7fa9940a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 monclient: found mon.b 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9940a26b0 con 0x7fa9941375a0 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:52.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 5 monclient: authenticate success, global_id 6124 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 10 monclient: _renew_subs 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9940a03b0 con 0x7fa9941375a0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 10 monclient: _renew_subs 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa994136730 con 0x7fa9941375a0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa984001c80 con 0x7fa9941375a0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9840051e0 con 0x7fa9941375a0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:52.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa9917fa640 4 set_mon_vals no callback set 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.400+0000 7fa97b7fe640 10 monclient: dump: 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa984010610 con 0x7fa9941375a0 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa999366640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:52.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa999366640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa999366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa974005080 con 0x7fa9941375a0 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(211..211 src has 1..211) ==== 13292+0+0 (unknown 2737055759 0 0) 0x7fa984049780 con 0x7fa9941375a0 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.401+0000 7fa992ffd640 10 monclient: get_auth_request con 0x7fa980038fb0 auth_method 0 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.406+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa984016170 con 0x7fa9941375a0 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.406+0000 7fa97b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:52.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.406+0000 7fa97b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:52.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.571+0000 7fa999366640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:52.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.571+0000 7fa999366640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:52.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.571+0000 7fa999366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:52.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:52.571+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fa974005370 con 0x7fa9941375a0 2024-09-16T04:11:53.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.199+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v212) ==== 168+0+0 (unknown 467631842 0 0) 0x7fa984015a70 con 0x7fa9941375a0 2024-09-16T04:11:53.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.199+0000 7fa97b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:53.201 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.199+0000 7fa97b7fe640 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-16T04:11:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.300+0000 7fa999366640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:53.302 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.301+0000 7fa999366640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:53.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.301+0000 7fa999366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.301+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7fa974005ab0 con 0x7fa9941375a0 2024-09-16T04:11:53.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.302+0000 7fa97b7fe640 1 -- 172.21.15.171:0/3469735042 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v212) ==== 147+0+0 (unknown 261254474 0 0) 0x7fa984013dd0 con 0x7fa9941375a0 2024-09-16T04:11:53.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.302+0000 7fa97b7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:53.303 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.302+0000 7fa97b7fe640 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-16T04:11:53.303 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h29' already exists 2024-09-16T04:11:53.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.303+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 >> v1:172.21.15.171:6812/7327273 conn(0x7fa980038fb0 legacy=0x7fa98003b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:53.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.303+0000 7fa999366640 10 monclient: shutdown 2024-09-16T04:11:53.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.303+0000 7fa999366640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:53.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.303+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 >> v1:172.21.15.171:6789/0 conn(0x7fa9941375a0 legacy=0x7fa9941d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:53.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.303+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 shutdown_connections 2024-09-16T04:11:53.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.303+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 >> 172.21.15.171:0/3469735042 conn(0x7fa9940a9ae0 msgr2=0x7fa994072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:53.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.304+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 shutdown_connections 2024-09-16T04:11:53.307 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.304+0000 7fa999366640 1 -- 172.21.15.171:0/3469735042 wait complete. 2024-09-16T04:11:53.315 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:53.315 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.394+0000 7f735025e640 10 monclient: get_monmap_and_config 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.394+0000 7f735025e640 10 monclient: build_initial_monmap 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.394+0000 7f735025e640 10 monclient: monmap: 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:53.394953+0000 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:53.394953+0000 2024-09-16T04:11:53.396 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:53.397 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:53.397 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:53.397 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:53.397 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:53.397 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:53.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.394+0000 7f735025e640 1 Processor -- start 2024-09-16T04:11:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 1 -- start start 2024-09-16T04:11:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient: init 2024-09-16T04:11:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:53.408 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient(hunting): picked mon.noname-a con 0x7f7348136d60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient(hunting): picked mon.noname-c con 0x7f7348137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient(hunting): picked mon.noname-b con 0x7f73481375a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f734809b2c0 con 0x7f7348136d60 2024-09-16T04:11:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f734809b5a0 con 0x7f7348137140 2024-09-16T04:11:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73480a6640 con 0x7f73481375a0 2024-09-16T04:11:53.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f735025e640 10 monclient(hunting): authenticate will time out at 1088.900548s 2024-09-16T04:11:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734e7fb640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f73481375a0 legacy=0x7f734813f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734d7f9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7348137140 0x7f734813bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58114/0 (socket says 172.21.15.171:58114) 2024-09-16T04:11:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734d7f9640 1 -- 172.21.15.171:0/3721236793 learned_addr learned my addr 172.21.15.171:0/3721236793 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:53.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2099557828 0 0) 0x7f734809b5a0 con 0x7f7348137140 2024-09-16T04:11:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734cff8640 10 monclient(hunting): _negotiate 2024-09-16T04:11:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734cff8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734cff8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734cff8640 10 monclient(hunting): my global_id is 6127 2024-09-16T04:11:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.395+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7334003520 con 0x7f7348137140 2024-09-16T04:11:53.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621127054 0 0) 0x7f734809b2c0 con 0x7f7348136d60 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient(hunting): _negotiate 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient(hunting): my global_id is 64560 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f734809b5a0 con 0x7f7348136d60 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 128663211 0 0) 0x7f7334003520 con 0x7f7348137140 2024-09-16T04:11:53.412 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 >> v1:172.21.15.148:6790/0 conn(0x7f73481375a0 legacy=0x7f734813f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 >> v1:172.21.15.148:6789/0 conn(0x7f7348136d60 legacy=0x7f734809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: _finish_hunting 0 2024-09-16T04:11:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 monclient: found mon.noname-c 2024-09-16T04:11:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7348143b20 con 0x7f7348137140 2024-09-16T04:11:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: _finish_auth 0 2024-09-16T04:11:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: _check_auth_tickets 2024-09-16T04:11:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7338001610 con 0x7f7348137140 2024-09-16T04:11:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 5 monclient: authenticate success, global_id 6127 2024-09-16T04:11:53.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: dump: 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.415 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.416 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:53.416 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:53.416 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:53.416 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:53.417 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:53.417 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:53.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:53.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73380018c0 con 0x7f7348137140 2024-09-16T04:11:53.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:53.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 1 -- 172.21.15.171:0/3721236793 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7338001c90 con 0x7f7348137140 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 4 set_mon_vals no callback set 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f734cff8640 10 monclient: dump: 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:53.418 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:53.419 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.419 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.419 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:53.419 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:53.419 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:53.419 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:53.420 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:53.420 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:53.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 10 monclient: shutdown 2024-09-16T04:11:53.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:53.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 1 -- 172.21.15.171:0/3721236793 >> v1:172.21.15.171:6789/0 conn(0x7f7348137140 legacy=0x7f734813bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:53.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 1 -- 172.21.15.171:0/3721236793 shutdown_connections 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 1 -- 172.21.15.171:0/3721236793 >> 172.21.15.171:0/3721236793 conn(0x7f73480a9ae0 msgr2=0x7f73480a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 1 -- 172.21.15.171:0/3721236793 shutdown_connections 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.396+0000 7f735025e640 1 -- 172.21.15.171:0/3721236793 wait complete. 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.397+0000 7f735025e640 10 monclient: build_initial_monmap 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.397+0000 7f735025e640 10 monclient: monmap: 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:53.421 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:53.422 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:53.422 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:53.422 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:53.422 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:53.422 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:53.422 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.397+0000 7f735025e640 1 Processor -- start 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.397+0000 7f735025e640 1 -- start start 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.397+0000 7f735025e640 10 monclient: init 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:53.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient(hunting): picked mon.noname-c con 0x7f7348136d60 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:53.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient(hunting): picked mon.noname-b con 0x7f7348137140 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:53.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient(hunting): picked mon.noname-a con 0x7f73481375a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:53.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73480a03b0 con 0x7f73481375a0 2024-09-16T04:11:53.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7348130c20 con 0x7f7348137140 2024-09-16T04:11:53.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f734dffa640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7348136d60 legacy=0x7f734812dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:53.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73480a3450 con 0x7f7348136d60 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient(hunting): authenticate will time out at 1088.903394s 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f734d7f9640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7348137140 0x7f734812e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58126/0 (socket says 172.21.15.171:58126) 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f734d7f9640 1 -- 172.21.15.171:0/1823411213 learned_addr learned my addr 172.21.15.171:0/1823411213 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070965593 0 0) 0x7f7348130c20 con 0x7f7348137140 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:53.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:53.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:53.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): my global_id is 6130 2024-09-16T04:11:53.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f732c003520 con 0x7f7348137140 2024-09-16T04:11:53.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914973147 0 0) 0x7f7340003d70 con 0x7f73481375a0 2024-09-16T04:11:53.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:11:53.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:53.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:53.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): my global_id is 64563 2024-09-16T04:11:53.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7348130c20 con 0x7f73481375a0 2024-09-16T04:11:53.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3489541257 0 0) 0x7f732c003520 con 0x7f7348137140 2024-09-16T04:11:53.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7340003d70 con 0x7f7348137140 2024-09-16T04:11:53.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7338001c10 con 0x7f7348137140 2024-09-16T04:11:53.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:53.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:53.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): dump: 2024-09-16T04:11:53.428 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:53.429 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:53.429 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.429 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.429 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:53.430 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:53.430 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:53.430 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:53.430 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:53.430 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:53.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2446721933 0 0) 0x7f7340003d70 con 0x7f7348137140 2024-09-16T04:11:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 >> v1:172.21.15.148:6790/0 conn(0x7f7348136d60 legacy=0x7f734812dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 >> v1:172.21.15.148:6789/0 conn(0x7f73481375a0 legacy=0x7f73481d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 monclient: found mon.b 2024-09-16T04:11:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7348130940 con 0x7f7348137140 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f733e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 5 monclient: authenticate success, global_id 6130 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient: _renew_subs 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.398+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f734812f7d0 con 0x7f7348137140 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f735025e640 10 monclient: _renew_subs 2024-09-16T04:11:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f735025e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7348136400 con 0x7f7348137140 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73380018c0 con 0x7f7348137140 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7338004de0 con 0x7f7348137140 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733ffff640 4 set_mon_vals no callback set 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 10 monclient: dump: 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:53.433 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.399+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7338010210 con 0x7f7348137140 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.400+0000 7f735025e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.400+0000 7f735025e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.400+0000 7f735025e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.400+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7330005080 con 0x7f7348137140 2024-09-16T04:11:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.400+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(212..212 src has 1..212) ==== 13505+0+0 (unknown 53625298 0 0) 0x7f7338049560 con 0x7f7348137140 2024-09-16T04:11:53.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.400+0000 7f734dffa640 10 monclient: get_auth_request con 0x7f732c038b90 auth_method 0 2024-09-16T04:11:53.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.404+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7338019990 con 0x7f7348137140 2024-09-16T04:11:53.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.404+0000 7f733e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:53.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.404+0000 7f733e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:53.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.571+0000 7f735025e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:53.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.571+0000 7f735025e640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:53.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.571+0000 7f735025e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:53.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:53.571+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7330005370 con 0x7f7348137140 2024-09-16T04:11:54.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.208+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v213) ==== 168+0+0 (unknown 2451073888 0 0) 0x7f733801e250 con 0x7f7348137140 2024-09-16T04:11:54.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.208+0000 7f733e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:54.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.208+0000 7f733e7fc640 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-16T04:11:54.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.310+0000 7f735025e640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:54.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.310+0000 7f735025e640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:54.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.310+0000 7f735025e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.312 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.310+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7330005ab0 con 0x7f7348137140 2024-09-16T04:11:54.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.311+0000 7f733e7fc640 1 -- 172.21.15.171:0/1823411213 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v213) ==== 147+0+0 (unknown 1877499307 0 0) 0x7f7338013a80 con 0x7f7348137140 2024-09-16T04:11:54.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.311+0000 7f733e7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:54.313 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.311+0000 7f733e7fc640 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-16T04:11:54.313 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h30' already exists 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 >> v1:172.21.15.171:6812/7327273 conn(0x7f732c038b90 legacy=0x7f732c03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 10 monclient: shutdown 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 >> v1:172.21.15.171:6789/0 conn(0x7f7348137140 legacy=0x7f734812e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 shutdown_connections 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 >> 172.21.15.171:0/1823411213 conn(0x7f73480a9ae0 msgr2=0x7f7348072420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:54.315 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 shutdown_connections 2024-09-16T04:11:54.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.313+0000 7f735025e640 1 -- 172.21.15.171:0/1823411213 wait complete. 2024-09-16T04:11:54.324 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:54.325 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-16T04:11:54.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.402+0000 7f3c71217640 10 monclient: get_monmap_and_config 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.402+0000 7f3c71217640 10 monclient: build_initial_monmap 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.402+0000 7f3c71217640 10 monclient: monmap: 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:54.403750+0000 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:54.403750+0000 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.405 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:54.406 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:54.406 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:54.406 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.403+0000 7f3c71217640 1 Processor -- start 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.403+0000 7f3c71217640 1 -- start start 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.403+0000 7f3c71217640 10 monclient: init 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c09f7c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6c09fba0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6c1395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:54.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c0a1490 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c09d1b0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c09d490 con 0x7f3c6c09fba0 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient(hunting): authenticate will time out at 1089.909251s 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c63fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3c6c09fba0 legacy=0x7f3c6c0a0030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6b59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c6c1395d0 0x7f3c6c13b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58134/0 (socket says 172.21.15.171:58134) 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6b59e640 1 -- 172.21.15.171:0/1817357584 learned_addr learned my addr 172.21.15.171:0/1817357584 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116797563 0 0) 0x7f3c6c09d1b0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient(hunting): _negotiate 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient(hunting): my global_id is 6133 2024-09-16T04:11:54.415 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c58003440 con 0x7f3c6c1395d0 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 431703105 0 0) 0x7f3c58003440 con 0x7f3c6c1395d0 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 >> v1:172.21.15.148:6790/0 conn(0x7f3c6c09fba0 legacy=0x7f3c6c0a0030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 >> v1:172.21.15.148:6789/0 conn(0x7f3c6c09f7c0 legacy=0x7f3c6c09de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 monclient: found mon.noname-c 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c09bca0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: _finish_auth 0 2024-09-16T04:11:54.416 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: _check_auth_tickets 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c64001990 con 0x7f3c6c1395d0 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 5 monclient: authenticate success, global_id 6133 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: dump: 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:54.417 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c64001cf0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 1 -- 172.21.15.171:0/1817357584 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c640020c0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c71217640 4 set_mon_vals no callback set 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:54.418 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.404+0000 7f3c6a59c640 10 monclient: dump: 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 10 monclient: shutdown 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:54.419 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 1 -- 172.21.15.171:0/1817357584 >> v1:172.21.15.171:6789/0 conn(0x7f3c6c1395d0 legacy=0x7f3c6c13b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 1 -- 172.21.15.171:0/1817357584 shutdown_connections 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 1 -- 172.21.15.171:0/1817357584 >> 172.21.15.171:0/1817357584 conn(0x7f3c6c0a5450 msgr2=0x7f3c6c0a18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 1 -- 172.21.15.171:0/1817357584 shutdown_connections 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.405+0000 7f3c71217640 1 -- 172.21.15.171:0/1817357584 wait complete. 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient: build_initial_monmap 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient: monmap: 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:54.420 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 1 Processor -- start 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 1 -- start start 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient: init 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c09f7c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6c09fba0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6c1395d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c1277b0 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c127a90 con 0x7f3c6c1395d0 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c128050 con 0x7f3c6c09fba0 2024-09-16T04:11:54.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c71217640 10 monclient(hunting): authenticate will time out at 1089.911928s 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.406+0000 7f3c63fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3c6c09fba0 legacy=0x7f3c6c1d4c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c6b59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c6c1395d0 0x7f3c6c1d8350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58136/0 (socket says 172.21.15.171:58136) 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c6b59e640 1 -- 172.21.15.171:0/1149099607 learned_addr learned my addr 172.21.15.171:0/1149099607 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676453660 0 0) 0x7f3c6c127a90 con 0x7f3c6c1395d0 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): my global_id is 6136 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c4c003520 con 0x7f3c6c1395d0 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666107852 0 0) 0x7f3c6c1277b0 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:54.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): my global_id is 64569 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c6c127a90 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3713232111 0 0) 0x7f3c4c003520 con 0x7f3c6c1395d0 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c6c1277b0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c64001950 con 0x7f3c6c1395d0 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): dump: 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.423 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 148104784 0 0) 0x7f3c6c127a90 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c4c003520 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c5c001830 con 0x7f3c6c09f7c0 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:54.424 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient(hunting): dump: 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:54.425 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1523061847 0 0) 0x7f3c6c1277b0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 >> v1:172.21.15.148:6790/0 conn(0x7f3c6c09fba0 legacy=0x7f3c6c1d4c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 >> v1:172.21.15.148:6789/0 conn(0x7f3c6c09f7c0 legacy=0x7f3c6c1cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 monclient: found mon.b 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c0a4150 con 0x7f3c6c1395d0 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 5 monclient: authenticate success, global_id 6136 2024-09-16T04:11:54.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 10 monclient: _renew_subs 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c0a3cb0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 10 monclient: _renew_subs 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c6c126640 con 0x7f3c6c1395d0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c64001cf0 con 0x7f3c6c1395d0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c64005030 con 0x7f3c6c1395d0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c69d9b640 4 set_mon_vals no callback set 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.407+0000 7f3c637fe640 10 monclient: dump: 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:54.427 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c71217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c71217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c71217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c50005080 con 0x7f3c6c1395d0 2024-09-16T04:11:54.428 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3c64010460 con 0x7f3c6c1395d0 2024-09-16T04:11:54.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(213..213 src has 1..213) ==== 13718+0+0 (unknown 1829242888 0 0) 0x7f3c64049810 con 0x7f3c6c1395d0 2024-09-16T04:11:54.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.408+0000 7f3c6ad9d640 10 monclient: get_auth_request con 0x7f3c4c038ef0 auth_method 0 2024-09-16T04:11:54.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.413+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3c64016000 con 0x7f3c6c1395d0 2024-09-16T04:11:54.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.413+0000 7f3c637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:54.429 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.413+0000 7f3c637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:54.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.579+0000 7f3c71217640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:54.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.579+0000 7f3c71217640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:54.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.579+0000 7f3c71217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:54.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:54.579+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3c50005370 con 0x7f3c6c1395d0 2024-09-16T04:11:55.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.209+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v214) ==== 168+0+0 (unknown 1581209104 0 0) 0x7f3c64015900 con 0x7f3c6c1395d0 2024-09-16T04:11:55.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.209+0000 7f3c637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.209+0000 7f3c637fe640 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-16T04:11:55.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.320+0000 7f3c71217640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.320+0000 7f3c71217640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.320+0000 7f3c71217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.322 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.320+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f3c50005bd0 con 0x7f3c6c1395d0 2024-09-16T04:11:55.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.321+0000 7f3c637fe640 1 -- 172.21.15.171:0/1149099607 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v214) ==== 147+0+0 (unknown 1292714927 0 0) 0x7f3c6401e450 con 0x7f3c6c1395d0 2024-09-16T04:11:55.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.321+0000 7f3c637fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.323 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.321+0000 7f3c637fe640 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-16T04:11:55.323 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h31' already exists 2024-09-16T04:11:55.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.323+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 >> v1:172.21.15.171:6812/7327273 conn(0x7f3c4c038ef0 legacy=0x7f3c4c03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.323+0000 7f3c71217640 10 monclient: shutdown 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.323+0000 7f3c71217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.323+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 >> v1:172.21.15.171:6789/0 conn(0x7f3c6c1395d0 legacy=0x7f3c6c1d8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.324+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 shutdown_connections 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.324+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 >> 172.21.15.171:0/1149099607 conn(0x7f3c6c0a5450 msgr2=0x7f3c6c0a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.324+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 shutdown_connections 2024-09-16T04:11:55.326 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.324+0000 7f3c71217640 1 -- 172.21.15.171:0/1149099607 wait complete. 2024-09-16T04:11:55.335 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:55.335 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-16T04:11:55.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.415+0000 7f7d20d0c640 10 monclient: get_monmap_and_config 2024-09-16T04:11:55.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.415+0000 7f7d20d0c640 10 monclient: build_initial_monmap 2024-09-16T04:11:55.417 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.415+0000 7f7d20d0c640 10 monclient: monmap: 2024-09-16T04:11:55.417 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:55.416350+0000 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:55.416350+0000 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:55.418 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:55.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 1 Processor -- start 2024-09-16T04:11:55.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 1 -- start start 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient: init 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1c09b920 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient(hunting): picked mon.noname-a con 0x7f7d1c136da0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient(hunting): picked mon.noname-b con 0x7f7d1c13d320 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c073a20 con 0x7f7d1c136da0 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c099920 con 0x7f7d1c09b920 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c099c00 con 0x7f7d1c13d320 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d20d0c640 10 monclient(hunting): authenticate will time out at 1090.921747s 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d1a59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7d1c09b920 0x7f7d1c09bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58150/0 (socket says 172.21.15.171:58150) 2024-09-16T04:11:55.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d1a59c640 1 -- 172.21.15.171:0/487083016 learned_addr learned my addr 172.21.15.171:0/487083016 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.416+0000 7f7d1ad9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7d1c13d320 legacy=0x7f7d1c13f710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369668283 0 0) 0x7f7d1c099920 con 0x7f7d1c09b920 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): my global_id is 6139 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d04003520 con 0x7f7d1c09b920 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2562380047 0 0) 0x7f7d1c073a20 con 0x7f7d1c136da0 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): _negotiate 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:55.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient(hunting): my global_id is 64572 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d1c099920 con 0x7f7d1c136da0 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1816677266 0 0) 0x7f7d04003520 con 0x7f7d1c09b920 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 >> v1:172.21.15.148:6790/0 conn(0x7f7d1c13d320 legacy=0x7f7d1c13f710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 >> v1:172.21.15.148:6789/0 conn(0x7f7d1c136da0 legacy=0x7f7d1c13bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: _finish_hunting 0 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 monclient: found mon.noname-c 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c099ee0 con 0x7f7d1c09b920 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: _finish_auth 0 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: _check_auth_tickets 2024-09-16T04:11:55.433 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d0c001610 con 0x7f7d1c09b920 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 5 monclient: authenticate success, global_id 6139 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: dump: 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:55.434 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d0c0018c0 con 0x7f7d1c09b920 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 1 -- 172.21.15.171:0/487083016 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d0c001c90 con 0x7f7d1c09b920 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 4 set_mon_vals no callback set 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d1959a640 10 monclient: dump: 2024-09-16T04:11:55.435 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 10 monclient: shutdown 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.417+0000 7f7d20d0c640 1 -- 172.21.15.171:0/487083016 >> v1:172.21.15.171:6789/0 conn(0x7f7d1c09b920 legacy=0x7f7d1c09bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.418+0000 7f7d20d0c640 1 -- 172.21.15.171:0/487083016 shutdown_connections 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.418+0000 7f7d20d0c640 1 -- 172.21.15.171:0/487083016 >> 172.21.15.171:0/487083016 conn(0x7f7d1c0a2150 msgr2=0x7f7d1c0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.418+0000 7f7d20d0c640 1 -- 172.21.15.171:0/487083016 shutdown_connections 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.418+0000 7f7d20d0c640 1 -- 172.21.15.171:0/487083016 wait complete. 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.418+0000 7f7d20d0c640 10 monclient: build_initial_monmap 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.418+0000 7f7d20d0c640 10 monclient: monmap: 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:55.437 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 1 Processor -- start 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 1 -- start start 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient: init 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient(hunting): picked mon.noname-c con 0x7f7d1c09b920 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient(hunting): picked mon.noname-b con 0x7f7d1c136da0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient(hunting): picked mon.noname-a con 0x7f7d1c13d320 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c12b980 con 0x7f7d1c13d320 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c12bf40 con 0x7f7d1c136da0 2024-09-16T04:11:55.438 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d1a59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7d1c09b920 legacy=0x7f7d1c0986a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c12c220 con 0x7f7d1c09b920 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d20d0c640 10 monclient(hunting): authenticate will time out at 1090.924783s 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d19d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7d1c136da0 0x7f7d1c1d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58162/0 (socket says 172.21.15.171:58162) 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.419+0000 7f7d19d9b640 1 -- 172.21.15.171:0/3617125484 learned_addr learned my addr 172.21.15.171:0/3617125484 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263405872 0 0) 0x7f7d1c12bf40 con 0x7f7d1c136da0 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): my global_id is 6142 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cfc003520 con 0x7f7d1c136da0 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1977631401 0 0) 0x7f7d1c12b980 con 0x7f7d1c13d320 2024-09-16T04:11:55.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): my global_id is 64575 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d1c12bf40 con 0x7f7d1c13d320 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2721581622 0 0) 0x7f7cfc003520 con 0x7f7d1c136da0 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7d1c12b980 con 0x7f7d1c136da0 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d10001910 con 0x7f7d1c136da0 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): dump: 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 508239277 0 0) 0x7f7d1c12b980 con 0x7f7d1c136da0 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 >> v1:172.21.15.148:6790/0 conn(0x7f7d1c09b920 legacy=0x7f7d1c0986a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 >> v1:172.21.15.148:6789/0 conn(0x7f7d1c13d320 legacy=0x7f7d1c1d3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 monclient: found mon.b 2024-09-16T04:11:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c098db0 con 0x7f7d1c136da0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 5 monclient: authenticate success, global_id 6142 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 10 monclient: _renew_subs 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1c12aa90 con 0x7f7d1c136da0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 10 monclient: _renew_subs 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7d1c13c630 con 0x7f7d1c136da0 2024-09-16T04:11:55.442 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7d10003610 con 0x7f7d1c136da0 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7d10003960 con 0x7f7d1c136da0 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d18d99640 4 set_mon_vals no callback set 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.420+0000 7f7d0affd640 10 monclient: dump: 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7d10003b80 con 0x7f7d1c136da0 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d20d0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d20d0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d20d0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d00005080 con 0x7f7d1c136da0 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(214..214 src has 1..214) ==== 13931+0+0 (unknown 482805912 0 0) 0x7f7d100495f0 con 0x7f7d1c136da0 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.421+0000 7f7d1a59c640 10 monclient: get_auth_request con 0x7f7cfc038f60 auth_method 0 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.426+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7d10015cc0 con 0x7f7d1c136da0 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.426+0000 7f7d0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.426+0000 7f7d0affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:55.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.594+0000 7f7d20d0c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.594+0000 7f7d20d0c640 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.594+0000 7f7d20d0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.594+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7d00005370 con 0x7f7d1c136da0 2024-09-16T04:11:55.830 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.829+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v215) ==== 168+0+0 (unknown 1782407322 0 0) 0x7f7d100155c0 con 0x7f7d1c136da0 2024-09-16T04:11:55.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.829+0000 7f7d0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.831 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.829+0000 7f7d0affd640 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-16T04:11:55.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.931+0000 7f7d20d0c640 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.931+0000 7f7d20d0c640 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.931+0000 7f7d20d0c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:55.933 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.931+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) -- 0x7f7d00005ab0 con 0x7f7d1c136da0 2024-09-16T04:11:55.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.932+0000 7f7d0affd640 1 -- 172.21.15.171:0/3617125484 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v215) ==== 147+0+0 (unknown 2021944945 0 0) 0x7f7d1001e170 con 0x7f7d1c136da0 2024-09-16T04:11:55.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.932+0000 7f7d0affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-16T04:11:55.934 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.932+0000 7f7d0affd640 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-16T04:11:55.934 INFO:tasks.workunit.client.0.smithi171.stderr:bucket 'h32' already exists 2024-09-16T04:11:55.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 >> v1:172.21.15.171:6812/7327273 conn(0x7f7cfc038f60 legacy=0x7f7cfc03b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 10 monclient: shutdown 2024-09-16T04:11:55.936 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:55.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 >> v1:172.21.15.171:6789/0 conn(0x7f7d1c136da0 legacy=0x7f7d1c1d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:55.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 shutdown_connections 2024-09-16T04:11:55.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 >> 172.21.15.171:0/3617125484 conn(0x7f7d1c0a2150 msgr2=0x7f7d1c0a0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:55.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 shutdown_connections 2024-09-16T04:11:55.937 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:55.934+0000 7f7d20d0c640 1 -- 172.21.15.171:0/3617125484 wait complete. 2024-09-16T04:11:55.945 INFO:tasks.workunit.client.0.smithi171.stderr:++ seq 1 32 2024-09-16T04:11:55.946 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:55.946 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h1 2024-09-16T04:11:56.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.023+0000 7f2964798640 10 monclient: get_monmap_and_config 2024-09-16T04:11:56.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.023+0000 7f2964798640 10 monclient: build_initial_monmap 2024-09-16T04:11:56.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.023+0000 7f2964798640 10 monclient: monmap: 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:56.024110+0000 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:56.024110+0000 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.023+0000 7f2964798640 1 Processor -- start 2024-09-16T04:11:56.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.023+0000 7f2964798640 1 -- start start 2024-09-16T04:11:56.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.023+0000 7f2964798640 10 monclient: init 2024-09-16T04:11:56.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:56.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:56.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient(hunting): picked mon.noname-a con 0x7f295c09ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:56.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient(hunting): picked mon.noname-c con 0x7f295c0a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient(hunting): picked mon.noname-b con 0x7f295c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295c09fbf0 con 0x7f295c09ba30 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295c0a0e10 con 0x7f295c0a0750 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295c09f640 con 0x7f295c09e570 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2964798640 10 monclient(hunting): authenticate will time out at 1091.529595s 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2962d35640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f295c09e570 legacy=0x7f295c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961d33640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f295c0a0750 0x7f295c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58170/0 (socket says 172.21.15.171:58170) 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961d33640 1 -- 172.21.15.171:0/2190479571 learned_addr learned my addr 172.21.15.171:0/2190479571 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:56.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653495646 0 0) 0x7f295c0a0e10 con 0x7f295c0a0750 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961532640 10 monclient(hunting): _negotiate 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961532640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961532640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961532640 10 monclient(hunting): my global_id is 6145 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.024+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2948003520 con 0x7f295c0a0750 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2026070780 0 0) 0x7f295c09fbf0 con 0x7f295c09ba30 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient(hunting): _negotiate 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient(hunting): my global_id is 64578 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f295c0a0e10 con 0x7f295c09ba30 2024-09-16T04:11:56.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1723967107 0 0) 0x7f2948003520 con 0x7f295c0a0750 2024-09-16T04:11:56.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 >> v1:172.21.15.148:6790/0 conn(0x7f295c09e570 legacy=0x7f295c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:56.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 >> v1:172.21.15.148:6789/0 conn(0x7f295c09ba30 legacy=0x7f295c0a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:56.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: _finish_hunting 0 2024-09-16T04:11:56.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 monclient: found mon.noname-c 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295c09f810 con 0x7f295c0a0750 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: _finish_auth 0 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: _check_auth_tickets 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:56.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f294c001610 con 0x7f295c0a0750 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 5 monclient: authenticate success, global_id 6145 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: dump: 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:56.041 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f294c001930 con 0x7f295c0a0750 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 1 -- 172.21.15.171:0/2190479571 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f294c001d00 con 0x7f295c0a0750 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 4 set_mon_vals no callback set 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2961532640 10 monclient: dump: 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:56.042 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 10 monclient: shutdown 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 1 -- 172.21.15.171:0/2190479571 >> v1:172.21.15.171:6789/0 conn(0x7f295c0a0750 legacy=0x7f295c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 1 -- 172.21.15.171:0/2190479571 shutdown_connections 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 1 -- 172.21.15.171:0/2190479571 >> 172.21.15.171:0/2190479571 conn(0x7f295c0a52a0 msgr2=0x7f295c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 1 -- 172.21.15.171:0/2190479571 shutdown_connections 2024-09-16T04:11:56.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.025+0000 7f2964798640 1 -- 172.21.15.171:0/2190479571 wait complete. 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.026+0000 7f2964798640 10 monclient: build_initial_monmap 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.026+0000 7f2964798640 10 monclient: monmap: 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.026+0000 7f2964798640 1 Processor -- start 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.026+0000 7f2964798640 1 -- start start 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.026+0000 7f2964798640 10 monclient: init 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient(hunting): picked mon.noname-b con 0x7f295c09ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient(hunting): picked mon.noname-c con 0x7f295c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient(hunting): picked mon.noname-a con 0x7f295c0a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295c131140 con 0x7f295c0a0750 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295c12cf60 con 0x7f295c09ba30 2024-09-16T04:11:56.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f295c06ceb0 con 0x7f295c09e570 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient(hunting): authenticate will time out at 1091.532193s 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2961d33640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f295c09e570 legacy=0x7f295c1cef20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2962534640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f295c09ba30 0x7f295c127650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58180/0 (socket says 172.21.15.171:58180) 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2962534640 1 -- 172.21.15.171:0/1541583834 learned_addr learned my addr 172.21.15.171:0/1541583834 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1144183231 0 0) 0x7f295c12cf60 con 0x7f295c09ba30 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): my global_id is 6148 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2940003520 con 0x7f295c09ba30 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988697072 0 0) 0x7f295c131140 con 0x7f295c0a0750 2024-09-16T04:11:56.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): my global_id is 64581 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f295c12cf60 con 0x7f295c0a0750 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2832537425 0 0) 0x7f2940003520 con 0x7f295c09ba30 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f295c131140 con 0x7f295c09ba30 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2958001850 con 0x7f295c09ba30 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:56.047 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 872540027 0 0) 0x7f295c12cf60 con 0x7f295c0a0750 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2940003520 con 0x7f295c0a0750 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f29540043b0 con 0x7f295c0a0750 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:56.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient(hunting): dump: 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1363554694 0 0) 0x7f295c131140 con 0x7f295c09ba30 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 >> v1:172.21.15.148:6790/0 conn(0x7f295c09e570 legacy=0x7f295c1cef20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:56.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 >> v1:172.21.15.148:6789/0 conn(0x7f295c0a0750 legacy=0x7f295c1d2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 monclient: found mon.b 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295c1328f0 con 0x7f295c09ba30 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient: _finish_auth 0 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2952ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 5 monclient: authenticate success, global_id 6148 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient: _renew_subs 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f295c12efc0 con 0x7f295c09ba30 2024-09-16T04:11:56.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient: _renew_subs 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.027+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f295c12e6f0 con 0x7f295c09ba30 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2958001c60 con 0x7f295c09ba30 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2958003a60 con 0x7f295c09ba30 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2960d31640 4 set_mon_vals no callback set 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 10 monclient: dump: 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:56.051 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2958003c80 con 0x7f295c09ba30 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2964798640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2964798640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2964798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.028+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2944005080 con 0x7f295c09ba30 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.029+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(215..215 src has 1..215) ==== 14144+0+0 (unknown 3325574279 0 0) 0x7f29580496e0 con 0x7f295c09ba30 2024-09-16T04:11:56.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.029+0000 7f2961d33640 10 monclient: get_auth_request con 0x7f2940038ea0 auth_method 0 2024-09-16T04:11:56.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.033+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2958015c00 con 0x7f295c09ba30 2024-09-16T04:11:56.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.033+0000 7f2952ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:56.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.033+0000 7f2952ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:56.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.205+0000 7f2964798640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T04:11:56.207 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.205+0000 7f2964798640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T04:11:56.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.205+0000 7f2964798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.208 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.205+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) -- 0x7f2944005370 con 0x7f295c09ba30 2024-09-16T04:11:56.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.833+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v216) ==== 118+0+0 (unknown 2049919437 0 0) 0x7f29580198b0 con 0x7f295c09ba30 2024-09-16T04:11:56.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.833+0000 7f2952ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T04:11:56.835 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.833+0000 7f2952ffd640 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-16T04:11:56.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.938+0000 7f2964798640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T04:11:56.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.938+0000 7f2964798640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T04:11:56.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.938+0000 7f2964798640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:56.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.938+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) -- 0x7f2944002210 con 0x7f295c09ba30 2024-09-16T04:11:56.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.939+0000 7f2952ffd640 1 -- 172.21.15.171:0/1541583834 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v216) ==== 118+0+0 (unknown 1184635400 0 0) 0x7f295801e150 con 0x7f295c09ba30 2024-09-16T04:11:56.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.939+0000 7f2952ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-16T04:11:56.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.939+0000 7f2952ffd640 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-16T04:11:56.941 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h1' does not appear in the crush map 2024-09-16T04:11:56.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 >> v1:172.21.15.171:6812/7327273 conn(0x7f2940038ea0 legacy=0x7f294003b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:56.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 10 monclient: shutdown 2024-09-16T04:11:56.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:56.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 >> v1:172.21.15.171:6789/0 conn(0x7f295c09ba30 legacy=0x7f295c127650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:56.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 shutdown_connections 2024-09-16T04:11:56.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 >> 172.21.15.171:0/1541583834 conn(0x7f295c0a52a0 msgr2=0x7f295c131c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:56.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 shutdown_connections 2024-09-16T04:11:56.944 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:56.941+0000 7f2964798640 1 -- 172.21.15.171:0/1541583834 wait complete. 2024-09-16T04:11:56.952 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:56.953 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h2 2024-09-16T04:11:57.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.029+0000 7fc93d4d0640 10 monclient: get_monmap_and_config 2024-09-16T04:11:57.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.029+0000 7fc93d4d0640 10 monclient: build_initial_monmap 2024-09-16T04:11:57.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.029+0000 7fc93d4d0640 10 monclient: monmap: 2024-09-16T04:11:57.031 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:57.031 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:57.029887+0000 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:57.029887+0000 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.029+0000 7fc93d4d0640 1 Processor -- start 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.029+0000 7fc93d4d0640 1 -- start start 2024-09-16T04:11:57.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.029+0000 7fc93d4d0640 10 monclient: init 2024-09-16T04:11:57.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:57.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:57.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc938136dd0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:57.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient(hunting): picked mon.noname-c con 0x7fc9381371b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:57.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc9381375b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:57.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc93809af40 con 0x7fc9381375b0 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc93809b220 con 0x7fc9381371b0 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc93809aa20 con 0x7fc938136dd0 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient(hunting): authenticate will time out at 1092.535196s 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc936ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc938136dd0 legacy=0x7fc93809bc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc9367fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc9381371b0 0x7fc93813bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58190/0 (socket says 172.21.15.171:58190) 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc9367fc640 1 -- 172.21.15.171:0/3382719235 learned_addr learned my addr 172.21.15.171:0/3382719235 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168894317 0 0) 0x7fc93809b220 con 0x7fc9381371b0 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:57.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:57.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): my global_id is 6151 2024-09-16T04:11:57.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc920003520 con 0x7fc9381371b0 2024-09-16T04:11:57.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1775361794 0 0) 0x7fc93809af40 con 0x7fc9381375b0 2024-09-16T04:11:57.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:57.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:57.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:57.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient(hunting): my global_id is 64584 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc93809b220 con 0x7fc9381375b0 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 782054058 0 0) 0x7fc920003520 con 0x7fc9381371b0 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 >> v1:172.21.15.148:6790/0 conn(0x7fc938136dd0 legacy=0x7fc93809bc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 >> v1:172.21.15.148:6789/0 conn(0x7fc9381375b0 legacy=0x7fc93813f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 monclient: found mon.noname-c 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:57.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9380a4d30 con 0x7fc9381371b0 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc92c001610 con 0x7fc9381371b0 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 5 monclient: authenticate success, global_id 6151 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: dump: 2024-09-16T04:11:57.046 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:57.047 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:57.047 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc92c001930 con 0x7fc9381371b0 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:57.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 1 -- 172.21.15.171:0/3382719235 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc92c001d00 con 0x7fc9381371b0 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc93d4d0640 4 set_mon_vals no callback set 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.030+0000 7fc935ffb640 10 monclient: dump: 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.049 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 10 monclient: shutdown 2024-09-16T04:11:57.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 1 -- 172.21.15.171:0/3382719235 >> v1:172.21.15.171:6789/0 conn(0x7fc9381371b0 legacy=0x7fc93813bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 1 -- 172.21.15.171:0/3382719235 shutdown_connections 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 1 -- 172.21.15.171:0/3382719235 >> 172.21.15.171:0/3382719235 conn(0x7fc9380a9860 msgr2=0x7fc9380a5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 1 -- 172.21.15.171:0/3382719235 shutdown_connections 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 1 -- 172.21.15.171:0/3382719235 wait complete. 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 10 monclient: build_initial_monmap 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.031+0000 7fc93d4d0640 10 monclient: monmap: 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:57.051 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 1 Processor -- start 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 1 -- start start 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 10 monclient: init 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc938136dd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc9381371b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:57.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 10 monclient(hunting): picked mon.noname-c con 0x7fc9381375b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc93813ba60 con 0x7fc938136dd0 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.032+0000 7fc93d4d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc938138bd0 con 0x7fc9381371b0 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc93813bd40 con 0x7fc9381375b0 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 10 monclient(hunting): authenticate will time out at 1092.538029s 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc9367fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc9381371b0 0x7fc9381384c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:58194/0 (socket says 172.21.15.171:58194) 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc9367fc640 1 -- 172.21.15.171:0/409515161 learned_addr learned my addr 172.21.15.171:0/409515161 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc9377fe640 1 -- 172.21.15.171:0/409515161 >> v1:172.21.15.148:6790/0 conn(0x7fc9381375b0 legacy=0x7fc9381d3a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 183392835 0 0) 0x7fc938138bd0 con 0x7fc9381371b0 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): my global_id is 6154 2024-09-16T04:11:57.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc924003520 con 0x7fc9381371b0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718637820 0 0) 0x7fc93813ba60 con 0x7fc938136dd0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): my global_id is 64587 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc938138bd0 con 0x7fc938136dd0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1559924443 0 0) 0x7fc924003520 con 0x7fc9381371b0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc93813ba60 con 0x7fc9381371b0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc92c0015d0 con 0x7fc9381371b0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:57.054 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 272313247 0 0) 0x7fc938138bd0 con 0x7fc938136dd0 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc924003520 con 0x7fc938136dd0 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc930004140 con 0x7fc938136dd0 2024-09-16T04:11:57.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient(hunting): dump: 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1065508586 0 0) 0x7fc93813ba60 con 0x7fc9381371b0 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 >> v1:172.21.15.148:6790/0 conn(0x7fc9381375b0 legacy=0x7fc9381d3a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 >> v1:172.21.15.148:6789/0 conn(0x7fc938136dd0 legacy=0x7fc938137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 monclient: found mon.b 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc938135c30 con 0x7fc9381371b0 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:57.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 5 monclient: authenticate success, global_id 6154 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 10 monclient: _renew_subs 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9380a0340 con 0x7fc9381371b0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 10 monclient: _renew_subs 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc93813f470 con 0x7fc9381371b0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc92c001930 con 0x7fc9381371b0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc92c004e60 con 0x7fc9381371b0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc9357fa640 4 set_mon_vals no callback set 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.033+0000 7fc91f7fe640 10 monclient: dump: 2024-09-16T04:11:57.058 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:57.059 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:57.059 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.059 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:57.059 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:57.059 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.034+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc92c005080 con 0x7fc9381371b0 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.034+0000 7fc93d4d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.034+0000 7fc93d4d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.034+0000 7fc93d4d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.034+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc918005080 con 0x7fc9381371b0 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.035+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(216..216 src has 1..216) ==== 13934+0+0 (unknown 520896467 0 0) 0x7fc92c049750 con 0x7fc9381371b0 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.035+0000 7fc936ffd640 10 monclient: get_auth_request con 0x7fc924038ea0 auth_method 0 2024-09-16T04:11:57.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.039+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc92c015f20 con 0x7fc9381371b0 2024-09-16T04:11:57.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.039+0000 7fc91f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:57.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.039+0000 7fc91f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:57.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.208+0000 7fc93d4d0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T04:11:57.210 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.208+0000 7fc93d4d0640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T04:11:57.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.208+0000 7fc93d4d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.211 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.208+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) -- 0x7fc918005370 con 0x7fc9381371b0 2024-09-16T04:11:57.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.835+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v217) ==== 118+0+0 (unknown 2250716610 0 0) 0x7fc92c015820 con 0x7fc9381371b0 2024-09-16T04:11:57.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.835+0000 7fc91f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T04:11:57.837 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.835+0000 7fc91f7fe640 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-16T04:11:57.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.940+0000 7fc93d4d0640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T04:11:57.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.940+0000 7fc93d4d0640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T04:11:57.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.940+0000 7fc93d4d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:57.942 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.940+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) -- 0x7fc918001090 con 0x7fc9381371b0 2024-09-16T04:11:57.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.941+0000 7fc91f7fe640 1 -- 172.21.15.171:0/409515161 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v217) ==== 118+0+0 (unknown 2197983673 0 0) 0x7fc92c015580 con 0x7fc9381371b0 2024-09-16T04:11:57.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.941+0000 7fc91f7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-16T04:11:57.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.941+0000 7fc91f7fe640 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-16T04:11:57.943 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h2' does not appear in the crush map 2024-09-16T04:11:57.946 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 >> v1:172.21.15.171:6812/7327273 conn(0x7fc924038ea0 legacy=0x7fc92403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:57.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 10 monclient: shutdown 2024-09-16T04:11:57.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:57.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 >> v1:172.21.15.171:6789/0 conn(0x7fc9381371b0 legacy=0x7fc9381384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:57.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 shutdown_connections 2024-09-16T04:11:57.947 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 >> 172.21.15.171:0/409515161 conn(0x7fc9380a9860 msgr2=0x7fc9380a8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:57.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 shutdown_connections 2024-09-16T04:11:57.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:57.943+0000 7fc93d4d0640 1 -- 172.21.15.171:0/409515161 wait complete. 2024-09-16T04:11:57.955 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:57.955 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h3 2024-09-16T04:11:58.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.030+0000 7f42074ba640 10 monclient: get_monmap_and_config 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.030+0000 7f42074ba640 10 monclient: build_initial_monmap 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.030+0000 7f42074ba640 10 monclient: monmap: 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:58.031714+0000 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:58.031714+0000 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:58.033 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.031+0000 7f42074ba640 1 Processor -- start 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.031+0000 7f42074ba640 1 -- start start 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.031+0000 7f42074ba640 10 monclient: init 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient(hunting): picked mon.noname-b con 0x7f420009eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:58.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4200134b20 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4200134f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420009d060 con 0x7f4200134f30 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420009d340 con 0x7f4200134b20 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4200068840 con 0x7f420009eaa0 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient(hunting): authenticate will time out at 1093.537196s 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f4205256640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f420009eaa0 legacy=0x7f420009ee80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f4204a55640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4200134b20 0x7f4200139980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36562/0 (socket says 172.21.15.171:36562) 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f4204a55640 1 -- 172.21.15.171:0/3402897473 learned_addr learned my addr 172.21.15.171:0/3402897473 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3565260487 0 0) 0x7f420009d340 con 0x7f4200134b20 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:58.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient(hunting): my global_id is 6157 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41ec003520 con 0x7f4200134b20 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2125293440 0 0) 0x7f41ec003520 con 0x7f4200134b20 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 >> v1:172.21.15.148:6790/0 conn(0x7f420009eaa0 legacy=0x7f420009ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 >> v1:172.21.15.148:6789/0 conn(0x7f4200134f30 legacy=0x7f420013d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 monclient: found mon.noname-c 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4200141800 con 0x7f4200134b20 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:58.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41f0001610 con 0x7f4200134b20 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 5 monclient: authenticate success, global_id 6157 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: dump: 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:58.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41f0001900 con 0x7f4200134b20 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f42074ba640 4 set_mon_vals no callback set 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 1 -- 172.21.15.171:0/3402897473 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41f0001d10 con 0x7f4200134b20 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.032+0000 7f41f7fff640 10 monclient: dump: 2024-09-16T04:11:58.044 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 10 monclient: shutdown 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 1 -- 172.21.15.171:0/3402897473 >> v1:172.21.15.171:6789/0 conn(0x7f4200134b20 legacy=0x7f4200139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 1 -- 172.21.15.171:0/3402897473 shutdown_connections 2024-09-16T04:11:58.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 1 -- 172.21.15.171:0/3402897473 >> 172.21.15.171:0/3402897473 conn(0x7f42000a1e90 msgr2=0x7f42000a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 1 -- 172.21.15.171:0/3402897473 shutdown_connections 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.033+0000 7f42074ba640 1 -- 172.21.15.171:0/3402897473 wait complete. 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient: build_initial_monmap 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient: monmap: 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 1 Processor -- start 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 1 -- start start 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient: init 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient(hunting): picked mon.noname-a con 0x7f420009eaa0 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4200134b20 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4200134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f420009c580 con 0x7f420009eaa0 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42000738f0 con 0x7f4200134f30 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4200073bd0 con 0x7f4200134b20 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f42074ba640 10 monclient(hunting): authenticate will time out at 1093.539816s 2024-09-16T04:11:58.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f4204a55640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f4200134b20 legacy=0x7f4200070e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f4205a57640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4200134f30 0x7f42001d1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36570/0 (socket says 172.21.15.171:36570) 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.034+0000 7f4205a57640 1 -- 172.21.15.171:0/1923185727 learned_addr learned my addr 172.21.15.171:0/1923185727 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999961187 0 0) 0x7f42000738f0 con 0x7f4200134f30 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): my global_id is 6160 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41e4003520 con 0x7f4200134f30 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955540977 0 0) 0x7f420009c580 con 0x7f420009eaa0 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:58.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): my global_id is 64593 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42000738f0 con 0x7f420009eaa0 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3030767969 0 0) 0x7f41e4003520 con 0x7f4200134f30 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f420009c580 con 0x7f4200134f30 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41f8001ce0 con 0x7f4200134f30 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): dump: 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.049 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2050457013 0 0) 0x7f42000738f0 con 0x7f420009eaa0 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41e4003520 con 0x7f420009eaa0 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41fc004140 con 0x7f420009eaa0 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient(hunting): dump: 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.050 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1737397933 0 0) 0x7f420009c580 con 0x7f4200134f30 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 >> v1:172.21.15.148:6790/0 conn(0x7f4200134b20 legacy=0x7f4200070e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 >> v1:172.21.15.148:6789/0 conn(0x7f420009eaa0 legacy=0x7f4200070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 monclient: found mon.b 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4200072c40 con 0x7f4200134f30 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 5 monclient: authenticate success, global_id 6160 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 10 monclient: _renew_subs 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4200072310 con 0x7f4200134f30 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f41f8003c40 con 0x7f4200134f30 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41f8004010 con 0x7f4200134f30 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f5ffb640 10 monclient: dump: 2024-09-16T04:11:58.052 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f41f77fe640 4 set_mon_vals no callback set 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 10 monclient: _renew_subs 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.035+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4200071570 con 0x7f4200134f30 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f41f80041b0 con 0x7f4200134f30 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(217..217 src has 1..217) ==== 13724+0+0 (unknown 3289987956 0 0) 0x7f41f80498d0 con 0x7f4200134f30 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f42074ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f42074ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f42074ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42000a3290 con 0x7f4200134f30 2024-09-16T04:11:58.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.036+0000 7f4205256640 10 monclient: get_auth_request con 0x7f41e4038ef0 auth_method 0 2024-09-16T04:11:58.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.041+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f41f8016030 con 0x7f4200134f30 2024-09-16T04:11:58.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.041+0000 7f41f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:58.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.041+0000 7f41f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:58.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.212+0000 7f42074ba640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T04:11:58.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.212+0000 7f42074ba640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T04:11:58.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.212+0000 7f42074ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.212+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) -- 0x7f420009f860 con 0x7f4200134f30 2024-09-16T04:11:58.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.844+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v218) ==== 118+0+0 (unknown 4133981529 0 0) 0x7f41f8015930 con 0x7f4200134f30 2024-09-16T04:11:58.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.844+0000 7f41f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T04:11:58.846 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.844+0000 7f41f5ffb640 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-16T04:11:58.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.950+0000 7f42074ba640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T04:11:58.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.950+0000 7f42074ba640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T04:11:58.952 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.950+0000 7f42074ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:58.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.950+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) -- 0x7f4200071860 con 0x7f4200134f30 2024-09-16T04:11:58.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.952+0000 7f41f5ffb640 1 -- 172.21.15.171:0/1923185727 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v218) ==== 118+0+0 (unknown 3316571889 0 0) 0x7f41f801e4e0 con 0x7f4200134f30 2024-09-16T04:11:58.953 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.952+0000 7f41f5ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-16T04:11:58.954 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.952+0000 7f41f5ffb640 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-16T04:11:58.954 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h3' does not appear in the crush map 2024-09-16T04:11:58.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 >> v1:172.21.15.171:6812/7327273 conn(0x7f41e4038ef0 legacy=0x7f41e403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:58.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 10 monclient: shutdown 2024-09-16T04:11:58.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:58.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 >> v1:172.21.15.171:6789/0 conn(0x7f4200134f30 legacy=0x7f42001d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:58.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 shutdown_connections 2024-09-16T04:11:58.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 >> 172.21.15.171:0/1923185727 conn(0x7f42000a1e90 msgr2=0x7f42000a0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:58.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 shutdown_connections 2024-09-16T04:11:58.957 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:58.954+0000 7f42074ba640 1 -- 172.21.15.171:0/1923185727 wait complete. 2024-09-16T04:11:58.965 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:58.965 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h4 2024-09-16T04:11:59.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.041+0000 7f6e32726640 10 monclient: get_monmap_and_config 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.041+0000 7f6e32726640 10 monclient: build_initial_monmap 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.041+0000 7f6e32726640 10 monclient: monmap: 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:11:59.041964+0000 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:11:59.041964+0000 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:11:59.043 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.041+0000 7f6e32726640 1 Processor -- start 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.041+0000 7f6e32726640 1 -- start start 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.041+0000 7f6e32726640 10 monclient: init 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c0a1340 con 0x7f6e2c136b60 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c137650 con 0x7f6e2c13d620 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c0a0aa0 con 0x7f6e2c139b10 2024-09-16T04:11:59.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e32726640 10 monclient(hunting): authenticate will time out at 1094.547465s 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2b7fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6e2c139b10 legacy=0x7f6e2c13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e30cc3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6e2c13d620 0x7f6e2c13fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36580/0 (socket says 172.21.15.171:36580) 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e30cc3640 1 -- 172.21.15.171:0/1398065811 learned_addr learned my addr 172.21.15.171:0/1398065811 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3628535474 0 0) 0x7f6e2c137650 con 0x7f6e2c13d620 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): my global_id is 6163 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e14003520 con 0x7f6e2c13d620 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97059075 0 0) 0x7f6e2c0a1340 con 0x7f6e2c136b60 2024-09-16T04:11:59.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.042+0000 7f6e2affd640 10 monclient(hunting): my global_id is 64596 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e2c137650 con 0x7f6e2c136b60 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3684819800 0 0) 0x7f6e14003520 con 0x7f6e2c13d620 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 >> v1:172.21.15.148:6790/0 conn(0x7f6e2c139b10 legacy=0x7f6e2c13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 >> v1:172.21.15.148:6789/0 conn(0x7f6e2c136b60 legacy=0x7f6e2c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 monclient: found mon.noname-c 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c0a0550 con 0x7f6e2c13d620 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:59.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6e24001990 con 0x7f6e2c13d620 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 5 monclient: authenticate success, global_id 6163 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: dump: 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:59.054 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e24001bf0 con 0x7f6e2c13d620 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 1 -- 172.21.15.171:0/1398065811 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6e24001fc0 con 0x7f6e2c13d620 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 4 set_mon_vals no callback set 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e2affd640 10 monclient: dump: 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:59.055 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 10 monclient: shutdown 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 1 -- 172.21.15.171:0/1398065811 >> v1:172.21.15.171:6789/0 conn(0x7f6e2c13d620 legacy=0x7f6e2c13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 1 -- 172.21.15.171:0/1398065811 shutdown_connections 2024-09-16T04:11:59.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 1 -- 172.21.15.171:0/1398065811 >> 172.21.15.171:0/1398065811 conn(0x7f6e2c0a98b0 msgr2=0x7f6e2c0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 1 -- 172.21.15.171:0/1398065811 shutdown_connections 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.043+0000 7f6e32726640 1 -- 172.21.15.171:0/1398065811 wait complete. 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.044+0000 7f6e32726640 10 monclient: build_initial_monmap 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.044+0000 7f6e32726640 10 monclient: monmap: 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:11:59.057 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.044+0000 7f6e32726640 1 Processor -- start 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.044+0000 7f6e32726640 1 -- start start 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.044+0000 7f6e32726640 10 monclient: init 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c120a80 con 0x7f6e2c136b60 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c120d60 con 0x7f6e2c13d620 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c0a2c20 con 0x7f6e2c139b10 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:59.058 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient(hunting): authenticate will time out at 1094.550224s 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e2b7fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6e2c139b10 legacy=0x7f6e2c09db90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e30cc3640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6e2c13d620 0x7f6e2c1d37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36592/0 (socket says 172.21.15.171:36592) 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e30cc3640 1 -- 172.21.15.171:0/79206709 learned_addr learned my addr 172.21.15.171:0/79206709 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319695552 0 0) 0x7f6e2c120d60 con 0x7f6e2c13d620 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): my global_id is 6166 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e18003550 con 0x7f6e2c13d620 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457433714 0 0) 0x7f6e2c120a80 con 0x7f6e2c136b60 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:59.059 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): my global_id is 64599 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e2c120d60 con 0x7f6e2c136b60 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2904176000 0 0) 0x7f6e18003550 con 0x7f6e2c13d620 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e2c120a80 con 0x7f6e2c13d620 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6e24001890 con 0x7f6e2c13d620 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): dump: 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.060 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2741817482 0 0) 0x7f6e2c120d60 con 0x7f6e2c136b60 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e18003550 con 0x7f6e2c136b60 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 565065125 0 0) 0x7f6e2c120a80 con 0x7f6e2c13d620 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 >> v1:172.21.15.148:6790/0 conn(0x7f6e2c139b10 legacy=0x7f6e2c09db90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 >> v1:172.21.15.148:6789/0 conn(0x7f6e2c136b60 legacy=0x7f6e2c09d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 monclient: found mon.b 2024-09-16T04:11:59.061 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c0a0240 con 0x7f6e2c13d620 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient: _finish_auth 0 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 5 monclient: authenticate success, global_id 6166 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient: _renew_subs 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2c1207a0 con 0x7f6e2c13d620 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient: _renew_subs 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.045+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e2c121180 con 0x7f6e2c13d620 2024-09-16T04:11:59.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e24004ff0 con 0x7f6e2c13d620 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6e240053c0 con 0x7f6e2c13d620 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e2a7fc640 4 set_mon_vals no callback set 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 10 monclient: dump: 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:11:59.063 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6e24005560 con 0x7f6e2c13d620 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e32726640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e32726640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e32726640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.046+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e0c005080 con 0x7f6e2c13d620 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.047+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(218..218 src has 1..218) ==== 13514+0+0 (unknown 1274657519 0 0) 0x7f6e240498b0 con 0x7f6e2c13d620 2024-09-16T04:11:59.064 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.047+0000 7f6e2bfff640 10 monclient: get_auth_request con 0x7f6e18038ee0 auth_method 0 2024-09-16T04:11:59.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.051+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6e24016160 con 0x7f6e2c13d620 2024-09-16T04:11:59.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.051+0000 7f6e28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:59.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.051+0000 7f6e28ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:59.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.223+0000 7f6e32726640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T04:11:59.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.223+0000 7f6e32726640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T04:11:59.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.223+0000 7f6e32726640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.223+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) -- 0x7f6e0c005370 con 0x7f6e2c13d620 2024-09-16T04:11:59.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.858+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v219) ==== 118+0+0 (unknown 2293417083 0 0) 0x7f6e24015a60 con 0x7f6e2c13d620 2024-09-16T04:11:59.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.858+0000 7f6e28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T04:11:59.860 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.858+0000 7f6e28ff9640 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-16T04:11:59.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.966+0000 7f6e32726640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T04:11:59.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.966+0000 7f6e32726640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T04:11:59.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.966+0000 7f6e32726640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:11:59.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.966+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) -- 0x7f6e0c002210 con 0x7f6e2c13d620 2024-09-16T04:11:59.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.967+0000 7f6e28ff9640 1 -- 172.21.15.171:0/79206709 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v219) ==== 118+0+0 (unknown 231993386 0 0) 0x7f6e2401e5b0 con 0x7f6e2c13d620 2024-09-16T04:11:59.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.967+0000 7f6e28ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-16T04:11:59.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.967+0000 7f6e28ff9640 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-16T04:11:59.969 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h4' does not appear in the crush map 2024-09-16T04:11:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 >> v1:172.21.15.171:6812/7327273 conn(0x7f6e18038ee0 legacy=0x7f6e1803b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 10 monclient: shutdown 2024-09-16T04:11:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 >> v1:172.21.15.171:6789/0 conn(0x7f6e2c13d620 legacy=0x7f6e2c1d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:59.973 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 shutdown_connections 2024-09-16T04:11:59.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 >> 172.21.15.171:0/79206709 conn(0x7f6e2c0a98b0 msgr2=0x7f6e2c139240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:59.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 shutdown_connections 2024-09-16T04:11:59.974 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:11:59.969+0000 7f6e32726640 1 -- 172.21.15.171:0/79206709 wait complete. 2024-09-16T04:11:59.981 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:11:59.981 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h5 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.058+0000 7feba0665640 10 monclient: get_monmap_and_config 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.058+0000 7feba0665640 10 monclient: build_initial_monmap 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.058+0000 7feba0665640 10 monclient: monmap: 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:00.059195+0000 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:00.059195+0000 2024-09-16T04:12:00.060 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:00.061 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.061 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:00.061 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:00.061 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:00.061 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 1 Processor -- start 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 1 -- start start 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 10 monclient: init 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 10 monclient(hunting): picked mon.noname-a con 0x7feb98132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.059+0000 7feba0665640 10 monclient(hunting): picked mon.noname-b con 0x7feb9809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 10 monclient(hunting): picked mon.noname-c con 0x7feb98132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb981332b0 con 0x7feb98132a80 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98134310 con 0x7feb98132e60 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb9809d980 con 0x7feb9809ec20 2024-09-16T04:12:00.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 10 monclient(hunting): authenticate will time out at 1095.565072s 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9dc00640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7feb9809ec20 legacy=0x7feb98133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9ec02640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7feb98132e60 0x7feb9813c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36608/0 (socket says 172.21.15.171:36608) 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9e401640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7feb98132a80 0x7feb981334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.171:44210/0 (socket says 172.21.15.171:44210) 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9ec02640 1 -- 172.21.15.171:0/2557578602 learned_addr learned my addr 172.21.15.171:0/2557578602 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835444710 0 0) 0x7feb98134310 con 0x7feb98132e60 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): my global_id is 6169 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb84003520 con 0x7feb98132e60 2024-09-16T04:12:00.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649720168 0 0) 0x7feb981332b0 con 0x7feb98132a80 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient(hunting): my global_id is 64602 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb98134310 con 0x7feb98132a80 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4236004505 0 0) 0x7feb84003520 con 0x7feb98132e60 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 >> v1:172.21.15.148:6790/0 conn(0x7feb9809ec20 legacy=0x7feb98133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 >> v1:172.21.15.148:6789/0 conn(0x7feb98132a80 legacy=0x7feb981334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 monclient: found mon.noname-c 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9809dc60 con 0x7feb98132e60 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: _finish_auth 0 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 5 monclient: authenticate success, global_id 6169 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feb90001990 con 0x7feb98132e60 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feb9d3ff640 10 monclient: dump: 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.073 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.060+0000 7feba0665640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb90001c40 con 0x7feb98132e60 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feb9d3ff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feb9d3ff640 1 -- 172.21.15.171:0/2557578602 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feb90002050 con 0x7feb98132e60 2024-09-16T04:12:00.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feb9d3ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 4 set_mon_vals no callback set 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feb9d3ff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feb9d3ff640 10 monclient: dump: 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 10 monclient: shutdown 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 1 -- 172.21.15.171:0/2557578602 >> v1:172.21.15.171:6789/0 conn(0x7feb98132e60 legacy=0x7feb9813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 1 -- 172.21.15.171:0/2557578602 shutdown_connections 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 1 -- 172.21.15.171:0/2557578602 >> 172.21.15.171:0/2557578602 conn(0x7feb980a52a0 msgr2=0x7feb980a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 1 -- 172.21.15.171:0/2557578602 shutdown_connections 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.061+0000 7feba0665640 1 -- 172.21.15.171:0/2557578602 wait complete. 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.062+0000 7feba0665640 10 monclient: build_initial_monmap 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.062+0000 7feba0665640 10 monclient: monmap: 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:00.076 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.062+0000 7feba0665640 1 Processor -- start 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.062+0000 7feba0665640 1 -- start start 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.062+0000 7feba0665640 10 monclient: init 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient(hunting): picked mon.noname-c con 0x7feb9809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient(hunting): picked mon.noname-b con 0x7feb98132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:00.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient(hunting): picked mon.noname-a con 0x7feb98132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98122f40 con 0x7feb98132e60 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb98122b20 con 0x7feb98132a80 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb980a30f0 con 0x7feb9809ec20 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb9e401640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7feb9809ec20 legacy=0x7feb981c9660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient(hunting): authenticate will time out at 1095.568262s 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb9dc00640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7feb98132a80 0x7feb981ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36616/0 (socket says 172.21.15.171:36616) 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb9dc00640 1 -- 172.21.15.171:0/1985016815 learned_addr learned my addr 172.21.15.171:0/1985016815 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622160683 0 0) 0x7feb98122b20 con 0x7feb98132a80 2024-09-16T04:12:00.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): my global_id is 6172 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb7c003520 con 0x7feb98132a80 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750155654 0 0) 0x7feb9809dc60 con 0x7feb98132e60 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): my global_id is 64605 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb98122b20 con 0x7feb98132e60 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 704100575 0 0) 0x7feb7c003520 con 0x7feb98132a80 2024-09-16T04:12:00.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb9809dc60 con 0x7feb98132a80 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feb8c0041d0 con 0x7feb98132a80 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): dump: 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:00.080 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4142388211 0 0) 0x7feb98122b20 con 0x7feb98132e60 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb7c003520 con 0x7feb98132e60 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feb90001df0 con 0x7feb98132e60 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient(hunting): dump: 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.081 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3109172514 0 0) 0x7feb9809dc60 con 0x7feb98132a80 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 >> v1:172.21.15.148:6790/0 conn(0x7feb9809ec20 legacy=0x7feb981c9660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 >> v1:172.21.15.148:6789/0 conn(0x7feb98132e60 legacy=0x7feb981d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 monclient: found mon.b 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9806ceb0 con 0x7feb98132a80 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient: _finish_auth 0 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feb8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 5 monclient: authenticate success, global_id 6172 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient: _renew_subs 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.063+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb98120830 con 0x7feb98132a80 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 10 monclient: _renew_subs 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feb981215b0 con 0x7feb98132a80 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb8c004430 con 0x7feb98132a80 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:00.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7feb8c004800 con 0x7feb98132a80 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb9cbfe640 4 set_mon_vals no callback set 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 10 monclient: dump: 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:00.084 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7feb8c0049a0 con 0x7feb98132a80 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.064+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb80005080 con 0x7feb98132a80 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.065+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(219..219 src has 1..219) ==== 13304+0+0 (unknown 1116276388 0 0) 0x7feb8c049330 con 0x7feb98132a80 2024-09-16T04:12:00.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.065+0000 7feb9e401640 10 monclient: get_auth_request con 0x7feb7c038f90 auth_method 0 2024-09-16T04:12:00.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.069+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7feb8c015c60 con 0x7feb98132a80 2024-09-16T04:12:00.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.069+0000 7feb8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:00.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.069+0000 7feb8affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:00.241 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.239+0000 7feba0665640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T04:12:00.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.239+0000 7feba0665640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T04:12:00.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.239+0000 7feba0665640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.239+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) -- 0x7feb80005370 con 0x7feb98132a80 2024-09-16T04:12:00.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.859+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v220) ==== 118+0+0 (unknown 1674769786 0 0) 0x7feb8c015560 con 0x7feb98132a80 2024-09-16T04:12:00.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.859+0000 7feb8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T04:12:00.861 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.859+0000 7feb8affd640 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-16T04:12:00.968 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.965+0000 7feba0665640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.965+0000 7feba0665640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.965+0000 7feba0665640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.965+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) -- 0x7feb80001090 con 0x7feb98132a80 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.966+0000 7feb8affd640 1 -- 172.21.15.171:0/1985016815 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v220) ==== 118+0+0 (unknown 1159062795 0 0) 0x7feb8c01e110 con 0x7feb98132a80 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.966+0000 7feb8affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.966+0000 7feb8affd640 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-16T04:12:00.969 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h5' does not appear in the crush map 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 >> v1:172.21.15.171:6812/7327273 conn(0x7feb7c038f90 legacy=0x7feb7c03b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 10 monclient: shutdown 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 >> v1:172.21.15.171:6789/0 conn(0x7feb98132a80 legacy=0x7feb981ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 shutdown_connections 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 >> 172.21.15.171:0/1985016815 conn(0x7feb980a52a0 msgr2=0x7feb980727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 shutdown_connections 2024-09-16T04:12:00.970 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:00.968+0000 7feba0665640 1 -- 172.21.15.171:0/1985016815 wait complete. 2024-09-16T04:12:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:00.979 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h6 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.059+0000 7f7e5afe8640 10 monclient: get_monmap_and_config 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.059+0000 7f7e5afe8640 10 monclient: build_initial_monmap 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.060+0000 7f7e5afe8640 10 monclient: monmap: 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:01.060774+0000 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:01.060774+0000 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:01.062 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.063 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:01.063 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:01.063 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:01.063 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.060+0000 7f7e5afe8640 1 Processor -- start 2024-09-16T04:12:01.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.060+0000 7f7e5afe8640 1 -- start start 2024-09-16T04:12:01.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.060+0000 7f7e5afe8640 10 monclient: init 2024-09-16T04:12:01.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:01.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:01.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient(hunting): picked mon.noname-c con 0x7f7e54136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient(hunting): picked mon.noname-b con 0x7f7e54139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient(hunting): picked mon.noname-a con 0x7f7e5413d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e540a1340 con 0x7f7e5413d620 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e54137650 con 0x7f7e54136b60 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e540a0aa0 con 0x7f7e54139b10 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 10 monclient(hunting): authenticate will time out at 1096.566406s 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e58d84640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7e54136b60 0x7f7e54136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36618/0 (socket says 172.21.15.171:36618) 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e58d84640 1 -- 172.21.15.171:0/2430895086 learned_addr learned my addr 172.21.15.171:0/2430895086 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:01.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e53fff640 1 -- 172.21.15.171:0/2430895086 >> v1:172.21.15.148:6790/0 conn(0x7f7e54139b10 legacy=0x7f7e5413bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2495481671 0 0) 0x7f7e54137650 con 0x7f7e54136b60 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient(hunting): my global_id is 6175 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e44003520 con 0x7f7e54136b60 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 727531675 0 0) 0x7f7e44003520 con 0x7f7e54136b60 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 >> v1:172.21.15.148:6790/0 conn(0x7f7e54139b10 legacy=0x7f7e5413bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 >> v1:172.21.15.148:6789/0 conn(0x7f7e5413d620 legacy=0x7f7e5413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:01.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e540a0550 con 0x7f7e54136b60 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 5 monclient: authenticate success, global_id 6175 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e48001610 con 0x7f7e54136b60 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:01.074 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e537fe640 10 monclient: dump: 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.075 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.061+0000 7f7e5afe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e48001970 con 0x7f7e54136b60 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e537fe640 1 -- 172.21.15.171:0/2430895086 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e480048a0 con 0x7f7e54136b60 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 4 set_mon_vals no callback set 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e537fe640 10 monclient: dump: 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:01.076 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 10 monclient: shutdown 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:01.077 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2430895086 >> v1:172.21.15.171:6789/0 conn(0x7f7e54136b60 legacy=0x7f7e54136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2430895086 shutdown_connections 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2430895086 >> 172.21.15.171:0/2430895086 conn(0x7f7e540a98b0 msgr2=0x7f7e540a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2430895086 shutdown_connections 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.062+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2430895086 wait complete. 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.063+0000 7f7e5afe8640 10 monclient: build_initial_monmap 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.063+0000 7f7e5afe8640 10 monclient: monmap: 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:01.078 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.063+0000 7f7e5afe8640 1 Processor -- start 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.063+0000 7f7e5afe8640 1 -- start start 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.063+0000 7f7e5afe8640 10 monclient: init 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient(hunting): picked mon.noname-b con 0x7f7e54136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:01.079 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient(hunting): picked mon.noname-a con 0x7f7e54139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient(hunting): picked mon.noname-c con 0x7f7e5413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e54120870 con 0x7f7e54139b10 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e54120b50 con 0x7f7e54136b60 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e540a6fc0 con 0x7f7e5413d620 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient(hunting): authenticate will time out at 1096.569157s 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e58d84640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f7e54136b60 0x7f7e541cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36634/0 (socket says 172.21.15.171:36634) 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e58d84640 1 -- 172.21.15.171:0/2888952145 learned_addr learned my addr 172.21.15.171:0/2888952145 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e59585640 1 -- 172.21.15.171:0/2888952145 >> v1:172.21.15.148:6790/0 conn(0x7f7e5413d620 legacy=0x7f7e541d4110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 53065761 0 0) 0x7f7e54120b50 con 0x7f7e54136b60 2024-09-16T04:12:01.080 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): my global_id is 6178 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e38003550 con 0x7f7e54136b60 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 540388401 0 0) 0x7f7e54120870 con 0x7f7e54139b10 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): my global_id is 64611 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e54120b50 con 0x7f7e54139b10 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2567555155 0 0) 0x7f7e38003550 con 0x7f7e54136b60 2024-09-16T04:12:01.081 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e54120870 con 0x7f7e54136b60 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e48001430 con 0x7f7e54136b60 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): dump: 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.082 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4169426927 0 0) 0x7f7e54120b50 con 0x7f7e54139b10 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e38003550 con 0x7f7e54139b10 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e40004140 con 0x7f7e54139b10 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient(hunting): dump: 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:01.083 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 237000766 0 0) 0x7f7e54120870 con 0x7f7e54136b60 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 >> v1:172.21.15.148:6790/0 conn(0x7f7e5413d620 legacy=0x7f7e541d4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:01.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 >> v1:172.21.15.148:6789/0 conn(0x7f7e54139b10 legacy=0x7f7e541d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 monclient: found mon.b 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e540a2a60 con 0x7f7e54136b60 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 5 monclient: authenticate success, global_id 6178 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient: _renew_subs 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e54120590 con 0x7f7e54136b60 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient: _renew_subs 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e5409e2c0 con 0x7f7e54136b60 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.064+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e48001970 con 0x7f7e54136b60 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e517fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e48004e60 con 0x7f7e54136b60 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e52ffd640 4 set_mon_vals no callback set 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e517fa640 10 monclient: dump: 2024-09-16T04:12:01.086 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7e48010290 con 0x7f7e54136b60 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e5afe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:01.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e5afe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:01.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e5afe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.065+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e34005080 con 0x7f7e54136b60 2024-09-16T04:12:01.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.066+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(220..220 src has 1..220) ==== 13094+0+0 (unknown 20954196 0 0) 0x7f7e480493b0 con 0x7f7e54136b60 2024-09-16T04:12:01.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.066+0000 7f7e53fff640 10 monclient: get_auth_request con 0x7f7e38038ed0 auth_method 0 2024-09-16T04:12:01.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.070+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7e48015db0 con 0x7f7e54136b60 2024-09-16T04:12:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.070+0000 7f7e517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:01.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.070+0000 7f7e517fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:01.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.240+0000 7f7e5afe8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T04:12:01.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.240+0000 7f7e5afe8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T04:12:01.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.240+0000 7f7e5afe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.242 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.240+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) -- 0x7f7e34005370 con 0x7f7e54136b60 2024-09-16T04:12:01.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.871+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v221) ==== 118+0+0 (unknown 3865982686 0 0) 0x7f7e4801e260 con 0x7f7e54136b60 2024-09-16T04:12:01.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.872+0000 7f7e517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T04:12:01.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.872+0000 7f7e517fa640 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-16T04:12:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.979+0000 7f7e5afe8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T04:12:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.979+0000 7f7e5afe8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T04:12:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.979+0000 7f7e5afe8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.979+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) -- 0x7f7e34001090 con 0x7f7e54136b60 2024-09-16T04:12:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.981+0000 7f7e517fa640 1 -- 172.21.15.171:0/2888952145 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v221) ==== 118+0+0 (unknown 2156621498 0 0) 0x7f7e48044c90 con 0x7f7e54136b60 2024-09-16T04:12:01.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.981+0000 7f7e517fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-16T04:12:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.981+0000 7f7e517fa640 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-16T04:12:01.985 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h6' does not appear in the crush map 2024-09-16T04:12:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 >> v1:172.21.15.171:6812/7327273 conn(0x7f7e38038ed0 legacy=0x7f7e3803b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 10 monclient: shutdown 2024-09-16T04:12:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:01.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 >> v1:172.21.15.171:6789/0 conn(0x7f7e54136b60 legacy=0x7f7e541cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 shutdown_connections 2024-09-16T04:12:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 >> 172.21.15.171:0/2888952145 conn(0x7f7e540a98b0 msgr2=0x7f7e541391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 shutdown_connections 2024-09-16T04:12:01.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:01.983+0000 7f7e5afe8640 1 -- 172.21.15.171:0/2888952145 wait complete. 2024-09-16T04:12:01.994 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:01.994 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h7 2024-09-16T04:12:02.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.069+0000 7f5d51a74640 10 monclient: get_monmap_and_config 2024-09-16T04:12:02.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.069+0000 7f5d51a74640 10 monclient: build_initial_monmap 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.069+0000 7f5d51a74640 10 monclient: monmap: 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:02.070744+0000 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:02.070744+0000 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:02.072 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:02.073 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:02.073 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:02.073 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:02.073 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:02.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.070+0000 7f5d51a74640 1 Processor -- start 2024-09-16T04:12:02.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.070+0000 7f5d51a74640 1 -- start start 2024-09-16T04:12:02.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.070+0000 7f5d51a74640 10 monclient: init 2024-09-16T04:12:02.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:02.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:02.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c0a4040 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:02.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c09fc20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:02.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c0a0050 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:02.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c0737d0 con 0x7f5d4c0a0050 2024-09-16T04:12:02.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c06c630 con 0x7f5d4c0a4040 2024-09-16T04:12:02.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c06d8e0 con 0x7f5d4c09fc20 2024-09-16T04:12:02.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:02.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 10 monclient(hunting): authenticate will time out at 1097.576318s 2024-09-16T04:12:02.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4affd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f5d4c09fc20 legacy=0x7f5d4c13b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:02.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5d4c0a4040 0x7f5d4c0a0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36640/0 (socket says 172.21.15.171:36640) 2024-09-16T04:12:02.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4b7fe640 1 -- 172.21.15.171:0/141293382 learned_addr learned my addr 172.21.15.171:0/141293382 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:02.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565646074 0 0) 0x7f5d4c06c630 con 0x7f5d4c0a4040 2024-09-16T04:12:02.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:02.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:02.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:02.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): my global_id is 6181 2024-09-16T04:12:02.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d34003520 con 0x7f5d4c0a4040 2024-09-16T04:12:02.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993031419 0 0) 0x7f5d4c0737d0 con 0x7f5d4c0a0050 2024-09-16T04:12:02.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:02.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:02.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:02.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient(hunting): my global_id is 64614 2024-09-16T04:12:02.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c06c630 con 0x7f5d4c0a0050 2024-09-16T04:12:02.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2361750309 0 0) 0x7f5d34003520 con 0x7f5d4c0a4040 2024-09-16T04:12:02.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 >> v1:172.21.15.148:6790/0 conn(0x7f5d4c09fc20 legacy=0x7f5d4c13b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:02.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 >> v1:172.21.15.148:6789/0 conn(0x7f5d4c0a0050 legacy=0x7f5d4c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:02.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:02.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:12:02.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:02.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c143840 con 0x7f5d4c0a4040 2024-09-16T04:12:02.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:02.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:02.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:02.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 5 monclient: authenticate success, global_id 6181 2024-09-16T04:12:02.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.071+0000 7f5d51a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:02.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5d3c001990 con 0x7f5d4c0a4040 2024-09-16T04:12:02.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:02.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:02.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: dump: 2024-09-16T04:12:02.099 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:02.099 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:02.099 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.099 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.099 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:02.099 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d3c001bb0 con 0x7f5d4c0a4040 2024-09-16T04:12:02.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 1 -- 172.21.15.171:0/141293382 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5d3c001f80 con 0x7f5d4c0a4040 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 4 set_mon_vals no callback set 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d4a7fc640 10 monclient: dump: 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.101 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.102 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:02.102 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:02.102 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:02.102 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:02.102 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:02.102 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:02.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 10 monclient: shutdown 2024-09-16T04:12:02.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:02.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 1 -- 172.21.15.171:0/141293382 >> v1:172.21.15.171:6789/0 conn(0x7f5d4c0a4040 legacy=0x7f5d4c0a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:02.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 1 -- 172.21.15.171:0/141293382 shutdown_connections 2024-09-16T04:12:02.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 1 -- 172.21.15.171:0/141293382 >> 172.21.15.171:0/141293382 conn(0x7f5d4c0a7520 msgr2=0x7f5d4c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:02.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 1 -- 172.21.15.171:0/141293382 shutdown_connections 2024-09-16T04:12:02.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.072+0000 7f5d51a74640 1 -- 172.21.15.171:0/141293382 wait complete. 2024-09-16T04:12:02.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.073+0000 7f5d51a74640 10 monclient: build_initial_monmap 2024-09-16T04:12:02.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.073+0000 7f5d51a74640 10 monclient: monmap: 2024-09-16T04:12:02.104 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:02.104 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:02.104 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:02.105 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:02.105 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:02.105 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:02.105 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:02.105 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:02.105 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:02.106 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:02.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.073+0000 7f5d51a74640 1 Processor -- start 2024-09-16T04:12:02.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.073+0000 7f5d51a74640 1 -- start start 2024-09-16T04:12:02.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.073+0000 7f5d51a74640 10 monclient: init 2024-09-16T04:12:02.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:02.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:02.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c09fc20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:02.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c0a0050 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:02.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c0a4040 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:02.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c09e0d0 con 0x7f5d4c0a0050 2024-09-16T04:12:02.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c139e20 con 0x7f5d4c09fc20 2024-09-16T04:12:02.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c139860 con 0x7f5d4c0a4040 2024-09-16T04:12:02.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:02.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d51a74640 10 monclient(hunting): authenticate will time out at 1097.579346s 2024-09-16T04:12:02.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d4b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f5d4c09fc20 0x7f5d4c1cb930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36654/0 (socket says 172.21.15.171:36654) 2024-09-16T04:12:02.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d4b7fe640 1 -- 172.21.15.171:0/1304095352 learned_addr learned my addr 172.21.15.171:0/1304095352 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:02.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d4bfff640 1 -- 172.21.15.171:0/1304095352 >> v1:172.21.15.148:6790/0 conn(0x7f5d4c0a4040 legacy=0x7f5d4c1d39b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:02.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2714467378 0 0) 0x7f5d4c139e20 con 0x7f5d4c09fc20 2024-09-16T04:12:02.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:02.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:02.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:02.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): my global_id is 6184 2024-09-16T04:12:02.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d38003520 con 0x7f5d4c09fc20 2024-09-16T04:12:02.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3937223703 0 0) 0x7f5d4c09e0d0 con 0x7f5d4c0a0050 2024-09-16T04:12:02.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:02.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:02.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:02.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): my global_id is 64617 2024-09-16T04:12:02.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c139e20 con 0x7f5d4c0a0050 2024-09-16T04:12:02.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 619009232 0 0) 0x7f5d38003520 con 0x7f5d4c09fc20 2024-09-16T04:12:02.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d4c09e0d0 con 0x7f5d4c09fc20 2024-09-16T04:12:02.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5d3c001f00 con 0x7f5d4c09fc20 2024-09-16T04:12:02.111 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:02.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:02.112 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): dump: 2024-09-16T04:12:02.112 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:02.112 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:02.112 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.113 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.113 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:02.113 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:02.113 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:02.113 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:02.113 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:02.114 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:02.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:02.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2038626253 0 0) 0x7f5d4c09e0d0 con 0x7f5d4c09fc20 2024-09-16T04:12:02.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 >> v1:172.21.15.148:6790/0 conn(0x7f5d4c0a4040 legacy=0x7f5d4c1d39b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:02.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.074+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 >> v1:172.21.15.148:6789/0 conn(0x7f5d4c0a0050 legacy=0x7f5d4c1cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:02.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:02.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 1 monclient: found mon.b 2024-09-16T04:12:02.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:02.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c139b40 con 0x7f5d4c09fc20 2024-09-16T04:12:02.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: _finish_auth 0 2024-09-16T04:12:02.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:02.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:02.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 5 monclient: authenticate success, global_id 6184 2024-09-16T04:12:02.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 10 monclient: _renew_subs 2024-09-16T04:12:02.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4c072390 con 0x7f5d4c09fc20 2024-09-16T04:12:02.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 10 monclient: _renew_subs 2024-09-16T04:12:02.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d4c1d50d0 con 0x7f5d4c09fc20 2024-09-16T04:12:02.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d3c001bb0 con 0x7f5d4c09fc20 2024-09-16T04:12:02.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:02.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5d3c005110 con 0x7f5d4c09fc20 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d49ffb640 4 set_mon_vals no callback set 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 10 monclient: dump: 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:02.118 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.119 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:02.119 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:02.120 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:02.120 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:02.120 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:02.120 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:02.120 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:02.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.075+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5d3c005330 con 0x7f5d4c09fc20 2024-09-16T04:12:02.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.076+0000 7f5d51a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:02.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.076+0000 7f5d51a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:02.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.076+0000 7f5d51a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.076+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c005080 con 0x7f5d4c09fc20 2024-09-16T04:12:02.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.076+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(221..221 src has 1..221) ==== 12884+0+0 (unknown 962344682 0 0) 0x7f5d3c0496e0 con 0x7f5d4c09fc20 2024-09-16T04:12:02.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.076+0000 7f5d4affd640 10 monclient: get_auth_request con 0x7f5d38038f60 auth_method 0 2024-09-16T04:12:02.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.080+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5d3c016210 con 0x7f5d4c09fc20 2024-09-16T04:12:02.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.080+0000 7f5d33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:02.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.080+0000 7f5d33fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:02.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.249+0000 7f5d51a74640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T04:12:02.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.249+0000 7f5d51a74640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T04:12:02.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.249+0000 7f5d51a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.251 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.249+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) -- 0x7f5d2c005370 con 0x7f5d4c09fc20 2024-09-16T04:12:02.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.870+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v222) ==== 118+0+0 (unknown 1669829578 0 0) 0x7f5d3c01d120 con 0x7f5d4c09fc20 2024-09-16T04:12:02.873 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.870+0000 7f5d33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T04:12:02.874 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.870+0000 7f5d33fff640 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-16T04:12:02.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.977+0000 7f5d51a74640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T04:12:02.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.977+0000 7f5d51a74640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T04:12:02.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.977+0000 7f5d51a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:02.980 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.977+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) -- 0x7f5d2c002210 con 0x7f5d4c09fc20 2024-09-16T04:12:02.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.978+0000 7f5d33fff640 1 -- 172.21.15.171:0/1304095352 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v222) ==== 118+0+0 (unknown 3324453362 0 0) 0x7f5d3c01d550 con 0x7f5d4c09fc20 2024-09-16T04:12:02.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.978+0000 7f5d33fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-16T04:12:02.981 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.978+0000 7f5d33fff640 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-16T04:12:02.981 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h7' does not appear in the crush map 2024-09-16T04:12:02.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 >> v1:172.21.15.171:6812/7327273 conn(0x7f5d38038f60 legacy=0x7f5d3803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:02.983 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 10 monclient: shutdown 2024-09-16T04:12:02.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:02.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 >> v1:172.21.15.171:6789/0 conn(0x7f5d4c09fc20 legacy=0x7f5d4c1cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:02.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 shutdown_connections 2024-09-16T04:12:02.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 >> 172.21.15.171:0/1304095352 conn(0x7f5d4c0a7520 msgr2=0x7f5d4c0a2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:02.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 shutdown_connections 2024-09-16T04:12:02.984 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:02.980+0000 7f5d51a74640 1 -- 172.21.15.171:0/1304095352 wait complete. 2024-09-16T04:12:02.993 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:02.993 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h8 2024-09-16T04:12:03.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.068+0000 7fa95b303640 10 monclient: get_monmap_and_config 2024-09-16T04:12:03.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.068+0000 7fa95b303640 10 monclient: build_initial_monmap 2024-09-16T04:12:03.071 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.068+0000 7fa95b303640 10 monclient: monmap: 2024-09-16T04:12:03.071 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:03.070387+0000 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:03.070387+0000 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:03.072 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:03.073 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 1 Processor -- start 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 1 -- start start 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient: init 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient(hunting): picked mon.noname-a con 0x7fa95409f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient(hunting): picked mon.noname-c con 0x7fa9540a0130 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient(hunting): picked mon.noname-b con 0x7fa95409e1d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9540a1260 con 0x7fa95409f640 2024-09-16T04:12:03.082 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95409d0c0 con 0x7fa9540a0130 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95409d3a0 con 0x7fa95409e1d0 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95b303640 10 monclient(hunting): authenticate will time out at 1098.575846s 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95889e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa9540a0130 0x7fa954138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36670/0 (socket says 172.21.15.171:36670) 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa95889e640 1 -- 172.21.15.171:0/2851709986 learned_addr learned my addr 172.21.15.171:0/2851709986 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.069+0000 7fa9598a0640 1 -- 172.21.15.171:0/2851709986 >> v1:172.21.15.148:6790/0 conn(0x7fa95409e1d0 legacy=0x7fa95413bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936528126 0 0) 0x7fa95409d0c0 con 0x7fa9540a0130 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient(hunting): my global_id is 6187 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa940003520 con 0x7fa9540a0130 2024-09-16T04:12:03.083 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2327102770 0 0) 0x7fa940003520 con 0x7fa9540a0130 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 >> v1:172.21.15.148:6790/0 conn(0x7fa95409e1d0 legacy=0x7fa95413bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 >> v1:172.21.15.148:6789/0 conn(0x7fa95409f640 legacy=0x7fa95409fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 monclient: found mon.noname-c 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95409d680 con 0x7fa9540a0130 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: _finish_auth 0 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa95b303640 5 monclient: authenticate success, global_id 6187 2024-09-16T04:12:03.084 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa944001610 con 0x7fa9540a0130 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa95b303640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: dump: 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:03.085 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa95b303640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa944001900 con 0x7fa9540a0130 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa95b303640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 1 -- 172.21.15.171:0/2851709986 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa944001cd0 con 0x7fa9540a0130 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa95b303640 4 set_mon_vals no callback set 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:03.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.070+0000 7fa94bfff640 10 monclient: dump: 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:03.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 10 monclient: shutdown 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 1 -- 172.21.15.171:0/2851709986 >> v1:172.21.15.171:6789/0 conn(0x7fa9540a0130 legacy=0x7fa954138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 1 -- 172.21.15.171:0/2851709986 shutdown_connections 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 1 -- 172.21.15.171:0/2851709986 >> 172.21.15.171:0/2851709986 conn(0x7fa9540a52a0 msgr2=0x7fa9540a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 1 -- 172.21.15.171:0/2851709986 shutdown_connections 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 1 -- 172.21.15.171:0/2851709986 wait complete. 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 10 monclient: build_initial_monmap 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 10 monclient: monmap: 2024-09-16T04:12:03.088 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.071+0000 7fa95b303640 1 Processor -- start 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 1 -- start start 2024-09-16T04:12:03.089 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient: init 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient(hunting): picked mon.noname-a con 0x7fa95409e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient(hunting): picked mon.noname-b con 0x7fa95409f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient(hunting): picked mon.noname-c con 0x7fa9540a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9541a7e50 con 0x7fa95409e1d0 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9541aa910 con 0x7fa95409f640 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9541c36a0 con 0x7fa9540a0130 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95b303640 10 monclient(hunting): authenticate will time out at 1098.578503s 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95889e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa95409f640 0x7fa9541ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36680/0 (socket says 172.21.15.171:36680) 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa95889e640 1 -- 172.21.15.171:0/573620915 learned_addr learned my addr 172.21.15.171:0/573620915 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:03.090 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa9598a0640 1 -- 172.21.15.171:0/573620915 >> v1:172.21.15.148:6790/0 conn(0x7fa9540a0130 legacy=0x7fa9541d1a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2180535699 0 0) 0x7fa9541aa910 con 0x7fa95409f640 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): my global_id is 6190 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa938003520 con 0x7fa95409f640 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980507549 0 0) 0x7fa9541a7e50 con 0x7fa95409e1d0 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 10 monclient(hunting): my global_id is 64623 2024-09-16T04:12:03.091 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9541aa910 con 0x7fa95409e1d0 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.072+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3222134034 0 0) 0x7fa938003520 con 0x7fa95409f640 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9541a7e50 con 0x7fa95409f640 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9440015d0 con 0x7fa95409f640 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient(hunting): dump: 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.092 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3080193879 0 0) 0x7fa9541a7e50 con 0x7fa95409f640 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 >> v1:172.21.15.148:6790/0 conn(0x7fa9540a0130 legacy=0x7fa9541d1a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 >> v1:172.21.15.148:6789/0 conn(0x7fa95409e1d0 legacy=0x7fa9541c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:12:03.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 monclient: found mon.b 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95406ceb0 con 0x7fa95409f640 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 5 monclient: authenticate success, global_id 6190 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 10 monclient: _renew_subs 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9540721e0 con 0x7fa95409f640 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa944004cc0 con 0x7fa95409f640 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:03.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa944005090 con 0x7fa95409f640 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa949ffb640 10 monclient: dump: 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:03.095 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa94b7fe640 4 set_mon_vals no callback set 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 10 monclient: _renew_subs 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.073+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9541aa630 con 0x7fa95409f640 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa944005230 con 0x7fa95409f640 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa95b303640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa95b303640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:03.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa95b303640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(222..222 src has 1..222) ==== 12674+0+0 (unknown 411414947 0 0) 0x7fa944049210 con 0x7fa95409f640 2024-09-16T04:12:03.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9541a36c0 con 0x7fa95409f640 2024-09-16T04:12:03.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.074+0000 7fa95909f640 10 monclient: get_auth_request con 0x7fa938038f60 auth_method 0 2024-09-16T04:12:03.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.078+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa944015e10 con 0x7fa95409f640 2024-09-16T04:12:03.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.078+0000 7fa949ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:03.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.078+0000 7fa949ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:03.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.250+0000 7fa95b303640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T04:12:03.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.250+0000 7fa95b303640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T04:12:03.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.250+0000 7fa95b303640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.253 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.250+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) -- 0x7fa95406bcf0 con 0x7fa95409f640 2024-09-16T04:12:03.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.881+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v223) ==== 118+0+0 (unknown 1727027295 0 0) 0x7fa944015710 con 0x7fa95409f640 2024-09-16T04:12:03.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.881+0000 7fa949ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T04:12:03.884 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.881+0000 7fa949ffb640 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-16T04:12:03.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.990+0000 7fa95b303640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T04:12:03.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.990+0000 7fa95b303640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T04:12:03.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.990+0000 7fa95b303640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:03.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.990+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) -- 0x7fa9541a93c0 con 0x7fa95409f640 2024-09-16T04:12:03.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.991+0000 7fa949ffb640 1 -- 172.21.15.171:0/573620915 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v223) ==== 118+0+0 (unknown 362628605 0 0) 0x7fa94401e260 con 0x7fa95409f640 2024-09-16T04:12:03.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.991+0000 7fa949ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-16T04:12:03.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.991+0000 7fa949ffb640 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-16T04:12:03.994 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h8' does not appear in the crush map 2024-09-16T04:12:03.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 >> v1:172.21.15.171:6812/7327273 conn(0x7fa938038f60 legacy=0x7fa93803b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 10 monclient: shutdown 2024-09-16T04:12:03.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:03.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 >> v1:172.21.15.171:6789/0 conn(0x7fa95409f640 legacy=0x7fa9541ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 shutdown_connections 2024-09-16T04:12:03.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 >> 172.21.15.171:0/573620915 conn(0x7fa9540a52a0 msgr2=0x7fa9540a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:03.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.993+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 shutdown_connections 2024-09-16T04:12:03.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:03.994+0000 7fa95b303640 1 -- 172.21.15.171:0/573620915 wait complete. 2024-09-16T04:12:04.005 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:04.005 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h9 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.082+0000 7fa6240f1640 10 monclient: get_monmap_and_config 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.082+0000 7fa6240f1640 10 monclient: build_initial_monmap 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.082+0000 7fa6240f1640 10 monclient: monmap: 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:04.084247+0000 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:04.084247+0000 2024-09-16T04:12:04.085 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:04.086 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:04.086 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:04.086 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:04.086 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:04.086 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 1 Processor -- start 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 1 -- start start 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient: init 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa61c132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa61c09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa61c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c1332b0 con 0x7fa61c132a80 2024-09-16T04:12:04.093 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c134310 con 0x7fa61c09ec20 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c09d980 con 0x7fa61c132e60 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa6240f1640 10 monclient(hunting): authenticate will time out at 1099.589787s 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa62268e640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa61c132e60 legacy=0x7fa61c13c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa62168c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa61c09ec20 0x7fa61c133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36692/0 (socket says 172.21.15.171:36692) 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.083+0000 7fa62168c640 1 -- 172.21.15.171:0/2841344762 learned_addr learned my addr 172.21.15.171:0/2841344762 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461554945 0 0) 0x7fa61c134310 con 0x7fa61c09ec20 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): my global_id is 6193 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa608003520 con 0x7fa61c09ec20 2024-09-16T04:12:04.094 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260691221 0 0) 0x7fa61c1332b0 con 0x7fa61c132a80 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient(hunting): my global_id is 64626 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61c134310 con 0x7fa61c132a80 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1666243497 0 0) 0x7fa608003520 con 0x7fa61c09ec20 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 >> v1:172.21.15.148:6790/0 conn(0x7fa61c132e60 legacy=0x7fa61c13c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 >> v1:172.21.15.148:6789/0 conn(0x7fa61c132a80 legacy=0x7fa61c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: _finish_hunting 0 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 monclient: found mon.noname-c 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c09dc60 con 0x7fa61c09ec20 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: _finish_auth 0 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: _check_auth_tickets 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa60c001590 con 0x7fa61c09ec20 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa6240f1640 5 monclient: authenticate success, global_id 6193 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa6240f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: dump: 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.096 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa6240f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa60c0017f0 con 0x7fa61c09ec20 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa6240f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 1 -- 172.21.15.171:0/2841344762 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa60c001b40 con 0x7fa61c09ec20 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa6240f1640 4 set_mon_vals no callback set 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:04.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.084+0000 7fa620e8b640 10 monclient: dump: 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 10 monclient: shutdown 2024-09-16T04:12:04.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 1 -- 172.21.15.171:0/2841344762 >> v1:172.21.15.171:6789/0 conn(0x7fa61c09ec20 legacy=0x7fa61c133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 1 -- 172.21.15.171:0/2841344762 shutdown_connections 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 1 -- 172.21.15.171:0/2841344762 >> 172.21.15.171:0/2841344762 conn(0x7fa61c0a52a0 msgr2=0x7fa61c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 1 -- 172.21.15.171:0/2841344762 shutdown_connections 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 1 -- 172.21.15.171:0/2841344762 wait complete. 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 10 monclient: build_initial_monmap 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 10 monclient: monmap: 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:04.099 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.085+0000 7fa6240f1640 1 Processor -- start 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 1 -- start start 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient: init 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa61c09ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa61c132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa61c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:04.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c0a3ef0 con 0x7fa61c132a80 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c09c9e0 con 0x7fa61c09ec20 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c122940 con 0x7fa61c132e60 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6240f1640 10 monclient(hunting): authenticate will time out at 1099.592507s 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa621e8d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa61c09ec20 0x7fa61c12e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36694/0 (socket says 172.21.15.171:36694) 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa621e8d640 1 -- 172.21.15.171:0/4280086530 learned_addr learned my addr 172.21.15.171:0/4280086530 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa62268e640 1 -- 172.21.15.171:0/4280086530 >> v1:172.21.15.148:6790/0 conn(0x7fa61c132e60 legacy=0x7fa61c1d3ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906510687 0 0) 0x7fa61c134310 con 0x7fa61c09ec20 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): my global_id is 6196 2024-09-16T04:12:04.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa600003520 con 0x7fa61c09ec20 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2786625578 0 0) 0x7fa600003520 con 0x7fa61c09ec20 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61c134310 con 0x7fa61c09ec20 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa618001970 con 0x7fa61c09ec20 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): dump: 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:04.102 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.086+0000 7fa6127fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482651507 0 0) 0x7fa61c09dc60 con 0x7fa61c132a80 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient(hunting): my global_id is 64629 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa600003520 con 0x7fa61c132a80 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 702304145 0 0) 0x7fa61c134310 con 0x7fa61c09ec20 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 >> v1:172.21.15.148:6790/0 conn(0x7fa61c132e60 legacy=0x7fa61c1d3ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 >> v1:172.21.15.148:6789/0 conn(0x7fa61c132a80 legacy=0x7fa61c1d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:04.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 monclient: found mon.b 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c122c20 con 0x7fa61c09ec20 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 5 monclient: authenticate success, global_id 6196 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 10 monclient: _renew_subs 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa61c122f00 con 0x7fa61c09ec20 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 10 monclient: _renew_subs 2024-09-16T04:12:04.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa61c072cf0 con 0x7fa61c09ec20 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa618003940 con 0x7fa61c09ec20 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa618003d10 con 0x7fa61c09ec20 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa613fff640 4 set_mon_vals no callback set 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6127fc640 10 monclient: dump: 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:04.105 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.087+0000 7fa6240f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.088+0000 7fa6240f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.088+0000 7fa6240f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.088+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa618003f30 con 0x7fa61c09ec20 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.088+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa604005080 con 0x7fa61c09ec20 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.088+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(223..223 src has 1..223) ==== 12464+0+0 (unknown 2467929421 0 0) 0x7fa6180491f0 con 0x7fa61c09ec20 2024-09-16T04:12:04.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.088+0000 7fa62168c640 10 monclient: get_auth_request con 0x7fa600038f70 auth_method 0 2024-09-16T04:12:04.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.092+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa61c09c9e0 con 0x7fa61c09ec20 2024-09-16T04:12:04.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.092+0000 7fa6127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:04.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.092+0000 7fa6127fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:04.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.264+0000 7fa6240f1640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T04:12:04.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.264+0000 7fa6240f1640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T04:12:04.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.264+0000 7fa6240f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.267 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.264+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) -- 0x7fa604005370 con 0x7fa61c09ec20 2024-09-16T04:12:04.886 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.883+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v224) ==== 118+0+0 (unknown 3143411996 0 0) 0x7fa618019a50 con 0x7fa61c09ec20 2024-09-16T04:12:04.886 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.883+0000 7fa6127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T04:12:04.886 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.883+0000 7fa6127fc640 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-16T04:12:04.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.990+0000 7fa6240f1640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T04:12:04.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.990+0000 7fa6240f1640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T04:12:04.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.990+0000 7fa6240f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:04.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.990+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) -- 0x7fa604001090 con 0x7fa61c09ec20 2024-09-16T04:12:04.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.991+0000 7fa6127fc640 1 -- 172.21.15.171:0/4280086530 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v224) ==== 118+0+0 (unknown 192175842 0 0) 0x7fa61801e310 con 0x7fa61c09ec20 2024-09-16T04:12:04.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.991+0000 7fa6127fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-16T04:12:04.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.991+0000 7fa6127fc640 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-16T04:12:04.994 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h9' does not appear in the crush map 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 >> v1:172.21.15.171:6812/7327273 conn(0x7fa600038f70 legacy=0x7fa60003b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 10 monclient: shutdown 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 >> v1:172.21.15.171:6789/0 conn(0x7fa61c09ec20 legacy=0x7fa61c12e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 shutdown_connections 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 >> 172.21.15.171:0/4280086530 conn(0x7fa61c0a52a0 msgr2=0x7fa61c09bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 shutdown_connections 2024-09-16T04:12:04.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:04.994+0000 7fa6240f1640 1 -- 172.21.15.171:0/4280086530 wait complete. 2024-09-16T04:12:05.006 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:05.007 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h10 2024-09-16T04:12:05.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.083+0000 7fc9eceaf640 10 monclient: get_monmap_and_config 2024-09-16T04:12:05.086 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.083+0000 7fc9eceaf640 10 monclient: build_initial_monmap 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.083+0000 7fc9eceaf640 10 monclient: monmap: 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:05.085115+0000 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:05.085115+0000 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.087 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:05.088 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:05.088 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:05.088 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.083+0000 7fc9eceaf640 1 Processor -- start 2024-09-16T04:12:05.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.083+0000 7fc9eceaf640 1 -- start start 2024-09-16T04:12:05.088 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.083+0000 7fc9eceaf640 10 monclient: init 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e8132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e8132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e81332b0 con 0x7fc9e8132e60 2024-09-16T04:12:05.095 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e8134310 con 0x7fc9e8132a80 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e809d980 con 0x7fc9e809ec20 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9eceaf640 10 monclient(hunting): authenticate will time out at 1100.590548s 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e659c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc9e8132a80 0x7fc9e81334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36698/0 (socket says 172.21.15.171:36698) 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e659c640 1 -- 172.21.15.171:0/203544053 learned_addr learned my addr 172.21.15.171:0/203544053 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e5d9b640 1 -- 172.21.15.171:0/203544053 >> v1:172.21.15.148:6790/0 conn(0x7fc9e809ec20 legacy=0x7fc9e8133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021510741 0 0) 0x7fc9e8134310 con 0x7fc9e8132a80 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e559a640 10 monclient(hunting): _negotiate 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e559a640 10 monclient(hunting): my global_id is 6199 2024-09-16T04:12:05.096 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.084+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9d0003520 con 0x7fc9e8132a80 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066599738 0 0) 0x7fc9e81332b0 con 0x7fc9e8132e60 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient(hunting): _negotiate 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient(hunting): my global_id is 64632 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e8134310 con 0x7fc9e8132e60 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 181101023 0 0) 0x7fc9d0003520 con 0x7fc9e8132a80 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 >> v1:172.21.15.148:6790/0 conn(0x7fc9e809ec20 legacy=0x7fc9e8133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 >> v1:172.21.15.148:6789/0 conn(0x7fc9e8132e60 legacy=0x7fc9e813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: _finish_hunting 0 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 monclient: found mon.noname-c 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:05.097 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e809dc60 con 0x7fc9e8132a80 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: _finish_auth 0 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: _check_auth_tickets 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d8001590 con 0x7fc9e8132a80 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 5 monclient: authenticate success, global_id 6199 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: dump: 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:05.098 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9d80017f0 con 0x7fc9e8132a80 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 1 -- 172.21.15.171:0/203544053 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d8001b40 con 0x7fc9e8132a80 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 4 set_mon_vals no callback set 2024-09-16T04:12:05.099 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9e559a640 10 monclient: dump: 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:05.100 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 10 monclient: shutdown 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 1 -- 172.21.15.171:0/203544053 >> v1:172.21.15.171:6789/0 conn(0x7fc9e8132a80 legacy=0x7fc9e81334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.085+0000 7fc9eceaf640 1 -- 172.21.15.171:0/203544053 shutdown_connections 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.086+0000 7fc9eceaf640 1 -- 172.21.15.171:0/203544053 >> 172.21.15.171:0/203544053 conn(0x7fc9e80a52a0 msgr2=0x7fc9e80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.086+0000 7fc9eceaf640 1 -- 172.21.15.171:0/203544053 shutdown_connections 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.086+0000 7fc9eceaf640 1 -- 172.21.15.171:0/203544053 wait complete. 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.086+0000 7fc9eceaf640 10 monclient: build_initial_monmap 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.086+0000 7fc9eceaf640 10 monclient: monmap: 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:05.101 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.086+0000 7fc9eceaf640 1 Processor -- start 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 1 -- start start 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient: init 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e809ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:05.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e8132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e8132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e80a33f0 con 0x7fc9e809ec20 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e8122040 con 0x7fc9e8132a80 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e8120580 con 0x7fc9e8132e60 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9eceaf640 10 monclient(hunting): authenticate will time out at 1100.593488s 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9e5d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc9e8132a80 0x7fc9e811fe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36708/0 (socket says 172.21.15.171:36708) 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9e5d9b640 1 -- 172.21.15.171:0/2607723730 learned_addr learned my addr 172.21.15.171:0/2607723730 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9e6d9d640 1 -- 172.21.15.171:0/2607723730 >> v1:172.21.15.148:6790/0 conn(0x7fc9e8132e60 legacy=0x7fc9e81d1860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249908651 0 0) 0x7fc9e8122040 con 0x7fc9e8132a80 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:05.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): my global_id is 6202 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c8003520 con 0x7fc9e8132a80 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565690248 0 0) 0x7fc9e809dc60 con 0x7fc9e809ec20 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): my global_id is 64635 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e8122040 con 0x7fc9e809ec20 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3465745384 0 0) 0x7fc9c8003520 con 0x7fc9e8132a80 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e809dc60 con 0x7fc9e8132a80 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9dc004180 con 0x7fc9e8132a80 2024-09-16T04:12:05.104 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.087+0000 7fc9d6ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:05.105 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3786535940 0 0) 0x7fc9e8122040 con 0x7fc9e809ec20 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c8003520 con 0x7fc9e809ec20 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d8001510 con 0x7fc9e809ec20 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:05.106 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1389812610 0 0) 0x7fc9e809dc60 con 0x7fc9e8132a80 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 >> v1:172.21.15.148:6790/0 conn(0x7fc9e8132e60 legacy=0x7fc9e81d1860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 >> v1:172.21.15.148:6789/0 conn(0x7fc9e809ec20 legacy=0x7fc9e811f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 monclient: found mon.b 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e8131a50 con 0x7fc9e8132a80 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:05.107 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 5 monclient: authenticate success, global_id 6202 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 10 monclient: _renew_subs 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e8122600 con 0x7fc9e8132a80 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 10 monclient: _renew_subs 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9e8122320 con 0x7fc9e8132a80 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9dc0044a0 con 0x7fc9e8132a80 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9dc004870 con 0x7fc9e8132a80 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:05.108 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9e4d99640 4 set_mon_vals no callback set 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 10 monclient: dump: 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.088+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc9dc004a90 con 0x7fc9e8132a80 2024-09-16T04:12:05.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.089+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(224..224 src has 1..224) ==== 12254+0+0 (unknown 250349417 0 0) 0x7fc9dc049050 con 0x7fc9e8132a80 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.089+0000 7fc9e659c640 10 monclient: get_auth_request con 0x7fc9c8038f10 auth_method 0 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.089+0000 7fc9eceaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.089+0000 7fc9eceaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.089+0000 7fc9eceaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.089+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9cc005080 con 0x7fc9e8132a80 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.093+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc9dc015d90 con 0x7fc9e8132a80 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.093+0000 7fc9d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:05.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.093+0000 7fc9d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:05.265 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.263+0000 7fc9eceaf640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T04:12:05.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.263+0000 7fc9eceaf640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T04:12:05.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.263+0000 7fc9eceaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.266 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.263+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) -- 0x7fc9cc005370 con 0x7fc9e8132a80 2024-09-16T04:12:05.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.831+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v225) ==== 120+0+0 (unknown 656416124 0 0) 0x7fc9dc015690 con 0x7fc9e8132a80 2024-09-16T04:12:05.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.831+0000 7fc9d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T04:12:05.834 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.831+0000 7fc9d6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-16T04:12:05.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.936+0000 7fc9eceaf640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T04:12:05.940 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.936+0000 7fc9eceaf640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T04:12:05.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.936+0000 7fc9eceaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:05.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.936+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) -- 0x7fc9cc002210 con 0x7fc9e8132a80 2024-09-16T04:12:05.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.938+0000 7fc9d6ffd640 1 -- 172.21.15.171:0/2607723730 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v225) ==== 120+0+0 (unknown 3714582501 0 0) 0x7fc9dc01e240 con 0x7fc9e8132a80 2024-09-16T04:12:05.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.938+0000 7fc9d6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-16T04:12:05.941 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.938+0000 7fc9d6ffd640 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-16T04:12:05.941 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h10' does not appear in the crush map 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.939+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 >> v1:172.21.15.171:6812/7327273 conn(0x7fc9c8038f10 legacy=0x7fc9c803b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 10 monclient: shutdown 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 >> v1:172.21.15.171:6789/0 conn(0x7fc9e8132a80 legacy=0x7fc9e811fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 shutdown_connections 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 >> 172.21.15.171:0/2607723730 conn(0x7fc9e80a52a0 msgr2=0x7fc9e809bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 shutdown_connections 2024-09-16T04:12:05.943 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:05.940+0000 7fc9eceaf640 1 -- 172.21.15.171:0/2607723730 wait complete. 2024-09-16T04:12:05.952 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:05.952 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h11 2024-09-16T04:12:06.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.026+0000 7f13dc56d640 10 monclient: get_monmap_and_config 2024-09-16T04:12:06.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.026+0000 7f13dc56d640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.026+0000 7f13dc56d640 10 monclient: monmap: 2024-09-16T04:12:06.029 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:06.028569+0000 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:06.028569+0000 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:06.030 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.027+0000 7f13dc56d640 1 Processor -- start 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.027+0000 7f13dc56d640 1 -- start start 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.027+0000 7f13dc56d640 10 monclient: init 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient(hunting): picked mon.noname-a con 0x7f13d409ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:06.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient(hunting): picked mon.noname-c con 0x7f13d40a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient(hunting): picked mon.noname-b con 0x7f13d409e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d409fbf0 con 0x7f13d409ba30 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d40a0e10 con 0x7f13d40a0750 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d409f640 con 0x7f13d409e570 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient(hunting): authenticate will time out at 1101.534167s 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dab0a640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f13d409e570 legacy=0x7f13d413d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9b08640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f13d40a0750 0x7f13d41399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36718/0 (socket says 172.21.15.171:36718) 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9b08640 1 -- 172.21.15.171:0/2484238886 learned_addr learned my addr 172.21.15.171:0/2484238886 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2590871506 0 0) 0x7f13d40a0e10 con 0x7f13d40a0750 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): _negotiate 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): my global_id is 6205 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c0003520 con 0x7f13d40a0750 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2456307619 0 0) 0x7f13d409fbf0 con 0x7f13d409ba30 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): _negotiate 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient(hunting): my global_id is 64638 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d40a0e10 con 0x7f13d409ba30 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3662384693 0 0) 0x7f13c0003520 con 0x7f13d40a0750 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 >> v1:172.21.15.148:6790/0 conn(0x7f13d409e570 legacy=0x7f13d413d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 >> v1:172.21.15.148:6789/0 conn(0x7f13d409ba30 legacy=0x7f13d40a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 monclient: found mon.noname-c 2024-09-16T04:12:06.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d409f810 con 0x7f13d40a0750 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13c4001610 con 0x7f13d40a0750 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: dump: 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.040 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 5 monclient: authenticate success, global_id 6205 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c4001930 con 0x7f13d40a0750 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 1 -- 172.21.15.171:0/2484238886 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13c4001d00 con 0x7f13d40a0750 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13dc56d640 4 set_mon_vals no callback set 2024-09-16T04:12:06.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.028+0000 7f13d9307640 10 monclient: dump: 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 10 monclient: shutdown 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 1 -- 172.21.15.171:0/2484238886 >> v1:172.21.15.171:6789/0 conn(0x7f13d40a0750 legacy=0x7f13d41399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 1 -- 172.21.15.171:0/2484238886 shutdown_connections 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 1 -- 172.21.15.171:0/2484238886 >> 172.21.15.171:0/2484238886 conn(0x7f13d40a52a0 msgr2=0x7f13d40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 1 -- 172.21.15.171:0/2484238886 shutdown_connections 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.029+0000 7f13dc56d640 1 -- 172.21.15.171:0/2484238886 wait complete. 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient: monmap: 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 1 Processor -- start 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 1 -- start start 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient: init 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient(hunting): picked mon.noname-c con 0x7f13d409ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient(hunting): picked mon.noname-b con 0x7f13d409e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient(hunting): picked mon.noname-a con 0x7f13d40a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d412ede0 con 0x7f13d40a0750 2024-09-16T04:12:06.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d412c670 con 0x7f13d409e570 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13da309640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f13d409ba30 legacy=0x7f13d412b850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d412f0c0 con 0x7f13d409ba30 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13dc56d640 10 monclient(hunting): authenticate will time out at 1101.536900s 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13d9b08640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f13d409e570 0x7f13d412bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36728/0 (socket says 172.21.15.171:36728) 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.030+0000 7f13d9b08640 1 -- 172.21.15.171:0/1285579173 learned_addr learned my addr 172.21.15.171:0/1285579173 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566702572 0 0) 0x7f13d412c670 con 0x7f13d409e570 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): my global_id is 6208 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13b8003520 con 0x7f13d409e570 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3431671285 0 0) 0x7f13d412ede0 con 0x7f13d40a0750 2024-09-16T04:12:06.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): my global_id is 64641 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d412c670 con 0x7f13d40a0750 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3752126910 0 0) 0x7f13b8003520 con 0x7f13d409e570 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13d412ede0 con 0x7f13d409e570 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13c40015d0 con 0x7f13d409e570 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): dump: 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:06.046 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2083378155 0 0) 0x7f13d412c670 con 0x7f13d40a0750 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13b8003520 con 0x7f13d40a0750 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13cc0043b0 con 0x7f13d40a0750 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient(hunting): dump: 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1979705551 0 0) 0x7f13d412ede0 con 0x7f13d409e570 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 >> v1:172.21.15.148:6790/0 conn(0x7f13d409ba30 legacy=0x7f13d412b850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:06.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 >> v1:172.21.15.148:6789/0 conn(0x7f13d40a0750 legacy=0x7f13d41d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 monclient: found mon.b 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d4132a50 con 0x7f13d409e570 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 5 monclient: authenticate success, global_id 6208 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 10 monclient: _renew_subs 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13d406ceb0 con 0x7f13d409e570 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 10 monclient: _renew_subs 2024-09-16T04:12:06.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f13d412d220 con 0x7f13d409e570 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c4001930 con 0x7f13d409e570 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13c4004e10 con 0x7f13d409e570 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13d8b06640 4 set_mon_vals no callback set 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.031+0000 7f13caffd640 10 monclient: dump: 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:06.050 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f13c4005030 con 0x7f13d409e570 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13dc56d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13dc56d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13dc56d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13bc005080 con 0x7f13d409e570 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(225..225 src has 1..225) ==== 12041+0+0 (unknown 2105636450 0 0) 0x7f13c4048fa0 con 0x7f13d409e570 2024-09-16T04:12:06.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.032+0000 7f13da309640 10 monclient: get_auth_request con 0x7f13b8038ea0 auth_method 0 2024-09-16T04:12:06.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.037+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f13c4015ed0 con 0x7f13d409e570 2024-09-16T04:12:06.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.037+0000 7f13caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.037+0000 7f13caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:06.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.209+0000 7f13dc56d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T04:12:06.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.209+0000 7f13dc56d640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T04:12:06.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.209+0000 7f13dc56d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:06.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.209+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) -- 0x7f13bc005370 con 0x7f13d409e570 2024-09-16T04:12:06.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.908+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v226) ==== 120+0+0 (unknown 1751865568 0 0) 0x7f13c40157d0 con 0x7f13d409e570 2024-09-16T04:12:06.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.908+0000 7f13caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T04:12:06.911 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:06.908+0000 7f13caffd640 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-16T04:12:07.018 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.015+0000 7f13dc56d640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T04:12:07.018 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.015+0000 7f13dc56d640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T04:12:07.018 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.015+0000 7f13dc56d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.018 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.015+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) -- 0x7f13bc002210 con 0x7f13d409e570 2024-09-16T04:12:07.019 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.016+0000 7f13caffd640 1 -- 172.21.15.171:0/1285579173 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v226) ==== 120+0+0 (unknown 1931931246 0 0) 0x7f13c4013b30 con 0x7f13d409e570 2024-09-16T04:12:07.019 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.016+0000 7f13caffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-16T04:12:07.019 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.016+0000 7f13caffd640 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-16T04:12:07.019 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h11' does not appear in the crush map 2024-09-16T04:12:07.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 >> v1:172.21.15.171:6812/7327273 conn(0x7f13b8038ea0 legacy=0x7f13b803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.021 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 10 monclient: shutdown 2024-09-16T04:12:07.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:07.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 >> v1:172.21.15.171:6789/0 conn(0x7f13d409e570 legacy=0x7f13d412bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 shutdown_connections 2024-09-16T04:12:07.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 >> 172.21.15.171:0/1285579173 conn(0x7f13d40a52a0 msgr2=0x7f13d4131cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:07.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.018+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 shutdown_connections 2024-09-16T04:12:07.022 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.019+0000 7f13dc56d640 1 -- 172.21.15.171:0/1285579173 wait complete. 2024-09-16T04:12:07.030 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:07.031 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h12 2024-09-16T04:12:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.106+0000 7f40ea63c640 10 monclient: get_monmap_and_config 2024-09-16T04:12:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.106+0000 7f40ea63c640 10 monclient: build_initial_monmap 2024-09-16T04:12:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.106+0000 7f40ea63c640 10 monclient: monmap: 2024-09-16T04:12:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:07.109 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:07.108350+0000 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:07.108350+0000 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:07.110 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:07.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 1 Processor -- start 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 1 -- start start 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient: init 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient(hunting): picked mon.noname-a con 0x7f40e4132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient(hunting): picked mon.noname-c con 0x7f40e409ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient(hunting): picked mon.noname-b con 0x7f40e4132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e41332b0 con 0x7f40e4132a80 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e4134310 con 0x7f40e409ec20 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e409d980 con 0x7f40e4132e60 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:07.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40ea63c640 10 monclient(hunting): authenticate will time out at 1102.613842s 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40e8bd9640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f40e4132e60 legacy=0x7f40e413c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40e37fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f40e409ec20 0x7f40e4133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36744/0 (socket says 172.21.15.171:36744) 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.107+0000 7f40e37fe640 1 -- 172.21.15.171:0/3728224938 learned_addr learned my addr 172.21.15.171:0/3728224938 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896219040 0 0) 0x7f40e4134310 con 0x7f40e409ec20 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): my global_id is 6211 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40cc003520 con 0x7f40e409ec20 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486049555 0 0) 0x7f40e41332b0 con 0x7f40e4132a80 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:07.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient(hunting): my global_id is 64644 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40e4134310 con 0x7f40e4132a80 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 935056380 0 0) 0x7f40cc003520 con 0x7f40e409ec20 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 >> v1:172.21.15.148:6790/0 conn(0x7f40e4132e60 legacy=0x7f40e413c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 >> v1:172.21.15.148:6789/0 conn(0x7f40e4132a80 legacy=0x7f40e41334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 monclient: found mon.noname-c 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e409dc60 con 0x7f40e409ec20 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:07.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40d8001610 con 0x7f40e409ec20 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40ea63c640 5 monclient: authenticate success, global_id 6211 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40ea63c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: dump: 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:07.121 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40ea63c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40d80018c0 con 0x7f40e409ec20 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40ea63c640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40ea63c640 4 set_mon_vals no callback set 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 1 -- 172.21.15.171:0/3728224938 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40d8001c90 con 0x7f40e409ec20 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.108+0000 7f40e2ffd640 10 monclient: dump: 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:07.122 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 10 monclient: shutdown 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 1 -- 172.21.15.171:0/3728224938 >> v1:172.21.15.171:6789/0 conn(0x7f40e409ec20 legacy=0x7f40e4133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 1 -- 172.21.15.171:0/3728224938 shutdown_connections 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 1 -- 172.21.15.171:0/3728224938 >> 172.21.15.171:0/3728224938 conn(0x7f40e40a52a0 msgr2=0x7f40e40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 1 -- 172.21.15.171:0/3728224938 shutdown_connections 2024-09-16T04:12:07.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 1 -- 172.21.15.171:0/3728224938 wait complete. 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 10 monclient: build_initial_monmap 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.109+0000 7f40ea63c640 10 monclient: monmap: 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:07.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 1 Processor -- start 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 1 -- start start 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient: init 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient(hunting): picked mon.noname-b con 0x7f40e409ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient(hunting): picked mon.noname-a con 0x7f40e4132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient(hunting): picked mon.noname-c con 0x7f40e4132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e411c9b0 con 0x7f40e4132a80 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e411c0b0 con 0x7f40e409ec20 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e411e820 con 0x7f40e4132e60 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40ea63c640 10 monclient(hunting): authenticate will time out at 1102.616657s 2024-09-16T04:12:07.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e3fff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f40e409ec20 0x7f40e411b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:36746/0 (socket says 172.21.15.171:36746) 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e3fff640 1 -- 172.21.15.171:0/341381062 learned_addr learned my addr 172.21.15.171:0/341381062 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e8bd9640 1 -- 172.21.15.171:0/341381062 >> v1:172.21.15.148:6790/0 conn(0x7f40e4132e60 legacy=0x7f40e41cf630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282238175 0 0) 0x7f40e411c0b0 con 0x7f40e409ec20 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e0ff9640 10 monclient(hunting): my global_id is 6214 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.110+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40d0003520 con 0x7f40e409ec20 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2860393677 0 0) 0x7f40d0003520 con 0x7f40e409ec20 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40e411c0b0 con 0x7f40e409ec20 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40d4001910 con 0x7f40e409ec20 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:07.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): dump: 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559490298 0 0) 0x7f40e411c9b0 con 0x7f40e4132a80 2024-09-16T04:12:07.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient(hunting): my global_id is 64647 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40d0003520 con 0x7f40e4132a80 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1071080715 0 0) 0x7f40e411c0b0 con 0x7f40e409ec20 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 >> v1:172.21.15.148:6790/0 conn(0x7f40e4132e60 legacy=0x7f40e41cf630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 >> v1:172.21.15.148:6789/0 conn(0x7f40e4132a80 legacy=0x7f40e411b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 monclient: found mon.b 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e40a3e60 con 0x7f40e409ec20 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:12:07.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 5 monclient: authenticate success, global_id 6214 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 10 monclient: _renew_subs 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40e409c940 con 0x7f40e409ec20 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 10 monclient: _renew_subs 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f40e411e540 con 0x7f40e409ec20 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40d4001d20 con 0x7f40e409ec20 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f40d4003ae0 con 0x7f40e409ec20 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:07.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e27fc640 4 set_mon_vals no callback set 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.111+0000 7f40e0ff9640 10 monclient: dump: 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f40d4003c80 con 0x7f40e409ec20 2024-09-16T04:12:07.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40ea63c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:07.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40ea63c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:07.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40ea63c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c4005080 con 0x7f40e409ec20 2024-09-16T04:12:07.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(226..226 src has 1..226) ==== 11828+0+0 (unknown 3421203097 0 0) 0x7f40d4048f70 con 0x7f40e409ec20 2024-09-16T04:12:07.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.112+0000 7f40e37fe640 10 monclient: get_auth_request con 0x7f40d003d430 auth_method 0 2024-09-16T04:12:07.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.117+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f40d4015e00 con 0x7f40e409ec20 2024-09-16T04:12:07.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.117+0000 7f40e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:07.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.117+0000 7f40e0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:07.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.287+0000 7f40ea63c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T04:12:07.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.287+0000 7f40ea63c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T04:12:07.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.287+0000 7f40ea63c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:07.294 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.287+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) -- 0x7f40c4005370 con 0x7f40e409ec20 2024-09-16T04:12:07.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.914+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v227) ==== 120+0+0 (unknown 1250119241 0 0) 0x7f40d4015700 con 0x7f40e409ec20 2024-09-16T04:12:07.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.914+0000 7f40e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T04:12:07.917 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:07.914+0000 7f40e0ff9640 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-16T04:12:08.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.020+0000 7f40ea63c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T04:12:08.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.020+0000 7f40ea63c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T04:12:08.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.020+0000 7f40ea63c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.023 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.020+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) -- 0x7f40c4001090 con 0x7f40e409ec20 2024-09-16T04:12:08.024 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.022+0000 7f40e0ff9640 1 -- 172.21.15.171:0/341381062 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v227) ==== 120+0+0 (unknown 2968789066 0 0) 0x7f40d4019a50 con 0x7f40e409ec20 2024-09-16T04:12:08.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.022+0000 7f40e0ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-16T04:12:08.025 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.022+0000 7f40e0ff9640 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-16T04:12:08.025 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h12' does not appear in the crush map 2024-09-16T04:12:08.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 >> v1:172.21.15.171:6812/7327273 conn(0x7f40d003d430 legacy=0x7f40d003f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:08.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 10 monclient: shutdown 2024-09-16T04:12:08.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:08.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 >> v1:172.21.15.171:6789/0 conn(0x7f40e409ec20 legacy=0x7f40e411b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:08.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 shutdown_connections 2024-09-16T04:12:08.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 >> 172.21.15.171:0/341381062 conn(0x7f40e40a52a0 msgr2=0x7f40e4133290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:08.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 shutdown_connections 2024-09-16T04:12:08.028 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.024+0000 7f40ea63c640 1 -- 172.21.15.171:0/341381062 wait complete. 2024-09-16T04:12:08.035 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:08.035 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h13 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.111+0000 7f50f95d8640 10 monclient: get_monmap_and_config 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.111+0000 7f50f95d8640 10 monclient: build_initial_monmap 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.111+0000 7f50f95d8640 10 monclient: monmap: 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:08.113043+0000 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:08.113043+0000 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:08.114 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:08.115 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:08.115 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.111+0000 7f50f95d8640 1 Processor -- start 2024-09-16T04:12:08.115 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.111+0000 7f50f95d8640 1 -- start start 2024-09-16T04:12:08.116 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.111+0000 7f50f95d8640 10 monclient: init 2024-09-16T04:12:08.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:08.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:08.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient(hunting): picked mon.noname-b con 0x7f50f4136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient(hunting): picked mon.noname-a con 0x7f50f4139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient(hunting): picked mon.noname-c con 0x7f50f413d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a1340 con 0x7f50f4139b10 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f4137650 con 0x7f50f413d620 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a0aa0 con 0x7f50f4136b60 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 10 monclient(hunting): authenticate will time out at 1103.618420s 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f37fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f50f413d620 0x7f50f413fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52832/0 (socket says 172.21.15.171:52832) 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f37fe640 1 -- 172.21.15.171:0/1453463059 learned_addr learned my addr 172.21.15.171:0/1453463059 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f2ffd640 1 -- 172.21.15.171:0/1453463059 >> v1:172.21.15.148:6790/0 conn(0x7f50f4136b60 legacy=0x7f50f4136f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 965556811 0 0) 0x7f50f4137650 con 0x7f50f413d620 2024-09-16T04:12:08.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient(hunting): my global_id is 6217 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50e0003520 con 0x7f50f413d620 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2302248021 0 0) 0x7f50e0003520 con 0x7f50f413d620 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 >> v1:172.21.15.148:6790/0 conn(0x7f50f4136b60 legacy=0x7f50f4136f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 >> v1:172.21.15.148:6789/0 conn(0x7f50f4139b10 legacy=0x7f50f413bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 monclient: found mon.noname-c 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f40a0550 con 0x7f50f413d620 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 5 monclient: authenticate success, global_id 6217 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50ec001990 con 0x7f50f413d620 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f95d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.112+0000 7f50f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 10 monclient: dump: 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.126 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50ec001cf0 con 0x7f50f413d620 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 1 -- 172.21.15.171:0/1453463059 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50ec004c20 con 0x7f50f413d620 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 4 set_mon_vals no callback set 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:08.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f1ffb640 10 monclient: dump: 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 10 monclient: shutdown 2024-09-16T04:12:08.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 1 -- 172.21.15.171:0/1453463059 >> v1:172.21.15.171:6789/0 conn(0x7f50f413d620 legacy=0x7f50f413fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 1 -- 172.21.15.171:0/1453463059 shutdown_connections 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 1 -- 172.21.15.171:0/1453463059 >> 172.21.15.171:0/1453463059 conn(0x7f50f40a98b0 msgr2=0x7f50f40a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 1 -- 172.21.15.171:0/1453463059 shutdown_connections 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.113+0000 7f50f95d8640 1 -- 172.21.15.171:0/1453463059 wait complete. 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.114+0000 7f50f95d8640 10 monclient: build_initial_monmap 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.114+0000 7f50f95d8640 10 monclient: monmap: 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.129 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.114+0000 7f50f95d8640 1 Processor -- start 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.114+0000 7f50f95d8640 1 -- start start 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.114+0000 7f50f95d8640 10 monclient: init 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient(hunting): picked mon.noname-a con 0x7f50f4136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient(hunting): picked mon.noname-b con 0x7f50f4139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient(hunting): picked mon.noname-c con 0x7f50f413d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f409b4a0 con 0x7f50f4136b60 2024-09-16T04:12:08.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a2a90 con 0x7f50f4139b10 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a0240 con 0x7f50f413d620 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient(hunting): authenticate will time out at 1103.621158s 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f37fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f50f413d620 legacy=0x7f50f41d37f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f27fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f50f4139b10 0x7f50f413a870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52848/0 (socket says 172.21.15.171:52848) 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f27fc640 1 -- 172.21.15.171:0/1037369776 learned_addr learned my addr 172.21.15.171:0/1037369776 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797306545 0 0) 0x7f50f40a2a90 con 0x7f50f4139b10 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): my global_id is 6220 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d4003520 con 0x7f50f4139b10 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319021301 0 0) 0x7f50e40039f0 con 0x7f50f4136b60 2024-09-16T04:12:08.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): my global_id is 64653 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50f40a2a90 con 0x7f50f4136b60 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1422742199 0 0) 0x7f50d4003520 con 0x7f50f4139b10 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50e40039f0 con 0x7f50f4139b10 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50dc001930 con 0x7f50f4139b10 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:08.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): dump: 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:08.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1604493264 0 0) 0x7f50f40a2a90 con 0x7f50f4136b60 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50d4003520 con 0x7f50f4136b60 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50e40042c0 con 0x7f50f4136b60 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient(hunting): dump: 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:08.134 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2936182776 0 0) 0x7f50e40039f0 con 0x7f50f4139b10 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 >> v1:172.21.15.148:6790/0 conn(0x7f50f413d620 legacy=0x7f50f41d37f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 >> v1:172.21.15.148:6789/0 conn(0x7f50f4136b60 legacy=0x7f50f413a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 monclient: found mon.b 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f409b080 con 0x7f50f4139b10 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 5 monclient: authenticate success, global_id 6220 2024-09-16T04:12:08.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient: _renew_subs 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50f409f830 con 0x7f50f4139b10 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient: _renew_subs 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50f413af80 con 0x7f50f4139b10 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50dc001d00 con 0x7f50f4139b10 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.115+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50dc003a60 con 0x7f50f4139b10 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50f17fa640 4 set_mon_vals no callback set 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50eb7fe640 10 monclient: dump: 2024-09-16T04:12:08.136 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f50dc003c00 con 0x7f50f4139b10 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50f95d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50f95d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50f95d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.116+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d8005080 con 0x7f50f4139b10 2024-09-16T04:12:08.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.117+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(227..227 src has 1..227) ==== 11615+0+0 (unknown 2094374206 0 0) 0x7f50dc048c50 con 0x7f50f4139b10 2024-09-16T04:12:08.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.117+0000 7f50f2ffd640 10 monclient: get_auth_request con 0x7f50d4038ea0 auth_method 0 2024-09-16T04:12:08.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.121+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f50dc015cf0 con 0x7f50f4139b10 2024-09-16T04:12:08.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.121+0000 7f50eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:08.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.121+0000 7f50eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.290+0000 7f50f95d8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T04:12:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.290+0000 7f50f95d8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T04:12:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.290+0000 7f50f95d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:08.293 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.290+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) -- 0x7f50d8005370 con 0x7f50f4139b10 2024-09-16T04:12:08.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.919+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v228) ==== 120+0+0 (unknown 1397809998 0 0) 0x7f50dc0155f0 con 0x7f50f4139b10 2024-09-16T04:12:08.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.919+0000 7f50eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T04:12:08.922 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:08.919+0000 7f50eb7fe640 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-16T04:12:09.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.023+0000 7f50f95d8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T04:12:09.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.023+0000 7f50f95d8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T04:12:09.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.023+0000 7f50f95d8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.026 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.023+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) -- 0x7f50d8002210 con 0x7f50f4139b10 2024-09-16T04:12:09.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.024+0000 7f50eb7fe640 1 -- 172.21.15.171:0/1037369776 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v228) ==== 120+0+0 (unknown 710364041 0 0) 0x7f50dc019940 con 0x7f50f4139b10 2024-09-16T04:12:09.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.024+0000 7f50eb7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-16T04:12:09.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.024+0000 7f50eb7fe640 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-16T04:12:09.027 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h13' does not appear in the crush map 2024-09-16T04:12:09.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 >> v1:172.21.15.171:6812/7327273 conn(0x7f50d4038ea0 legacy=0x7f50d403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:09.029 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 10 monclient: shutdown 2024-09-16T04:12:09.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:09.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 >> v1:172.21.15.171:6789/0 conn(0x7f50f4139b10 legacy=0x7f50f413a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:09.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 shutdown_connections 2024-09-16T04:12:09.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 >> 172.21.15.171:0/1037369776 conn(0x7f50f40a98b0 msgr2=0x7f50f40a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:09.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 shutdown_connections 2024-09-16T04:12:09.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.026+0000 7f50f95d8640 1 -- 172.21.15.171:0/1037369776 wait complete. 2024-09-16T04:12:09.038 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:09.039 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h14 2024-09-16T04:12:09.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.114+0000 7efe41d11640 10 monclient: get_monmap_and_config 2024-09-16T04:12:09.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.114+0000 7efe41d11640 10 monclient: build_initial_monmap 2024-09-16T04:12:09.117 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.114+0000 7efe41d11640 10 monclient: monmap: 2024-09-16T04:12:09.117 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:09.116545+0000 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:09.116545+0000 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:09.118 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:09.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 1 Processor -- start 2024-09-16T04:12:09.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 1 -- start start 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 10 monclient: init 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 10 monclient(hunting): picked mon.noname-c con 0x7efe3c09f640 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.115+0000 7efe41d11640 10 monclient(hunting): picked mon.noname-b con 0x7efe3c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 10 monclient(hunting): picked mon.noname-a con 0x7efe3c09e1d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c0a1260 con 0x7efe3c09e1d0 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c09d0c0 con 0x7efe3c09f640 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c09d3a0 con 0x7efe3c0a0130 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 10 monclient(hunting): authenticate will time out at 1104.622051s 2024-09-16T04:12:09.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7efe3c09f640 0x7efe3c09fa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52850/0 (socket says 172.21.15.171:52850) 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3affd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7efe3c0a0130 legacy=0x7efe3c138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3b7fe640 1 -- 172.21.15.171:0/3429089338 learned_addr learned my addr 172.21.15.171:0/3429089338 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252474065 0 0) 0x7efe3c09d0c0 con 0x7efe3c09f640 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient(hunting): my global_id is 6223 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe24003520 con 0x7efe3c09f640 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2192962807 0 0) 0x7efe24003520 con 0x7efe3c09f640 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 >> v1:172.21.15.148:6790/0 conn(0x7efe3c0a0130 legacy=0x7efe3c138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:09.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 >> v1:172.21.15.148:6789/0 conn(0x7efe3c09e1d0 legacy=0x7efe3c13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c09d680 con 0x7efe3c09f640 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe2c001990 con 0x7efe3c09f640 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:09.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: dump: 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 5 monclient: authenticate success, global_id 6223 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe2c001c80 con 0x7efe3c09f640 2024-09-16T04:12:09.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 1 -- 172.21.15.171:0/3429089338 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe2c002090 con 0x7efe3c09f640 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe41d11640 4 set_mon_vals no callback set 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.116+0000 7efe3a7fc640 10 monclient: dump: 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:09.130 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 10 monclient: shutdown 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 1 -- 172.21.15.171:0/3429089338 >> v1:172.21.15.171:6789/0 conn(0x7efe3c09f640 legacy=0x7efe3c09fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 1 -- 172.21.15.171:0/3429089338 shutdown_connections 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 1 -- 172.21.15.171:0/3429089338 >> 172.21.15.171:0/3429089338 conn(0x7efe3c0a52a0 msgr2=0x7efe3c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 1 -- 172.21.15.171:0/3429089338 shutdown_connections 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 1 -- 172.21.15.171:0/3429089338 wait complete. 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 10 monclient: build_initial_monmap 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.117+0000 7efe41d11640 10 monclient: monmap: 2024-09-16T04:12:09.131 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 1 Processor -- start 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 1 -- start start 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient: init 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:09.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient(hunting): picked mon.noname-b con 0x7efe3c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient(hunting): picked mon.noname-a con 0x7efe3c09f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient(hunting): picked mon.noname-c con 0x7efe3c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c1c5660 con 0x7efe3c09f640 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c1a9df0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c1ac8b0 con 0x7efe3c0a0130 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe41d11640 10 monclient(hunting): authenticate will time out at 1104.624830s 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe3b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7efe3c09e1d0 0x7efe3c1c4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52862/0 (socket says 172.21.15.171:52862) 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe3b7fe640 1 -- 172.21.15.171:0/1735654761 learned_addr learned my addr 172.21.15.171:0/1735654761 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.118+0000 7efe3bfff640 1 -- 172.21.15.171:0/1735654761 >> v1:172.21.15.148:6790/0 conn(0x7efe3c0a0130 legacy=0x7efe3c1d39b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:09.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263992041 0 0) 0x7efe3c1a9df0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): my global_id is 6226 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe28003520 con 0x7efe3c09e1d0 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3898048900 0 0) 0x7efe28003520 con 0x7efe3c09e1d0 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe3c1a9df0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081038563 0 0) 0x7efe3c1c5660 con 0x7efe3c09f640 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): my global_id is 64659 2024-09-16T04:12:09.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe28003520 con 0x7efe3c09f640 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe2c001950 con 0x7efe3c09e1d0 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): dump: 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:09.135 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3473164940 0 0) 0x7efe3c1a9df0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 >> v1:172.21.15.148:6790/0 conn(0x7efe3c0a0130 legacy=0x7efe3c1d39b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 >> v1:172.21.15.148:6789/0 conn(0x7efe3c09f640 legacy=0x7efe3c1d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 monclient: found mon.b 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c072150 con 0x7efe3c09e1d0 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: _finish_auth 0 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:09.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 5 monclient: authenticate success, global_id 6226 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 10 monclient: _renew_subs 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efe3c1ac5d0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 10 monclient: _renew_subs 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efe3c1c79c0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe2c001c80 con 0x7efe3c09e1d0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe2c0051e0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:09.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe39ffb640 4 set_mon_vals no callback set 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.119+0000 7efe23fff640 10 monclient: dump: 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7efe2c005400 con 0x7efe3c09e1d0 2024-09-16T04:12:09.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(228..228 src has 1..228) ==== 11402+0+0 (unknown 2036082463 0 0) 0x7efe2c049130 con 0x7efe3c09e1d0 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe41d11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe41d11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe41d11640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe1c005080 con 0x7efe3c09e1d0 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.120+0000 7efe3affd640 10 monclient: get_auth_request con 0x7efe28038f70 auth_method 0 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.125+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7efe2c0161c0 con 0x7efe3c09e1d0 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.125+0000 7efe23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:09.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.125+0000 7efe23fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:09.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.294+0000 7efe41d11640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T04:12:09.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.294+0000 7efe41d11640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T04:12:09.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.294+0000 7efe41d11640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:09.297 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.294+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) -- 0x7efe1c005370 con 0x7efe3c09e1d0 2024-09-16T04:12:09.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.924+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v229) ==== 120+0+0 (unknown 1605276742 0 0) 0x7efe2c01e670 con 0x7efe3c09e1d0 2024-09-16T04:12:09.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.924+0000 7efe23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T04:12:09.927 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:09.924+0000 7efe23fff640 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-16T04:12:10.030 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.027+0000 7efe41d11640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T04:12:10.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.027+0000 7efe41d11640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T04:12:10.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.027+0000 7efe41d11640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.027+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) -- 0x7efe1c002210 con 0x7efe3c09e1d0 2024-09-16T04:12:10.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.029+0000 7efe23fff640 1 -- 172.21.15.171:0/1735654761 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v229) ==== 120+0+0 (unknown 105924795 0 0) 0x7efe2c0460e0 con 0x7efe3c09e1d0 2024-09-16T04:12:10.031 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.029+0000 7efe23fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-16T04:12:10.032 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.029+0000 7efe23fff640 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-16T04:12:10.032 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h14' does not appear in the crush map 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 >> v1:172.21.15.171:6812/7327273 conn(0x7efe28038f70 legacy=0x7efe2803b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 10 monclient: shutdown 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 >> v1:172.21.15.171:6789/0 conn(0x7efe3c09e1d0 legacy=0x7efe3c1c4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 shutdown_connections 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 >> 172.21.15.171:0/1735654761 conn(0x7efe3c0a52a0 msgr2=0x7efe3c0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 shutdown_connections 2024-09-16T04:12:10.034 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.031+0000 7efe41d11640 1 -- 172.21.15.171:0/1735654761 wait complete. 2024-09-16T04:12:10.043 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:10.043 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h15 2024-09-16T04:12:10.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.117+0000 7f131bd31640 10 monclient: get_monmap_and_config 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.117+0000 7f131bd31640 10 monclient: build_initial_monmap 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.117+0000 7f131bd31640 10 monclient: monmap: 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:10.119456+0000 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:10.119456+0000 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:10.121 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:10.122 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:10.122 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:10.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 1 Processor -- start 2024-09-16T04:12:10.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 1 -- start start 2024-09-16T04:12:10.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient: init 2024-09-16T04:12:10.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:10.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient(hunting): picked mon.noname-b con 0x7f1314132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient(hunting): picked mon.noname-a con 0x7f131409ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient(hunting): picked mon.noname-c con 0x7f1314132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13141332b0 con 0x7f131409ec20 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1314134310 con 0x7f1314132e60 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131409d980 con 0x7f1314132a80 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.118+0000 7f131bd31640 10 monclient(hunting): authenticate will time out at 1105.624964s 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131a2ce640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1314132e60 0x7f131413c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52878/0 (socket says 172.21.15.171:52878) 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1319acd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f1314132a80 legacy=0x7f13141334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131a2ce640 1 -- 172.21.15.171:0/2958798152 learned_addr learned my addr 172.21.15.171:0/2958798152 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033762342 0 0) 0x7f1314134310 con 0x7f1314132e60 2024-09-16T04:12:10.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): my global_id is 6229 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1300003520 con 0x7f1314132e60 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609158444 0 0) 0x7f13141332b0 con 0x7f131409ec20 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient(hunting): my global_id is 64662 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1314134310 con 0x7f131409ec20 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1147122546 0 0) 0x7f1300003520 con 0x7f1314132e60 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 >> v1:172.21.15.148:6790/0 conn(0x7f1314132a80 legacy=0x7f13141334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:10.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 >> v1:172.21.15.148:6789/0 conn(0x7f131409ec20 legacy=0x7f1314133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: _finish_hunting 0 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 monclient: found mon.noname-c 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131409dc60 con 0x7f1314132e60 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: _finish_auth 0 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131bd31640 5 monclient: authenticate success, global_id 6229 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131bd31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f130c001990 con 0x7f1314132e60 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:10.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: dump: 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131bd31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f130c001c40 con 0x7f1314132e60 2024-09-16T04:12:10.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 1 -- 172.21.15.171:0/2958798152 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f130c002010 con 0x7f1314132e60 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131bd31640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f131bd31640 4 set_mon_vals no callback set 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.119+0000 7f1318acb640 10 monclient: dump: 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:10.133 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 10 monclient: shutdown 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 1 -- 172.21.15.171:0/2958798152 >> v1:172.21.15.171:6789/0 conn(0x7f1314132e60 legacy=0x7f131413c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 1 -- 172.21.15.171:0/2958798152 shutdown_connections 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 1 -- 172.21.15.171:0/2958798152 >> 172.21.15.171:0/2958798152 conn(0x7f13140a52a0 msgr2=0x7f13140a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 1 -- 172.21.15.171:0/2958798152 shutdown_connections 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 1 -- 172.21.15.171:0/2958798152 wait complete. 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 10 monclient: build_initial_monmap 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.120+0000 7f131bd31640 10 monclient: monmap: 2024-09-16T04:12:10.134 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 1 Processor -- start 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 1 -- start start 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient: init 2024-09-16T04:12:10.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient(hunting): picked mon.noname-b con 0x7f131409ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient(hunting): picked mon.noname-c con 0x7f1314132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient(hunting): picked mon.noname-a con 0x7f1314132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13140a3460 con 0x7f1314132e60 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1314121f20 con 0x7f131409ec20 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1314122200 con 0x7f1314132a80 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f131bd31640 10 monclient(hunting): authenticate will time out at 1105.627618s 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f1319acd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f131409ec20 0x7f131411f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52882/0 (socket says 172.21.15.171:52882) 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f1319acd640 1 -- 172.21.15.171:0/1471483575 learned_addr learned my addr 172.21.15.171:0/1471483575 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f13192cc640 1 -- 172.21.15.171:0/1471483575 >> v1:172.21.15.148:6790/0 conn(0x7f1314132a80 legacy=0x7f131411fa00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:10.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194177283 0 0) 0x7f1314121f20 con 0x7f131409ec20 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f130a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f130a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f130a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f130a7fc640 10 monclient(hunting): my global_id is 6232 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.121+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12f8003520 con 0x7f131409ec20 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3654663354 0 0) 0x7f12f8003520 con 0x7f131409ec20 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1314121f20 con 0x7f131409ec20 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13100041d0 con 0x7f131409ec20 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:10.137 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890611418 0 0) 0x7f131409dc60 con 0x7f1314132e60 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:10.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient(hunting): my global_id is 64665 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12f8003520 con 0x7f1314132e60 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2018639864 0 0) 0x7f1314121f20 con 0x7f131409ec20 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 >> v1:172.21.15.148:6790/0 conn(0x7f1314132a80 legacy=0x7f131411fa00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 >> v1:172.21.15.148:6789/0 conn(0x7f1314132e60 legacy=0x7f13141d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 monclient: found mon.b 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1314120110 con 0x7f131409ec20 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:10.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 5 monclient: authenticate success, global_id 6232 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 10 monclient: _renew_subs 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1314121c40 con 0x7f131409ec20 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 10 monclient: _renew_subs 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1314122be0 con 0x7f131409ec20 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1310004430 con 0x7f131409ec20 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1310004780 con 0x7f131409ec20 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130bfff640 4 set_mon_vals no callback set 2024-09-16T04:12:10.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.122+0000 7f130a7fc640 10 monclient: dump: 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1310004920 con 0x7f131409ec20 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f131bd31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:10.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f131bd31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f131bd31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12fc005080 con 0x7f131409ec20 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(229..229 src has 1..229) ==== 11189+0+0 (unknown 2401966766 0 0) 0x7f1310048b20 con 0x7f131409ec20 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.123+0000 7f13192cc640 10 monclient: get_auth_request con 0x7f12f8039010 auth_method 0 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.127+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f1310019880 con 0x7f131409ec20 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.127+0000 7f130a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:10.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.127+0000 7f130a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:10.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.296+0000 7f131bd31640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T04:12:10.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.296+0000 7f131bd31640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T04:12:10.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.296+0000 7f131bd31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:10.299 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.296+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) -- 0x7f12fc005370 con 0x7f131409ec20 2024-09-16T04:12:10.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.927+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v230) ==== 120+0+0 (unknown 3006135946 0 0) 0x7f131001e140 con 0x7f131409ec20 2024-09-16T04:12:10.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.927+0000 7f130a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T04:12:10.930 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:10.927+0000 7f130a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-16T04:12:11.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.032+0000 7f131bd31640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T04:12:11.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.032+0000 7f131bd31640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T04:12:11.035 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.032+0000 7f131bd31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.032+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) -- 0x7f12fc002210 con 0x7f131409ec20 2024-09-16T04:12:11.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.033+0000 7f130a7fc640 1 -- 172.21.15.171:0/1471483575 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v230) ==== 120+0+0 (unknown 2820579632 0 0) 0x7f13100138b0 con 0x7f131409ec20 2024-09-16T04:12:11.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.033+0000 7f130a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-16T04:12:11.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.033+0000 7f130a7fc640 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-16T04:12:11.036 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h15' does not appear in the crush map 2024-09-16T04:12:11.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 >> v1:172.21.15.171:6812/7327273 conn(0x7f12f8039010 legacy=0x7f12f803b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 10 monclient: shutdown 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 >> v1:172.21.15.171:6789/0 conn(0x7f131409ec20 legacy=0x7f131411f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 shutdown_connections 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 >> 172.21.15.171:0/1471483575 conn(0x7f13140a52a0 msgr2=0x7f1314131230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.035+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 shutdown_connections 2024-09-16T04:12:11.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.036+0000 7f131bd31640 1 -- 172.21.15.171:0/1471483575 wait complete. 2024-09-16T04:12:11.048 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:11.048 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h16 2024-09-16T04:12:11.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.124+0000 7f3c68952640 10 monclient: get_monmap_and_config 2024-09-16T04:12:11.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.124+0000 7f3c68952640 10 monclient: build_initial_monmap 2024-09-16T04:12:11.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.124+0000 7f3c68952640 10 monclient: monmap: 2024-09-16T04:12:11.127 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:11.127 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:11.127 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:11.126396+0000 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:11.126396+0000 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:11.128 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:11.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 1 Processor -- start 2024-09-16T04:12:11.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 1 -- start start 2024-09-16T04:12:11.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient: init 2024-09-16T04:12:11.134 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6409ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient(hunting): picked mon.noname-c con 0x7f3c640a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6409e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6409fbf0 con 0x7f3c6409ba30 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c640a0e10 con 0x7f3c640a0750 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6409f640 con 0x7f3c6409e570 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c68952640 10 monclient(hunting): authenticate will time out at 1106.631867s 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c61d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c640a0750 0x7f3c641399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52898/0 (socket says 172.21.15.171:52898) 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c62d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3c6409e570 legacy=0x7f3c6413d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:11.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.125+0000 7f3c61d9b640 1 -- 172.21.15.171:0/2617563741 learned_addr learned my addr 172.21.15.171:0/2617563741 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912460986 0 0) 0x7f3c640a0e10 con 0x7f3c640a0750 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient(hunting): _negotiate 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient(hunting): my global_id is 6235 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c50003520 con 0x7f3c640a0750 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 957248949 0 0) 0x7f3c50003520 con 0x7f3c640a0750 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 >> v1:172.21.15.148:6790/0 conn(0x7f3c6409e570 legacy=0x7f3c6413d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 >> v1:172.21.15.148:6789/0 conn(0x7f3c6409ba30 legacy=0x7f3c640a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: _finish_hunting 0 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 monclient: found mon.noname-c 2024-09-16T04:12:11.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6409f810 con 0x7f3c640a0750 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: _finish_auth 0 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: _check_auth_tickets 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 5 monclient: authenticate success, global_id 6235 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c54001610 con 0x7f3c640a0750 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: dump: 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:11.137 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c54001970 con 0x7f3c640a0750 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 1 -- 172.21.15.171:0/2617563741 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c540048a0 con 0x7f3c640a0750 2024-09-16T04:12:11.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 4 set_mon_vals no callback set 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c6159a640 10 monclient: dump: 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:11.139 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 10 monclient: shutdown 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.126+0000 7f3c68952640 1 -- 172.21.15.171:0/2617563741 >> v1:172.21.15.171:6789/0 conn(0x7f3c640a0750 legacy=0x7f3c641399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.127+0000 7f3c68952640 1 -- 172.21.15.171:0/2617563741 shutdown_connections 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.127+0000 7f3c68952640 1 -- 172.21.15.171:0/2617563741 >> 172.21.15.171:0/2617563741 conn(0x7f3c640a52a0 msgr2=0x7f3c640a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.127+0000 7f3c68952640 1 -- 172.21.15.171:0/2617563741 shutdown_connections 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.127+0000 7f3c68952640 1 -- 172.21.15.171:0/2617563741 wait complete. 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.127+0000 7f3c68952640 10 monclient: build_initial_monmap 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.127+0000 7f3c68952640 10 monclient: monmap: 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:11.140 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 1 Processor -- start 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 1 -- start start 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient: init 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6409ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:11.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6409e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient(hunting): picked mon.noname-b con 0x7f3c640a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c64071800 con 0x7f3c6409ba30 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c641cc8d0 con 0x7f3c640a0750 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6406f810 con 0x7f3c6409e570 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c68952640 10 monclient(hunting): authenticate will time out at 1106.634647s 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c61d9b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3c6409e570 legacy=0x7f3c641cc1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c62d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3c640a0750 0x7f3c641d3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52904/0 (socket says 172.21.15.171:52904) 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c62d9d640 1 -- 172.21.15.171:0/801292426 learned_addr learned my addr 172.21.15.171:0/801292426 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3425797857 0 0) 0x7f3c641cc8d0 con 0x7f3c640a0750 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c5affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:11.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c5affd640 10 monclient(hunting): my global_id is 6238 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.128+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c44003520 con 0x7f3c640a0750 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063870927 0 0) 0x7f3c64071800 con 0x7f3c6409ba30 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): my global_id is 64671 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c641cc8d0 con 0x7f3c6409ba30 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1484834090 0 0) 0x7f3c44003520 con 0x7f3c640a0750 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3c64071800 con 0x7f3c640a0750 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c5c0043f0 con 0x7f3c640a0750 2024-09-16T04:12:11.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): dump: 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:11.144 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4097912091 0 0) 0x7f3c64071800 con 0x7f3c640a0750 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 >> v1:172.21.15.148:6790/0 conn(0x7f3c6409e570 legacy=0x7f3c641cc1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 >> v1:172.21.15.148:6789/0 conn(0x7f3c6409ba30 legacy=0x7f3c641cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 monclient: found mon.b 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c64071520 con 0x7f3c640a0750 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: _finish_auth 0 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:11.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 5 monclient: authenticate success, global_id 6238 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 10 monclient: _renew_subs 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3c64071ae0 con 0x7f3c640a0750 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 10 monclient: _renew_subs 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3c6413e530 con 0x7f3c640a0750 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c0046a0 con 0x7f3c640a0750 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3c5c004a70 con 0x7f3c640a0750 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c60d99640 4 set_mon_vals no callback set 2024-09-16T04:12:11.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.129+0000 7f3c5affd640 10 monclient: dump: 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3c5c004c90 con 0x7f3c640a0750 2024-09-16T04:12:11.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c68952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c68952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c68952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c48005080 con 0x7f3c640a0750 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(230..230 src has 1..230) ==== 10976+0+0 (unknown 3980205922 0 0) 0x7f3c5c048dd0 con 0x7f3c640a0750 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.130+0000 7f3c6259c640 10 monclient: get_auth_request con 0x7f3c44038f60 auth_method 0 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.134+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3c5c079d40 con 0x7f3c640a0750 2024-09-16T04:12:11.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.134+0000 7f3c5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:11.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.134+0000 7f3c5affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:11.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.305+0000 7f3c68952640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T04:12:11.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.305+0000 7f3c68952640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T04:12:11.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.305+0000 7f3c68952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:11.308 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.305+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) -- 0x7f3c48005370 con 0x7f3c640a0750 2024-09-16T04:12:11.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.945+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v231) ==== 120+0+0 (unknown 557574609 0 0) 0x7f3c5c019c00 con 0x7f3c640a0750 2024-09-16T04:12:11.948 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.945+0000 7f3c5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T04:12:11.949 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:11.945+0000 7f3c5affd640 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-16T04:12:12.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.050+0000 7f3c68952640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T04:12:12.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.050+0000 7f3c68952640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T04:12:12.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.050+0000 7f3c68952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.050+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) -- 0x7f3c48001090 con 0x7f3c640a0750 2024-09-16T04:12:12.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.052+0000 7f3c5affd640 1 -- 172.21.15.171:0/801292426 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v231) ==== 120+0+0 (unknown 1808559892 0 0) 0x7f3c5c013c30 con 0x7f3c640a0750 2024-09-16T04:12:12.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.052+0000 7f3c5affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-16T04:12:12.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.052+0000 7f3c5affd640 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-16T04:12:12.055 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h16' does not appear in the crush map 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 >> v1:172.21.15.171:6812/7327273 conn(0x7f3c44038f60 legacy=0x7f3c4403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 10 monclient: shutdown 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 >> v1:172.21.15.171:6789/0 conn(0x7f3c640a0750 legacy=0x7f3c641d3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 shutdown_connections 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 >> 172.21.15.171:0/801292426 conn(0x7f3c640a52a0 msgr2=0x7f3c641343d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 shutdown_connections 2024-09-16T04:12:12.057 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.054+0000 7f3c68952640 1 -- 172.21.15.171:0/801292426 wait complete. 2024-09-16T04:12:12.066 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:12.066 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h17 2024-09-16T04:12:12.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.143+0000 7f8df585e640 10 monclient: get_monmap_and_config 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.143+0000 7f8df585e640 10 monclient: build_initial_monmap 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.143+0000 7f8df585e640 10 monclient: monmap: 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:12.145486+0000 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:12.145486+0000 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:12.147 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:12.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 1 Processor -- start 2024-09-16T04:12:12.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 1 -- start start 2024-09-16T04:12:12.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient: init 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient(hunting): picked mon.noname-a con 0x7f8df00a2ae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient(hunting): picked mon.noname-c con 0x7f8df00a3f10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient(hunting): picked mon.noname-b con 0x7f8df00a4320 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a0b50 con 0x7f8df00a2ae0 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a0dc0 con 0x7f8df00a3f10 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a0550 con 0x7f8df00a4320 2024-09-16T04:12:12.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8df585e640 10 monclient(hunting): authenticate will time out at 1107.650892s 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8dee7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8df00a3f10 0x7f8df013bba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52916/0 (socket says 172.21.15.171:52916) 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8dee7fc640 1 -- 172.21.15.171:0/3935837291 learned_addr learned my addr 172.21.15.171:0/3935837291 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.144+0000 7f8def7fe640 1 -- 172.21.15.171:0/3935837291 >> v1:172.21.15.148:6790/0 conn(0x7f8df00a4320 legacy=0x7f8df013f2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857396686 0 0) 0x7f8df00a0dc0 con 0x7f8df00a3f10 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): my global_id is 6241 2024-09-16T04:12:12.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd8003520 con 0x7f8df00a3f10 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3830338515 0 0) 0x7f8df00a0b50 con 0x7f8df00a2ae0 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient(hunting): my global_id is 64674 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df00a0dc0 con 0x7f8df00a2ae0 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 70806012 0 0) 0x7f8dd8003520 con 0x7f8df00a3f10 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 >> v1:172.21.15.148:6790/0 conn(0x7f8df00a4320 legacy=0x7f8df013f2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 >> v1:172.21.15.148:6789/0 conn(0x7f8df00a2ae0 legacy=0x7f8df00a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: _finish_hunting 0 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 monclient: found mon.noname-c 2024-09-16T04:12:12.157 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df00a64d0 con 0x7f8df00a3f10 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8df585e640 5 monclient: authenticate success, global_id 6241 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8de4001610 con 0x7f8df00a3f10 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: dump: 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.158 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8df585e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de4001930 con 0x7f8df00a3f10 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8df585e640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8df585e640 4 set_mon_vals no callback set 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 1 -- 172.21.15.171:0/3935837291 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8de4001d00 con 0x7f8df00a3f10 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:12.159 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.145+0000 7f8dedffb640 10 monclient: dump: 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 10 monclient: shutdown 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:12.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 1 -- 172.21.15.171:0/3935837291 >> v1:172.21.15.171:6789/0 conn(0x7f8df00a3f10 legacy=0x7f8df013bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 1 -- 172.21.15.171:0/3935837291 shutdown_connections 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 1 -- 172.21.15.171:0/3935837291 >> 172.21.15.171:0/3935837291 conn(0x7f8df00a9930 msgr2=0x7f8df00a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 1 -- 172.21.15.171:0/3935837291 shutdown_connections 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 1 -- 172.21.15.171:0/3935837291 wait complete. 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 10 monclient: build_initial_monmap 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.146+0000 7f8df585e640 10 monclient: monmap: 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:12.161 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:12.162 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:12.162 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:12.162 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 1 Processor -- start 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 1 -- start start 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient: init 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient(hunting): picked mon.noname-c con 0x7f8df00a2ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient(hunting): picked mon.noname-b con 0x7f8df00a3f10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient(hunting): picked mon.noname-a con 0x7f8df00a4320 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:12.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df013ecb0 con 0x7f8df00a4320 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a81f0 con 0x7f8df00a3f10 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df013e9d0 con 0x7f8df00a2ae0 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8df585e640 10 monclient(hunting): authenticate will time out at 1107.653681s 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8deeffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f8df00a2ae0 legacy=0x7f8df013f0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dee7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8df00a3f10 0x7f8df01cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52918/0 (socket says 172.21.15.171:52918) 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dee7fc640 1 -- 172.21.15.171:0/1803134258 learned_addr learned my addr 172.21.15.171:0/1803134258 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 327311605 0 0) 0x7f8df00a81f0 con 0x7f8df00a3f10 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dd7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dd7fff640 10 monclient(hunting): my global_id is 6244 2024-09-16T04:12:12.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.147+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc003520 con 0x7f8df00a3f10 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526368398 0 0) 0x7f8df013ecb0 con 0x7f8df00a4320 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): my global_id is 64677 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df00a81f0 con 0x7f8df00a4320 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1437376698 0 0) 0x7f8ddc003520 con 0x7f8df00a3f10 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df013ecb0 con 0x7f8df00a3f10 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8de40015d0 con 0x7f8df00a3f10 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): dump: 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:12.165 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 966298176 0 0) 0x7f8df013ecb0 con 0x7f8df00a3f10 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 >> v1:172.21.15.148:6790/0 conn(0x7f8df00a2ae0 legacy=0x7f8df013f0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 >> v1:172.21.15.148:6789/0 conn(0x7f8df00a4320 legacy=0x7f8df01d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:12.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 monclient: found mon.b 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df013d2a0 con 0x7f8df00a3f10 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: _finish_auth 0 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 5 monclient: authenticate success, global_id 6244 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 10 monclient: _renew_subs 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8df013d740 con 0x7f8df00a3f10 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 10 monclient: _renew_subs 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8df01d4b40 con 0x7f8df00a3f10 2024-09-16T04:12:12.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de4001930 con 0x7f8df00a3f10 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8de4004e60 con 0x7f8df00a3f10 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8ded7fa640 4 set_mon_vals no callback set 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.148+0000 7f8dd7fff640 10 monclient: dump: 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:12.168 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8de4005000 con 0x7f8df00a3f10 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8df585e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8df585e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8df585e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd0005080 con 0x7f8df00a3f10 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(231..231 src has 1..231) ==== 10763+0+0 (unknown 2940716985 0 0) 0x7f8de40489c0 con 0x7f8df00a3f10 2024-09-16T04:12:12.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.149+0000 7f8deeffd640 10 monclient: get_auth_request con 0x7f8ddc038f60 auth_method 0 2024-09-16T04:12:12.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.153+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8de4015dd0 con 0x7f8df00a3f10 2024-09-16T04:12:12.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.153+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:12.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.153+0000 7f8dd7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:12.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.325+0000 7f8df585e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T04:12:12.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.325+0000 7f8df585e640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T04:12:12.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.325+0000 7f8df585e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:12.329 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.325+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) -- 0x7f8dd0005370 con 0x7f8df00a3f10 2024-09-16T04:12:12.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.953+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v232) ==== 120+0+0 (unknown 3808415319 0 0) 0x7f8de401e220 con 0x7f8df00a3f10 2024-09-16T04:12:12.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.953+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T04:12:12.956 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:12.953+0000 7f8dd7fff640 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-16T04:12:13.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.059+0000 7f8df585e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T04:12:13.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.059+0000 7f8df585e640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T04:12:13.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.059+0000 7f8df585e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.062 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.059+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) -- 0x7f8dd0002210 con 0x7f8df00a3f10 2024-09-16T04:12:13.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.060+0000 7f8dd7fff640 1 -- 172.21.15.171:0/1803134258 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v232) ==== 120+0+0 (unknown 2561933383 0 0) 0x7f8de4044c50 con 0x7f8df00a3f10 2024-09-16T04:12:13.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.060+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-16T04:12:13.063 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.060+0000 7f8dd7fff640 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-16T04:12:13.063 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h17' does not appear in the crush map 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 >> v1:172.21.15.171:6812/7327273 conn(0x7f8ddc038f60 legacy=0x7f8ddc03b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 10 monclient: shutdown 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 >> v1:172.21.15.171:6789/0 conn(0x7f8df00a3f10 legacy=0x7f8df01cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 shutdown_connections 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 >> 172.21.15.171:0/1803134258 conn(0x7f8df00a9930 msgr2=0x7f8df0139300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 shutdown_connections 2024-09-16T04:12:13.065 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.062+0000 7f8df585e640 1 -- 172.21.15.171:0/1803134258 wait complete. 2024-09-16T04:12:13.076 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:13.076 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h18 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.149+0000 7fc5e8386640 10 monclient: get_monmap_and_config 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.149+0000 7fc5e8386640 10 monclient: build_initial_monmap 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.149+0000 7fc5e8386640 10 monclient: monmap: 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:13.150970+0000 2024-09-16T04:12:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:13.150970+0000 2024-09-16T04:12:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:13.153 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.149+0000 7fc5e8386640 1 Processor -- start 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.149+0000 7fc5e8386640 1 -- start start 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.149+0000 7fc5e8386640 10 monclient: init 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e009ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e00a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e009e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e009fbf0 con 0x7fc5e009ba30 2024-09-16T04:12:13.162 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e00a0e10 con 0x7fc5e00a0750 2024-09-16T04:12:13.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e009f640 con 0x7fc5e009e570 2024-09-16T04:12:13.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:13.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e8386640 10 monclient(hunting): authenticate will time out at 1108.656540s 2024-09-16T04:12:13.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e6923640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc5e009e570 legacy=0x7fc5e013d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:13.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5921640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc5e00a0750 0x7fc5e01399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52924/0 (socket says 172.21.15.171:52924) 2024-09-16T04:12:13.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5921640 1 -- 172.21.15.171:0/2091557041 learned_addr learned my addr 172.21.15.171:0/2091557041 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:13.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523226327 0 0) 0x7fc5e00a0e10 con 0x7fc5e00a0750 2024-09-16T04:12:13.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5120640 10 monclient(hunting): _negotiate 2024-09-16T04:12:13.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5120640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:13.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5120640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:13.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5120640 10 monclient(hunting): my global_id is 6247 2024-09-16T04:12:13.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.150+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5cc003520 con 0x7fc5e00a0750 2024-09-16T04:12:13.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2490425991 0 0) 0x7fc5cc003520 con 0x7fc5e00a0750 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 >> v1:172.21.15.148:6790/0 conn(0x7fc5e009e570 legacy=0x7fc5e013d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 >> v1:172.21.15.148:6789/0 conn(0x7fc5e009ba30 legacy=0x7fc5e00a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: _finish_hunting 0 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 monclient: found mon.noname-c 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e009f810 con 0x7fc5e00a0750 2024-09-16T04:12:13.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: _finish_auth 0 2024-09-16T04:12:13.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: _check_auth_tickets 2024-09-16T04:12:13.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:13.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 5 monclient: authenticate success, global_id 6247 2024-09-16T04:12:13.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5dc001610 con 0x7fc5e00a0750 2024-09-16T04:12:13.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:13.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:13.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:13.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: dump: 2024-09-16T04:12:13.169 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:13.169 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:13.170 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:13.171 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:13.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5dc001970 con 0x7fc5e00a0750 2024-09-16T04:12:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 1 -- 172.21.15.171:0/2091557041 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5dc0048a0 con 0x7fc5e00a0750 2024-09-16T04:12:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 4 set_mon_vals no callback set 2024-09-16T04:12:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:13.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e5120640 10 monclient: dump: 2024-09-16T04:12:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.173 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:13.174 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:13.175 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:13.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 10 monclient: shutdown 2024-09-16T04:12:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 1 -- 172.21.15.171:0/2091557041 >> v1:172.21.15.171:6789/0 conn(0x7fc5e00a0750 legacy=0x7fc5e01399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 1 -- 172.21.15.171:0/2091557041 shutdown_connections 2024-09-16T04:12:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 1 -- 172.21.15.171:0/2091557041 >> 172.21.15.171:0/2091557041 conn(0x7fc5e00a52a0 msgr2=0x7fc5e00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:13.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 1 -- 172.21.15.171:0/2091557041 shutdown_connections 2024-09-16T04:12:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.151+0000 7fc5e8386640 1 -- 172.21.15.171:0/2091557041 wait complete. 2024-09-16T04:12:13.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.152+0000 7fc5e8386640 10 monclient: build_initial_monmap 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.152+0000 7fc5e8386640 10 monclient: monmap: 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:13.178 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.152+0000 7fc5e8386640 1 Processor -- start 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.152+0000 7fc5e8386640 1 -- start start 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.152+0000 7fc5e8386640 10 monclient: init 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e009ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e009e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e00a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e0131430 con 0x7fc5e009ba30 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e0131710 con 0x7fc5e00a0750 2024-09-16T04:12:13.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e01319f0 con 0x7fc5e009e570 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient(hunting): authenticate will time out at 1108.659146s 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e5921640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc5e009e570 legacy=0x7fc5e009c8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e6923640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc5e00a0750 0x7fc5e009cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52938/0 (socket says 172.21.15.171:52938) 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e6923640 1 -- 172.21.15.171:0/688189900 learned_addr learned my addr 172.21.15.171:0/688189900 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862576291 0 0) 0x7fc5e0131710 con 0x7fc5e00a0750 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): my global_id is 6250 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c4003520 con 0x7fc5e00a0750 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2981918443 0 0) 0x7fc5c4003520 con 0x7fc5e00a0750 2024-09-16T04:12:13.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5e0131710 con 0x7fc5e00a0750 2024-09-16T04:12:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5d80043f0 con 0x7fc5e00a0750 2024-09-16T04:12:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:13.181 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:13.182 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:13.183 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146847221 0 0) 0x7fc5e0131430 con 0x7fc5e009ba30 2024-09-16T04:12:13.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient(hunting): my global_id is 64683 2024-09-16T04:12:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c4003520 con 0x7fc5e009ba30 2024-09-16T04:12:13.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2495433074 0 0) 0x7fc5e0131710 con 0x7fc5e00a0750 2024-09-16T04:12:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 >> v1:172.21.15.148:6790/0 conn(0x7fc5e009e570 legacy=0x7fc5e009c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 >> v1:172.21.15.148:6789/0 conn(0x7fc5e009ba30 legacy=0x7fc5e009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 monclient: found mon.b 2024-09-16T04:12:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:13.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e009dd50 con 0x7fc5e00a0750 2024-09-16T04:12:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 5 monclient: authenticate success, global_id 6250 2024-09-16T04:12:13.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient: _renew_subs 2024-09-16T04:12:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e0132460 con 0x7fc5e00a0750 2024-09-16T04:12:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient: _renew_subs 2024-09-16T04:12:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5e01d4cf0 con 0x7fc5e00a0750 2024-09-16T04:12:13.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5d80046e0 con 0x7fc5e00a0750 2024-09-16T04:12:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:13.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5d8004ab0 con 0x7fc5e00a0750 2024-09-16T04:12:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:13.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5e491f640 4 set_mon_vals no callback set 2024-09-16T04:12:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.153+0000 7fc5d6ffd640 10 monclient: dump: 2024-09-16T04:12:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:13.190 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:13.191 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.154+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc5d8004d10 con 0x7fc5e00a0750 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.154+0000 7fc5e8386640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.154+0000 7fc5e8386640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.154+0000 7fc5e8386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.154+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c8005080 con 0x7fc5e00a0750 2024-09-16T04:12:13.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.154+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(232..232 src has 1..232) ==== 10550+0+0 (unknown 4287866861 0 0) 0x7fc5d8048c90 con 0x7fc5e00a0750 2024-09-16T04:12:13.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.155+0000 7fc5e6122640 10 monclient: get_auth_request con 0x7fc5c4038f70 auth_method 0 2024-09-16T04:12:13.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.159+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc5d8016020 con 0x7fc5e00a0750 2024-09-16T04:12:13.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.159+0000 7fc5d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:13.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.159+0000 7fc5d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:13.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.329+0000 7fc5e8386640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T04:12:13.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.329+0000 7fc5e8386640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T04:12:13.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.329+0000 7fc5e8386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:13.332 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.329+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) -- 0x7fc5c8005370 con 0x7fc5e00a0750 2024-09-16T04:12:13.963 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.961+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v233) ==== 120+0+0 (unknown 300242253 0 0) 0x7fc5d8015920 con 0x7fc5e00a0750 2024-09-16T04:12:13.964 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.961+0000 7fc5d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T04:12:13.964 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:13.961+0000 7fc5d6ffd640 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-16T04:12:14.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.066+0000 7fc5e8386640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T04:12:14.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.066+0000 7fc5e8386640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T04:12:14.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.066+0000 7fc5e8386640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.069 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.066+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) -- 0x7fc5c8001090 con 0x7fc5e00a0750 2024-09-16T04:12:14.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.067+0000 7fc5d6ffd640 1 -- 172.21.15.171:0/688189900 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v233) ==== 120+0+0 (unknown 1861545896 0 0) 0x7fc5d8019c70 con 0x7fc5e00a0750 2024-09-16T04:12:14.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.067+0000 7fc5d6ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-16T04:12:14.070 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.067+0000 7fc5d6ffd640 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-16T04:12:14.070 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h18' does not appear in the crush map 2024-09-16T04:12:14.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 >> v1:172.21.15.171:6812/7327273 conn(0x7fc5c4038f70 legacy=0x7fc5c403b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:14.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 10 monclient: shutdown 2024-09-16T04:12:14.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:14.072 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 >> v1:172.21.15.171:6789/0 conn(0x7fc5e00a0750 legacy=0x7fc5e009cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:14.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 shutdown_connections 2024-09-16T04:12:14.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 >> 172.21.15.171:0/688189900 conn(0x7fc5e00a52a0 msgr2=0x7fc5e00a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:14.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 shutdown_connections 2024-09-16T04:12:14.073 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.069+0000 7fc5e8386640 1 -- 172.21.15.171:0/688189900 wait complete. 2024-09-16T04:12:14.081 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:14.082 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h19 2024-09-16T04:12:14.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.157+0000 7f8d69b2c640 10 monclient: get_monmap_and_config 2024-09-16T04:12:14.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.157+0000 7f8d69b2c640 10 monclient: build_initial_monmap 2024-09-16T04:12:14.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.157+0000 7f8d69b2c640 10 monclient: monmap: 2024-09-16T04:12:14.161 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:14.161 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:14.161 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:14.159232+0000 2024-09-16T04:12:14.161 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:14.159232+0000 2024-09-16T04:12:14.161 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:14.162 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:14.162 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:14.162 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:14.162 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:14.162 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:14.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 1 Processor -- start 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 1 -- start start 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient: init 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient(hunting): picked mon.noname-c con 0x7f8d6409eaa0 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient(hunting): picked mon.noname-b con 0x7f8d64134b20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient(hunting): picked mon.noname-a con 0x7f8d64134f30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d6409d060 con 0x7f8d64134f30 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d6409d340 con 0x7f8d6409eaa0 2024-09-16T04:12:14.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d64068840 con 0x7f8d64134b20 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d69b2c640 10 monclient(hunting): authenticate will time out at 1109.664733s 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d637fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8d6409eaa0 0x7f8d6409ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52950/0 (socket says 172.21.15.171:52950) 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d637fe640 1 -- 172.21.15.171:0/445801496 learned_addr learned my addr 172.21.15.171:0/445801496 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.158+0000 7f8d62ffd640 1 -- 172.21.15.171:0/445801496 >> v1:172.21.15.148:6790/0 conn(0x7f8d64134b20 legacy=0x7f8d64139980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640381404 0 0) 0x7f8d6409d340 con 0x7f8d6409eaa0 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient(hunting): my global_id is 6253 2024-09-16T04:12:14.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d4c003520 con 0x7f8d6409eaa0 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 990271377 0 0) 0x7f8d4c003520 con 0x7f8d6409eaa0 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 >> v1:172.21.15.148:6790/0 conn(0x7f8d64134b20 legacy=0x7f8d64139980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 >> v1:172.21.15.148:6789/0 conn(0x7f8d64134f30 legacy=0x7f8d6413d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 monclient: found mon.noname-c 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d64141800 con 0x7f8d6409eaa0 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:14.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d54001610 con 0x7f8d6409eaa0 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 5 monclient: authenticate success, global_id 6253 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: dump: 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.181 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d54001900 con 0x7f8d6409eaa0 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:14.182 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 1 -- 172.21.15.171:0/445801496 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d54001cd0 con 0x7f8d6409eaa0 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 4 set_mon_vals no callback set 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d627fc640 10 monclient: dump: 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:14.183 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:14.184 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 10 monclient: shutdown 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 1 -- 172.21.15.171:0/445801496 >> v1:172.21.15.171:6789/0 conn(0x7f8d6409eaa0 legacy=0x7f8d6409ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 1 -- 172.21.15.171:0/445801496 shutdown_connections 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 1 -- 172.21.15.171:0/445801496 >> 172.21.15.171:0/445801496 conn(0x7f8d640a1e90 msgr2=0x7f8d640a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 1 -- 172.21.15.171:0/445801496 shutdown_connections 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.159+0000 7f8d69b2c640 1 -- 172.21.15.171:0/445801496 wait complete. 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.160+0000 7f8d69b2c640 10 monclient: build_initial_monmap 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.160+0000 7f8d69b2c640 10 monclient: monmap: 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:14.185 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.160+0000 7f8d69b2c640 1 Processor -- start 2024-09-16T04:12:14.186 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.160+0000 7f8d69b2c640 1 -- start start 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.160+0000 7f8d69b2c640 10 monclient: init 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient(hunting): picked mon.noname-a con 0x7f8d6409eaa0 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient(hunting): picked mon.noname-b con 0x7f8d64134b20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient(hunting): picked mon.noname-c con 0x7f8d64134f30 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d64139c30 con 0x7f8d6409eaa0 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d64138710 con 0x7f8d64134b20 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d640a3520 con 0x7f8d64134f30 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d69b2c640 10 monclient(hunting): authenticate will time out at 1109.667413s 2024-09-16T04:12:14.187 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d63fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f8d64134f30 legacy=0x7f8d641d1760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d62ffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8d64134b20 0x7f8d64136a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:52960/0 (socket says 172.21.15.171:52960) 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d62ffd640 1 -- 172.21.15.171:0/519865444 learned_addr learned my addr 172.21.15.171:0/519865444 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165598572 0 0) 0x7f8d64138710 con 0x7f8d64134b20 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): my global_id is 6256 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d50003520 con 0x7f8d64134b20 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162318443 0 0) 0x7f8d64139c30 con 0x7f8d6409eaa0 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:14.188 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): my global_id is 64689 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d64138710 con 0x7f8d6409eaa0 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2577214302 0 0) 0x7f8d50003520 con 0x7f8d64134b20 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d64139c30 con 0x7f8d64134b20 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d580041d0 con 0x7f8d64134b20 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): dump: 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:14.189 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.161+0000 7f8d4bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 349689914 0 0) 0x7f8d64139c30 con 0x7f8d64134b20 2024-09-16T04:12:14.190 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 >> v1:172.21.15.148:6790/0 conn(0x7f8d64134f30 legacy=0x7f8d641d1760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 >> v1:172.21.15.148:6789/0 conn(0x7f8d6409eaa0 legacy=0x7f8d641362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 monclient: found mon.b 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6409c580 con 0x7f8d64134b20 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: _finish_auth 0 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 5 monclient: authenticate success, global_id 6256 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 10 monclient: _renew_subs 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d641392b0 con 0x7f8d64134b20 2024-09-16T04:12:14.191 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 10 monclient: _renew_subs 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d64137110 con 0x7f8d64134b20 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d580044f0 con 0x7f8d64134b20 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8d580048c0 con 0x7f8d64134b20 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d61ffb640 4 set_mon_vals no callback set 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 10 monclient: dump: 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.192 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.162+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8d58004a60 con 0x7f8d64134b20 2024-09-16T04:12:14.193 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.163+0000 7f8d69b2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.163+0000 7f8d69b2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.163+0000 7f8d69b2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.163+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d44005080 con 0x7f8d64134b20 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.163+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(233..233 src has 1..233) ==== 10337+0+0 (unknown 2213365384 0 0) 0x7f8d58048850 con 0x7f8d64134b20 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.163+0000 7f8d637fe640 10 monclient: get_auth_request con 0x7f8d50038b90 auth_method 0 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.167+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8d58015d10 con 0x7f8d64134b20 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.167+0000 7f8d4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:14.194 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.167+0000 7f8d4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:14.308 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:12:14.308 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:12:14.308 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:12:14.309 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:12:14.313 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:12:14.313 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:12:14.313 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:12:14.314 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:12:14.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.337+0000 7f8d69b2c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T04:12:14.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.337+0000 7f8d69b2c640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T04:12:14.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.337+0000 7f8d69b2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:14.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:14.337+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) -- 0x7f8d44005370 con 0x7f8d64134b20 2024-09-16T04:12:17.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:17.162+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:17.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:17.162+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:17.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:17.162+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:17.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:17.162+0000 7f8d4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:18.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:18.162+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:18.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:18.162+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:18.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:18.162+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:19.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:19.162+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:19.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:19.162+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:19.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:19.162+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:20.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:20.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:20.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:20.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:20.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:20.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:21.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:21.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:21.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:21.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:21.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:21.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:22.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:22.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:22.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:22.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:22.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:22.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:23.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:23.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:23.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:23.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:23.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:23.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:24.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:24.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:24.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:24.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:24.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:24.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:25.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:25.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:25.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:25.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:25.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:25.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:26.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:26.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:26.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:26.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:26.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:26.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:27.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:27.163+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:27.163+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:27.163+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:27.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:27.163+0000 7f8d4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:28.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:28.164+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:28.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:28.164+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:28.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:28.164+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.164+0000 7f8d4affd640 10 monclient: tick 2024-09-16T04:12:29.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.164+0000 7f8d4affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.164+0000 7f8d4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.314 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:12:29.315 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:12:29.315 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:12:29.315 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:29.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.380+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v234) ==== 120+0+0 (unknown 2400711699 0 0) 0x7f8d58015610 con 0x7f8d64134b20 2024-09-16T04:12:29.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.380+0000 7f8d4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T04:12:29.382 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.380+0000 7f8d4bfff640 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-16T04:12:29.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.429+0000 7f0d8f227640 10 monclient: get_monmap_and_config 2024-09-16T04:12:29.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.429+0000 7f0d8f227640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.429+0000 7f0d8f227640 10 monclient: monmap: 2024-09-16T04:12:29.431 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:29.431 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.431 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:29.430567+0000 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:29.430567+0000 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:29.432 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 1 Processor -- start 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 1 -- start start 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient: init 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d880a1010 con 0x7f0d8809bb70 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d8809f030 con 0x7f0d8809cc30 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d8809dc60 con 0x7f0d8809c660 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.430+0000 7f0d8f227640 10 monclient(hunting): authenticate will time out at 1170.055958s 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8cfc3640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0d8809bb70 0x7f0d8809bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40326/0 (socket says 172.21.15.148:40326) 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d87fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f0d8809c660 0x7f0d88138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35566/0 (socket says 172.21.15.148:35566) 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8cfc3640 1 -- 172.21.15.148:0/895899340 learned_addr learned my addr 172.21.15.148:0/895899340 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836902969 0 0) 0x7f0d880a1010 con 0x7f0d8809bb70 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): my global_id is 64692 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d78003520 con 0x7f0d8809bb70 2024-09-16T04:12:29.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4199677915 0 0) 0x7f0d8809dc60 con 0x7f0d8809c660 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): my global_id is 74099 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d880a1010 con 0x7f0d8809c660 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736319183 0 0) 0x7f0d8809f030 con 0x7f0d8809cc30 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient(hunting): my global_id is 6259 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d8809dc60 con 0x7f0d8809cc30 2024-09-16T04:12:29.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 124716096 0 0) 0x7f0d78003520 con 0x7f0d8809bb70 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 >> v1:172.21.15.148:6790/0 conn(0x7f0d8809c660 legacy=0x7f0d88138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 >> v1:172.21.15.171:6789/0 conn(0x7f0d8809cc30 legacy=0x7f0d8813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 monclient: found mon.noname-a 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8809df40 con 0x7f0d8809bb70 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8f227640 5 monclient: authenticate success, global_id 64692 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0d7c001610 con 0x7f0d8809bb70 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8f227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: dump: 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8f227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d7c0018c0 con 0x7f0d8809bb70 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 1 -- 172.21.15.148:0/895899340 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0d7c001c90 con 0x7f0d8809bb70 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d877fe640 10 monclient: dump: 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.445 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8f227640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.431+0000 7f0d8f227640 4 set_mon_vals no callback set 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 10 monclient: shutdown 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 1 -- 172.21.15.148:0/895899340 >> v1:172.21.15.148:6789/0 conn(0x7f0d8809bb70 legacy=0x7f0d8809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 1 -- 172.21.15.148:0/895899340 shutdown_connections 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 1 -- 172.21.15.148:0/895899340 >> 172.21.15.148:0/895899340 conn(0x7f0d880a5640 msgr2=0x7f0d880a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 1 -- 172.21.15.148:0/895899340 shutdown_connections 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 1 -- 172.21.15.148:0/895899340 wait complete. 2024-09-16T04:12:29.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.432+0000 7f0d8f227640 10 monclient: monmap: 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 1 Processor -- start 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 1 -- start start 2024-09-16T04:12:29.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient: init 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d880740a0 con 0x7f0d8809c660 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d88122140 con 0x7f0d8809bb70 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d88121190 con 0x7f0d8809cc30 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.433+0000 7f0d8f227640 10 monclient(hunting): authenticate will time out at 1170.058989s 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d87fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0d8809c660 0x7f0d8811f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40328/0 (socket says 172.21.15.148:40328) 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8d7c4640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f0d8809cc30 0x7f0d881cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35576/0 (socket says 172.21.15.148:35576) 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d87fff640 1 -- 172.21.15.148:0/259711921 learned_addr learned my addr 172.21.15.148:0/259711921 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8d7c4640 1 -- 172.21.15.148:0/259711921 >> v1:172.21.15.148:6790/0 conn(0x7f0d8809cc30 legacy=0x7f0d881cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8d7c4640 1 -- 172.21.15.148:0/259711921 >> v1:172.21.15.148:6790/0 conn(0x7f0d8809cc30 legacy=0x7f0d881cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8d7c4640 1 --1- 172.21.15.148:0/259711921 >> v1:172.21.15.148:6790/0 conn(0x7f0d8809cc30 0x7f0d881cf780 :-1 s=OPENED pgs=611 cs=1 l=1).handle_message read tag failed 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8d7c4640 1 --1- 172.21.15.148:0/259711921 >> v1:172.21.15.148:6790/0 conn(0x7f0d8809cc30 0x7f0d881cf780 :-1 s=OPENED pgs=611 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6790/0 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1870539258 0 0) 0x7f0d880a1010 con 0x7f0d8809c660 2024-09-16T04:12:29.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): my global_id is 64695 2024-09-16T04:12:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d6c003520 con 0x7f0d8809c660 2024-09-16T04:12:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227370964 0 0) 0x7f0d88122140 con 0x7f0d8809bb70 2024-09-16T04:12:29.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): my global_id is 6262 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d880a1010 con 0x7f0d8809bb70 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2298787951 0 0) 0x7f0d6c003520 con 0x7f0d8809c660 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d88122140 con 0x7f0d8809c660 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0d74001870 con 0x7f0d8809c660 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): dump: 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.452 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2750353455 0 0) 0x7f0d880a1010 con 0x7f0d8809bb70 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d6c003520 con 0x7f0d8809bb70 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0d7c001a70 con 0x7f0d8809bb70 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient(hunting): dump: 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3793861139 0 0) 0x7f0d88122140 con 0x7f0d8809c660 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 >> v1:172.21.15.148:6790/0 conn(0x7f0d8809cc30 legacy=0x7f0d881cf780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 >> v1:172.21.15.171:6789/0 conn(0x7f0d8809bb70 legacy=0x7f0d8811ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 monclient: found mon.a 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d881208b0 con 0x7f0d8809c660 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 5 monclient: authenticate success, global_id 64695 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 10 monclient: _renew_subs 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d88072350 con 0x7f0d8809c660 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d740047e0 con 0x7f0d8809c660 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 10 monclient: _renew_subs 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d8811f850 con 0x7f0d8809c660 2024-09-16T04:12:29.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0d74004b30 con 0x7f0d8809c660 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d857fa640 10 monclient: dump: 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.457 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.434+0000 7f0d86ffd640 4 set_mon_vals no callback set 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0d7400ff60 con 0x7f0d8809c660 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d8f227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d8f227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d8f227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(234..234 src has 1..234) ==== 10124+0+0 (unknown 293951658 0 0) 0x7f0d740486e0 con 0x7f0d8809c660 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d880720e0 con 0x7f0d8809c660 2024-09-16T04:12:29.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.435+0000 7f0d8cfc3640 10 monclient: get_auth_request con 0x7f0d6c038ea0 auth_method 0 2024-09-16T04:12:29.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.440+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0d880720e0 con 0x7f0d8809c660 2024-09-16T04:12:29.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.440+0000 7f0d857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.440+0000 7f0d857fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:29.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.489+0000 7f8d69b2c640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T04:12:29.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.489+0000 7f8d69b2c640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T04:12:29.491 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.489+0000 7f8d69b2c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.489+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) -- 0x7f8d44001090 con 0x7f8d64134b20 2024-09-16T04:12:29.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.490+0000 7f8d4bfff640 1 -- 172.21.15.171:0/519865444 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v234) ==== 120+0+0 (unknown 3233416739 0 0) 0x7f8d5801e1c0 con 0x7f8d64134b20 2024-09-16T04:12:29.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.491+0000 7f8d4bfff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-16T04:12:29.492 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.491+0000 7f8d4bfff640 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-16T04:12:29.492 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h19' does not appear in the crush map 2024-09-16T04:12:29.494 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.492+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 >> v1:172.21.15.171:6812/7327273 conn(0x7f8d50038b90 legacy=0x7f8d5003b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.492+0000 7f8d69b2c640 10 monclient: shutdown 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.492+0000 7f8d69b2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.492+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 >> v1:172.21.15.171:6789/0 conn(0x7f8d64134b20 legacy=0x7f8d64136a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.493+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 shutdown_connections 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.493+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 >> 172.21.15.171:0/519865444 conn(0x7f8d640a1e90 msgr2=0x7f8d640a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.493+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 shutdown_connections 2024-09-16T04:12:29.495 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.493+0000 7f8d69b2c640 1 -- 172.21.15.171:0/519865444 wait complete. 2024-09-16T04:12:29.504 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:29.504 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h20 2024-09-16T04:12:29.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.581+0000 7fdda0077640 10 monclient: get_monmap_and_config 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.581+0000 7fdda0077640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.581+0000 7fdda0077640 10 monclient: monmap: 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:29.582296+0000 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:29.582296+0000 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.584 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:29.585 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 1 Processor -- start 2024-09-16T04:12:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 1 -- start start 2024-09-16T04:12:29.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient: init 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient(hunting): picked mon.noname-b con 0x7fdd98132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9809ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient(hunting): picked mon.noname-a con 0x7fdd98132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd981332b0 con 0x7fdd98132e60 2024-09-16T04:12:29.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd98134310 con 0x7fdd9809ec20 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd9809d980 con 0x7fdd98132a80 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdda0077640 10 monclient(hunting): authenticate will time out at 1125.087785s 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdd9d612640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdd9809ec20 0x7fdd98133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59364/0 (socket says 172.21.15.171:59364) 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.582+0000 7fdd9d612640 1 -- 172.21.15.171:0/1172672508 learned_addr learned my addr 172.21.15.171:0/1172672508 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1002283783 0 0) 0x7fdd98134310 con 0x7fdd9809ec20 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): my global_id is 6265 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd84003520 con 0x7fdd9809ec20 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280987767 0 0) 0x7fdd981332b0 con 0x7fdd98132e60 2024-09-16T04:12:29.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): my global_id is 64698 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd98134310 con 0x7fdd98132e60 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522067711 0 0) 0x7fdd9809d980 con 0x7fdd98132a80 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient(hunting): my global_id is 74102 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd981332b0 con 0x7fdd98132a80 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3633128364 0 0) 0x7fdd84003520 con 0x7fdd9809ec20 2024-09-16T04:12:29.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 >> v1:172.21.15.148:6790/0 conn(0x7fdd98132a80 legacy=0x7fdd981334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 >> v1:172.21.15.148:6789/0 conn(0x7fdd98132e60 legacy=0x7fdd9813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 monclient: found mon.noname-c 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9809dc60 con 0x7fdd9809ec20 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 5 monclient: authenticate success, global_id 6265 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd94001610 con 0x7fdd9809ec20 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: dump: 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.598 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd940018c0 con 0x7fdd9809ec20 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 1 -- 172.21.15.171:0/1172672508 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd94001c90 con 0x7fdd9809ec20 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 4 set_mon_vals no callback set 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdd9ce11640 10 monclient: dump: 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.599 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 10 monclient: shutdown 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.583+0000 7fdda0077640 1 -- 172.21.15.171:0/1172672508 >> v1:172.21.15.171:6789/0 conn(0x7fdd9809ec20 legacy=0x7fdd98133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 1 -- 172.21.15.171:0/1172672508 shutdown_connections 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 1 -- 172.21.15.171:0/1172672508 >> 172.21.15.171:0/1172672508 conn(0x7fdd980a52a0 msgr2=0x7fdd980a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 1 -- 172.21.15.171:0/1172672508 shutdown_connections 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 1 -- 172.21.15.171:0/1172672508 wait complete. 2024-09-16T04:12:29.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 10 monclient: monmap: 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:29.601 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 1 Processor -- start 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 1 -- start start 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.584+0000 7fdda0077640 10 monclient: init 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9809ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient(hunting): picked mon.noname-a con 0x7fdd98132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient(hunting): picked mon.noname-b con 0x7fdd98132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd9809cb60 con 0x7fdd98132a80 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd9811bc20 con 0x7fdd98132e60 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd9811d790 con 0x7fdd9809ec20 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdda0077640 10 monclient(hunting): authenticate will time out at 1125.090413s 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd9e614640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fdd98132e60 0x7fdd981cfd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59370/0 (socket says 172.21.15.171:59370) 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd9e614640 1 -- 172.21.15.171:0/3029724566 learned_addr learned my addr 172.21.15.171:0/3029724566 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668177239 0 0) 0x7fdd9811bc20 con 0x7fdd98132e60 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): my global_id is 6268 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd7c003520 con 0x7fdd98132e60 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087563916 0 0) 0x7fdd9811d790 con 0x7fdd9809ec20 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): my global_id is 74105 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd9811bc20 con 0x7fdd9809ec20 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3613335437 0 0) 0x7fdd9809dc60 con 0x7fdd98132a80 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): my global_id is 64701 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd9811d790 con 0x7fdd98132a80 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1368975504 0 0) 0x7fdd7c003520 con 0x7fdd98132e60 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd9809dc60 con 0x7fdd98132e60 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd90001c30 con 0x7fdd98132e60 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:29.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.585+0000 7fdd8a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2801996153 0 0) 0x7fdd9811d790 con 0x7fdd98132a80 2024-09-16T04:12:29.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd7c003520 con 0x7fdd98132a80 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2259133412 0 0) 0x7fdd9811bc20 con 0x7fdd9809ec20 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd9811d790 con 0x7fdd9809ec20 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd94001c10 con 0x7fdd98132a80 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.606 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.607 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.607 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.607 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd8c001830 con 0x7fdd9809ec20 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.608 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2880025121 0 0) 0x7fdd9809dc60 con 0x7fdd98132e60 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 >> v1:172.21.15.148:6790/0 conn(0x7fdd9809ec20 legacy=0x7fdd981c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 >> v1:172.21.15.148:6789/0 conn(0x7fdd98132a80 legacy=0x7fdd981cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 monclient: found mon.b 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd980a4010 con 0x7fdd98132e60 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 5 monclient: authenticate success, global_id 6268 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 10 monclient: _renew_subs 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9811d4b0 con 0x7fdd98132e60 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 10 monclient: _renew_subs 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd980729e0 con 0x7fdd98132e60 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd90003990 con 0x7fdd98132e60 2024-09-16T04:12:29.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdd90003ce0 con 0x7fdd98132e60 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 10 monclient: dump: 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:29.611 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8bfff640 4 set_mon_vals no callback set 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.586+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdd90003f00 con 0x7fdd98132e60 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.587+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(234..234 src has 1..234) ==== 10124+0+0 (unknown 293951658 0 0) 0x7fdd90048950 con 0x7fdd98132e60 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.587+0000 7fdd9de13640 10 monclient: get_auth_request con 0x7fdd7c039130 auth_method 0 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.587+0000 7fdda0077640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.587+0000 7fdda0077640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.587+0000 7fdda0077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.587+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd80005080 con 0x7fdd98132e60 2024-09-16T04:12:29.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.591+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdd90015e80 con 0x7fdd98132e60 2024-09-16T04:12:29.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.591+0000 7fdd8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.591+0000 7fdd8a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:29.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d8f227640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:29.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d8f227640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:29.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d8f227640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0d880008d0 con 0x7f0d8809c660 2024-09-16T04:12:29.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d857fa640 1 -- 172.21.15.148:0/259711921 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3399668180) 0x7f0d880740a0 con 0x7f0d8809c660 2024-09-16T04:12:29.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:29.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.670+0000 7f0d857fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:29.675 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 >> v1:172.21.15.171:6812/7327273 conn(0x7f0d6c038ea0 legacy=0x7f0d6c03b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 10 monclient: shutdown 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 >> v1:172.21.15.148:6789/0 conn(0x7f0d8809c660 legacy=0x7f0d8811f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 shutdown_connections 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 >> 172.21.15.148:0/259711921 conn(0x7f0d880a5640 msgr2=0x7f0d88133890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 shutdown_connections 2024-09-16T04:12:29.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.672+0000 7f0d8f227640 1 -- 172.21.15.148:0/259711921 wait complete. 2024-09-16T04:12:29.684 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:12:29.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:12:29.685 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:12:29.763 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.761+0000 7fdda0077640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T04:12:29.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.761+0000 7fdda0077640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T04:12:29.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.761+0000 7fdda0077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:29.764 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:29.761+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) -- 0x7fdd80005370 con 0x7fdd98132e60 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.781+0000 7f5f02d6d640 10 monclient: get_monmap_and_config 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.781+0000 7f5f02d6d640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.781+0000 7f5f02d6d640 10 monclient: monmap: 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:29.782074+0000 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:29.782074+0000 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.783 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.784 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:29.784 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:29.784 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.781+0000 7f5f02d6d640 1 Processor -- start 2024-09-16T04:12:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.781+0000 7f5f02d6d640 1 -- start start 2024-09-16T04:12:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.781+0000 7f5f02d6d640 10 monclient: init 2024-09-16T04:12:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:29.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc13ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc135360 con 0x7f5efc13ae80 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc134440 con 0x7f5efc134870 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc09e650 con 0x7f5efc09e240 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f02d6d640 10 monclient(hunting): authenticate will time out at 1170.407462s 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f0130a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5efc13ae80 0x7f5efc13d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40330/0 (socket says 172.21.15.148:40330) 2024-09-16T04:12:29.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5f0130a640 1 -- 172.21.15.148:0/2272961410 learned_addr learned my addr 172.21.15.148:0/2272961410 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783829986 0 0) 0x7f5efc135360 con 0x7f5efc13ae80 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): my global_id is 64704 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee8003520 con 0x7f5efc13ae80 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751004529 0 0) 0x7f5efc09e650 con 0x7f5efc09e240 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): my global_id is 74108 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5efc135360 con 0x7f5efc09e240 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3242426715 0 0) 0x7f5efc134440 con 0x7f5efc134870 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 10 monclient(hunting): my global_id is 6271 2024-09-16T04:12:29.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.782+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5efc09e650 con 0x7f5efc134870 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2837149796 0 0) 0x7f5efc135360 con 0x7f5efc09e240 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 >> v1:172.21.15.171:6789/0 conn(0x7f5efc134870 legacy=0x7f5efc134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 >> v1:172.21.15.148:6789/0 conn(0x7f5efc13ae80 legacy=0x7f5efc13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 monclient: found mon.noname-b 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc073230 con 0x7f5efc09e240 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5eec001970 con 0x7f5efc09e240 2024-09-16T04:12:29.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 5 monclient: authenticate success, global_id 74108 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: dump: 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.799 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5eec001c50 con 0x7f5efc09e240 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 1 -- 172.21.15.148:0/2272961410 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5eec002060 con 0x7f5efc09e240 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5efb7fe640 10 monclient: dump: 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.800 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 4 set_mon_vals no callback set 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 10 monclient: shutdown 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 1 -- 172.21.15.148:0/2272961410 >> v1:172.21.15.148:6790/0 conn(0x7f5efc09e240 legacy=0x7f5efc139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 1 -- 172.21.15.148:0/2272961410 shutdown_connections 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 1 -- 172.21.15.148:0/2272961410 >> 172.21.15.148:0/2272961410 conn(0x7f5efc0a5640 msgr2=0x7f5efc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 1 -- 172.21.15.148:0/2272961410 shutdown_connections 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.783+0000 7f5f02d6d640 1 -- 172.21.15.148:0/2272961410 wait complete. 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.784+0000 7f5f02d6d640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.784+0000 7f5f02d6d640 10 monclient: monmap: 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:29.802 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.784+0000 7f5f02d6d640 1 Processor -- start 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.784+0000 7f5f02d6d640 1 -- start start 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.784+0000 7f5f02d6d640 10 monclient: init 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc13ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc11eb00 con 0x7f5efc13ae80 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc0a4520 con 0x7f5efc134870 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc0740a0 con 0x7f5efc09e240 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f02d6d640 10 monclient(hunting): authenticate will time out at 1170.410377s 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f00b09640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5efc09e240 0x7f5efc133400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35590/0 (socket says 172.21.15.148:35590) 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5f00b09640 1 -- 172.21.15.148:0/13518604 learned_addr learned my addr 172.21.15.148:0/13518604 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167015262 0 0) 0x7f5efc0740a0 con 0x7f5efc09e240 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 74111 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5edc003520 con 0x7f5efc09e240 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175636090 0 0) 0x7f5efc11eb00 con 0x7f5efc13ae80 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 64707 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5efc0740a0 con 0x7f5efc13ae80 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2014915964 0 0) 0x7f5efc0a4520 con 0x7f5efc134870 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 6274 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5efc11eb00 con 0x7f5efc134870 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3894394655 0 0) 0x7f5edc003520 con 0x7f5efc09e240 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5efc0a4520 con 0x7f5efc09e240 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1141248482 0 0) 0x7f5efc0740a0 con 0x7f5efc13ae80 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5edc003520 con 0x7f5efc13ae80 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5ef4001990 con 0x7f5efc09e240 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): dump: 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3266920673 0 0) 0x7f5efc11eb00 con 0x7f5efc134870 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5efc0740a0 con 0x7f5efc134870 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5ee40012b0 con 0x7f5efc13ae80 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): dump: 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.807 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5eec001e00 con 0x7f5efc134870 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.785+0000 7f5ef97fa640 10 monclient(hunting): dump: 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.808 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3240795397 0 0) 0x7f5efc0a4520 con 0x7f5efc09e240 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.171:6789/0 conn(0x7f5efc134870 legacy=0x7f5efc1ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.148:6789/0 conn(0x7f5efc13ae80 legacy=0x7f5efc1d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 monclient: found mon.c 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc11e820 con 0x7f5efc09e240 2024-09-16T04:12:29.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 5 monclient: authenticate success, global_id 74111 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 10 monclient: _renew_subs 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5efc11d3d0 con 0x7f5efc09e240 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 10 monclient: _renew_subs 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5efc11c250 con 0x7f5efc09e240 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ef40038c0 con 0x7f5efc09e240 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5ef4003af0 con 0x7f5efc09e240 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5efaffd640 4 set_mon_vals no callback set 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 10 monclient: dump: 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5ef4003af0 con 0x7f5efc09e240 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.786+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ee00065c0 con 0x7f5ee0002860 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(234..234 src has 1..234) ==== 10124+0+0 (unknown 293951658 0 0) 0x7f5ef40485a0 con 0x7f5efc09e240 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5f0130a640 10 monclient: get_auth_request con 0x7f5edc039130 auth_method 0 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052028463 0 0) 0x7f5ee00065c0 con 0x7f5ee0002860 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 64710 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5efc0a4520 con 0x7f5ee0002860 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1898618380 0 0) 0x7f5efc0a4520 con 0x7f5ee0002860 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ee00065c0 con 0x7f5ee0002860 2024-09-16T04:12:29.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5eec003b00 con 0x7f5ee0002860 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 10 monclient: dump: 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1378356122 0 0) 0x7f5ee00065c0 con 0x7f5ee0002860 2024-09-16T04:12:29.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.787+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ee000a150 con 0x7f5ee0002860 2024-09-16T04:12:29.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.788+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5eec003e30 con 0x7f5ee0002860 2024-09-16T04:12:29.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.788+0000 7f5ef97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.788+0000 7f5ef97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:29.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.788+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.148:6789/0 conn(0x7f5ee0002860 legacy=0x7f5ee0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.788+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.148:6789/0 conn(0x7f5ee0002860 legacy=0x7f5ee0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.793+0000 7f5f02d6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5ee0009e20 con 0x7f5ee00090e0 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1045008721 0 0) 0x7f5ee0009e20 con 0x7f5ee00090e0 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 64713 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee00065c0 con 0x7f5ee00090e0 2024-09-16T04:12:29.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 232140035 0 0) 0x7f5ee00065c0 con 0x7f5ee00090e0 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ee0009e20 con 0x7f5ee00090e0 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5eec004510 con 0x7f5ee00090e0 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient: dump: 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.817 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2403656320 0 0) 0x7f5ee0009e20 con 0x7f5ee00090e0 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ee000c390 con 0x7f5ee00090e0 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4146 (unknown 0 0 2273409741) 0x7f5eec005870 con 0x7f5ee00090e0 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.148:6789/0 conn(0x7f5ee00090e0 legacy=0x7f5ee0009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.794+0000 7f5ef97fa640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.148:6789/0 conn(0x7f5ee00090e0 legacy=0x7f5ee0009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:29.818 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 28, 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 128992, 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:29.819 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:29.820 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:29.821 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:29.822 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:29.823 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.824 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:29.825 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:29.826 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:29.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:12:29.829 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.830 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.830 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:29.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.830 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:29.830 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:29.830 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:29.831 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.831 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.831 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.831 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:29.831 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:29.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.171:6812/7327273 conn(0x7f5edc039130 legacy=0x7f5edc03b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 10 monclient: shutdown 2024-09-16T04:12:29.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 >> v1:172.21.15.148:6790/0 conn(0x7f5efc09e240 legacy=0x7f5efc133400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f0130a640 1 -- 172.21.15.148:0/13518604 reap_dead start 2024-09-16T04:12:29.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f0130a640 1 -- 172.21.15.148:0/13518604 reap_dead start 2024-09-16T04:12:29.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 shutdown_connections 2024-09-16T04:12:29.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 >> 172.21.15.148:0/13518604 conn(0x7f5efc0a5640 msgr2=0x7f5efc0a35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 shutdown_connections 2024-09-16T04:12:29.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.795+0000 7f5f02d6d640 1 -- 172.21.15.148:0/13518604 wait complete. 2024-09-16T04:12:29.834 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:29.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:12:29.834 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:12:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.930+0000 7f8fc02fa640 10 monclient: get_monmap_and_config 2024-09-16T04:12:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.930+0000 7f8fc02fa640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.930+0000 7f8fc02fa640 10 monclient: monmap: 2024-09-16T04:12:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.932 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:29.931282+0000 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:29.931282+0000 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:29.933 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.930+0000 7f8fc02fa640 1 Processor -- start 2024-09-16T04:12:29.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.930+0000 7f8fc02fa640 1 -- start start 2024-09-16T04:12:29.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.930+0000 7f8fc02fa640 10 monclient: init 2024-09-16T04:12:29.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:29.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb8134b40 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb8134610 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb813b050 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb80a1910 con 0x7f8fb8134b40 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb8099690 con 0x7f8fb813b050 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb8099970 con 0x7f8fb8134610 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fc02fa640 10 monclient(hunting): authenticate will time out at 1170.556633s 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbe096640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8fb8134b40 0x7f8fb8134f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40382/0 (socket says 172.21.15.148:40382) 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbe096640 1 -- 172.21.15.148:0/1515413210 learned_addr learned my addr 172.21.15.148:0/1515413210 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbd895640 1 -- 172.21.15.148:0/1515413210 >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134610 legacy=0x7f8fb8139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbd895640 1 -- 172.21.15.148:0/1515413210 >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134610 legacy=0x7f8fb8139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbd895640 1 --1- 172.21.15.148:0/1515413210 >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134610 0x7f8fb8139930 :-1 s=OPENED pgs=616 cs=1 l=1).handle_message read tag failed 2024-09-16T04:12:29.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbd895640 1 --1- 172.21.15.148:0/1515413210 >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134610 0x7f8fb8139930 :-1 s=OPENED pgs=616 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbd094640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6790/0 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.931+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 928963524 0 0) 0x7f8fb80a1910 con 0x7f8fb8134b40 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): my global_id is 64716 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fa4003520 con 0x7f8fb8134b40 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269379806 0 0) 0x7f8fb8099690 con 0x7f8fb813b050 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient(hunting): my global_id is 6277 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fb80a1910 con 0x7f8fb813b050 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2943901608 0 0) 0x7f8fa4003520 con 0x7f8fb8134b40 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134610 legacy=0x7f8fb8139930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 >> v1:172.21.15.171:6789/0 conn(0x7f8fb813b050 legacy=0x7f8fb813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 monclient: found mon.noname-a 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb809f160 con 0x7f8fb8134b40 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fb4001610 con 0x7f8fb8134b40 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 5 monclient: authenticate success, global_id 64716 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: dump: 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.950 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8fb4001930 con 0x7f8fb8134b40 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 1 -- 172.21.15.148:0/1515413210 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fb4001d00 con 0x7f8fb8134b40 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 4 set_mon_vals no callback set 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fbd094640 10 monclient: dump: 2024-09-16T04:12:29.951 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 10 monclient: shutdown 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 1 -- 172.21.15.148:0/1515413210 >> v1:172.21.15.148:6789/0 conn(0x7f8fb8134b40 legacy=0x7f8fb8134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 1 -- 172.21.15.148:0/1515413210 shutdown_connections 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 1 -- 172.21.15.148:0/1515413210 >> 172.21.15.148:0/1515413210 conn(0x7f8fb80a5860 msgr2=0x7f8fb80a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.932+0000 7f8fc02fa640 1 -- 172.21.15.148:0/1515413210 shutdown_connections 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.933+0000 7f8fc02fa640 1 -- 172.21.15.148:0/1515413210 wait complete. 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.933+0000 7f8fc02fa640 10 monclient: build_initial_monmap 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.933+0000 7f8fc02fa640 10 monclient: monmap: 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:29.953 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.933+0000 7f8fc02fa640 1 Processor -- start 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.933+0000 7f8fc02fa640 1 -- start start 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.933+0000 7f8fc02fa640 10 monclient: init 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb8134610 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb8134b40 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:29.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb813b050 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb8098f00 con 0x7f8fb8134610 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb808fba0 con 0x7f8fb813b050 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fbe096640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8fb8134610 0x7f8fb808d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40392/0 (socket says 172.21.15.148:40392) 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fbe096640 1 -- 172.21.15.148:0/3149455968 learned_addr learned my addr 172.21.15.148:0/3149455968 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fbd895640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134b40 0x7f8fb808dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35608/0 (socket says 172.21.15.148:35608) 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb813ab20 con 0x7f8fb8134b40 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8fc02fa640 10 monclient(hunting): authenticate will time out at 1170.559517s 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 82810750 0 0) 0x7f8fb8098f00 con 0x7f8fb8134610 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): my global_id is 64719 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f9c003520 con 0x7f8fb8134610 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8095026 0 0) 0x7f8fb813ab20 con 0x7f8fb8134b40 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): my global_id is 74114 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fb8098f00 con 0x7f8fb8134b40 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132788181 0 0) 0x7f8fb808fba0 con 0x7f8fb813b050 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): my global_id is 6280 2024-09-16T04:12:29.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fb813ab20 con 0x7f8fb813b050 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1618354212 0 0) 0x7f8f9c003520 con 0x7f8fb8134610 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fb808fba0 con 0x7f8fb8134610 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fb40015d0 con 0x7f8fb8134610 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): dump: 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.957 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2057258645 0 0) 0x7f8fb8098f00 con 0x7f8fb8134b40 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f9c003520 con 0x7f8fb8134b40 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fa8001840 con 0x7f8fb8134b40 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.934+0000 7f8faeffd640 10 monclient(hunting): dump: 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.958 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3893391274 0 0) 0x7f8fb813ab20 con 0x7f8fb813b050 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fb8098f00 con 0x7f8fb813b050 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fb0001c30 con 0x7f8fb813b050 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:29.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient(hunting): dump: 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2809869187 0 0) 0x7f8fb808fba0 con 0x7f8fb8134610 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.148:6790/0 conn(0x7f8fb8134b40 legacy=0x7f8fb808dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.171:6789/0 conn(0x7f8fb813b050 legacy=0x7f8fb81d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 monclient: found mon.a 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb808eeb0 con 0x7f8fb8134610 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 5 monclient: authenticate success, global_id 64719 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 10 monclient: _renew_subs 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb8136270 con 0x7f8fb8134610 2024-09-16T04:12:29.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 10 monclient: _renew_subs 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8fb8136830 con 0x7f8fb8134610 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8fb4001930 con 0x7f8fb8134610 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fb4004de0 con 0x7f8fb8134610 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fbc893640 4 set_mon_vals no callback set 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8faeffd640 10 monclient: dump: 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.962 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.935+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8fa00065c0 con 0x7f8fa0002860 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8fb4004de0 con 0x7f8fb8134610 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(234..234 src has 1..234) ==== 10124+0+0 (unknown 293951658 0 0) 0x7f8fb4048650 con 0x7f8fb8134610 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8fbe897640 10 monclient: get_auth_request con 0x7f8f9c039160 auth_method 0 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510846394 0 0) 0x7f8fa00065c0 con 0x7f8fa0002860 2024-09-16T04:12:29.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient(hunting): my global_id is 74117 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fb808fba0 con 0x7f8fa0002860 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2813783688 0 0) 0x7f8fb808fba0 con 0x7f8fa0002860 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fa00065c0 con 0x7f8fa0002860 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fa8003790 con 0x7f8fa0002860 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 10 monclient: dump: 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.964 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3566346418 0 0) 0x7f8fa00065c0 con 0x7f8fa0002860 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.936+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8fa000a150 con 0x7f8fa0002860 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.937+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8fa80057c0 con 0x7f8fa0002860 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.937+0000 7f8faeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.937+0000 7f8faeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.937+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.148:6790/0 conn(0x7f8fa0002860 legacy=0x7f8fa0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.937+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.148:6790/0 conn(0x7f8fa0002860 legacy=0x7f8fa0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:29.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8fc02fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8fa0009e40 con 0x7f8fa0009100 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016071881 0 0) 0x7f8fa0009e40 con 0x7f8fa0009100 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient(hunting): my global_id is 74120 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fa00065c0 con 0x7f8fa0009100 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1579812401 0 0) 0x7f8fa00065c0 con 0x7f8fa0009100 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8fa0009e40 con 0x7f8fa0009100 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8fa8003d90 con 0x7f8fa0009100 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:29.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient: dump: 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3606748764 0 0) 0x7f8fa0009e40 con 0x7f8fa0009100 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8fa000c3d0 con 0x7f8fa0009100 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 3126423355) 0x7f8fa8004010 con 0x7f8fa0009100 2024-09-16T04:12:29.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.148:6790/0 conn(0x7f8fa0009100 legacy=0x7f8fa0009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.943+0000 7f8faeffd640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.148:6790/0 conn(0x7f8fa0009100 legacy=0x7f8fa0009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:29.968 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 28, 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 15543, 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:29.969 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:29.970 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:29.972 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:29.973 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:29.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.976 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:29.977 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:29.978 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:29.979 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:29.980 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:29.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.171:6812/7327273 conn(0x7f8f9c039160 legacy=0x7f8f9c03b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 10 monclient: shutdown 2024-09-16T04:12:29.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 >> v1:172.21.15.148:6789/0 conn(0x7f8fb8134610 legacy=0x7f8fb808d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fbe897640 1 -- 172.21.15.148:0/3149455968 reap_dead start 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fbe897640 1 -- 172.21.15.148:0/3149455968 reap_dead start 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 shutdown_connections 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 >> 172.21.15.148:0/3149455968 conn(0x7f8fb80a5860 msgr2=0x7f8fb80a3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 shutdown_connections 2024-09-16T04:12:29.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:29.944+0000 7f8fc02fa640 1 -- 172.21.15.148:0/3149455968 wait complete. 2024-09-16T04:12:29.982 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:29.982 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:12:29.983 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.078+0000 7f0fe0647640 10 monclient: get_monmap_and_config 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.078+0000 7f0fe0647640 10 monclient: build_initial_monmap 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.078+0000 7f0fe0647640 10 monclient: monmap: 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:30.079870+0000 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:30.079870+0000 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:30.081 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.079+0000 7f0fe0647640 1 Processor -- start 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.079+0000 7f0fe0647640 1 -- start start 2024-09-16T04:12:30.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.079+0000 7f0fe0647640 10 monclient: init 2024-09-16T04:12:30.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient(hunting): picked mon.noname-c con 0x7f0fd809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient(hunting): picked mon.noname-b con 0x7f0fd809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient(hunting): picked mon.noname-a con 0x7f0fd809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd80a1010 con 0x7f0fd809cc30 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd809f030 con 0x7f0fd809bb70 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd809dc60 con 0x7f0fd809c660 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient(hunting): authenticate will time out at 1170.705186s 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fddbe2640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f0fd809c660 0x7f0fd8138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35636/0 (socket says 172.21.15.148:35636) 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fddbe2640 1 -- 172.21.15.148:0/3512048287 learned_addr learned my addr 172.21.15.148:0/3512048287 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:30.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330838918 0 0) 0x7f0fd80a1010 con 0x7f0fd809cc30 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): my global_id is 64722 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc0003520 con 0x7f0fd809cc30 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757733619 0 0) 0x7f0fd809dc60 con 0x7f0fd809c660 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): my global_id is 74123 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd80a1010 con 0x7f0fd809c660 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3674305738 0 0) 0x7f0fd809f030 con 0x7f0fd809bb70 2024-09-16T04:12:30.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient(hunting): my global_id is 6283 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd809dc60 con 0x7f0fd809bb70 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3051219907 0 0) 0x7f0fd80a1010 con 0x7f0fd809c660 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 >> v1:172.21.15.171:6789/0 conn(0x7f0fd809bb70 legacy=0x7f0fd809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 >> v1:172.21.15.148:6789/0 conn(0x7f0fd809cc30 legacy=0x7f0fd813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: _finish_hunting 0 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 monclient: found mon.noname-b 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd809df40 con 0x7f0fd809c660 2024-09-16T04:12:30.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: _finish_auth 0 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: _check_auth_tickets 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fcc0015f0 con 0x7f0fd809c660 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 5 monclient: authenticate success, global_id 74123 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: dump: 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.099 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fcc001860 con 0x7f0fd809c660 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 1 -- 172.21.15.148:0/3512048287 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fcc001c70 con 0x7f0fd809c660 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fe0647640 4 set_mon_vals no callback set 2024-09-16T04:12:30.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.080+0000 7f0fdd3e1640 10 monclient: dump: 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.101 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 10 monclient: shutdown 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 1 -- 172.21.15.148:0/3512048287 >> v1:172.21.15.148:6790/0 conn(0x7f0fd809c660 legacy=0x7f0fd8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 1 -- 172.21.15.148:0/3512048287 shutdown_connections 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 1 -- 172.21.15.148:0/3512048287 >> 172.21.15.148:0/3512048287 conn(0x7f0fd80a5640 msgr2=0x7f0fd80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 1 -- 172.21.15.148:0/3512048287 shutdown_connections 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.081+0000 7f0fe0647640 1 -- 172.21.15.148:0/3512048287 wait complete. 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient: build_initial_monmap 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient: monmap: 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:30.102 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 1 Processor -- start 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 1 -- start start 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient: init 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:30.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient(hunting): picked mon.noname-c con 0x7f0fd809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient(hunting): picked mon.noname-a con 0x7f0fd809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient(hunting): picked mon.noname-b con 0x7f0fd809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd8121370 con 0x7f0fd809c660 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd8072080 con 0x7f0fd809cc30 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd8120db0 con 0x7f0fd809bb70 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fe0647640 10 monclient(hunting): authenticate will time out at 1170.707970s 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.082+0000 7f0fde3e3640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f0fd809bb70 0x7f0fd81339b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35646/0 (socket says 172.21.15.148:35646) 2024-09-16T04:12:30.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fde3e3640 1 -- 172.21.15.148:0/2569106764 learned_addr learned my addr 172.21.15.148:0/2569106764 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275933129 0 0) 0x7f0fd8121370 con 0x7f0fd809c660 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): my global_id is 64725 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc4003520 con 0x7f0fd809c660 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036585827 0 0) 0x7f0fd8120db0 con 0x7f0fd809bb70 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): my global_id is 74126 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd8121370 con 0x7f0fd809bb70 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1462956045 0 0) 0x7f0fc4003520 con 0x7f0fd809c660 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fd8120db0 con 0x7f0fd809c660 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fcc001bb0 con 0x7f0fd809c660 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.106 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 900121707 0 0) 0x7f0fd8121370 con 0x7f0fd809bb70 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fc4003520 con 0x7f0fd809bb70 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854175032 0 0) 0x7f0fd8072080 con 0x7f0fd809cc30 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): my global_id is 6286 2024-09-16T04:12:30.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fd8121370 con 0x7f0fd809cc30 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fd0001930 con 0x7f0fd809bb70 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.108 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 809482202 0 0) 0x7f0fc4003520 con 0x7f0fd809bb70 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.171:6789/0 conn(0x7f0fd809cc30 legacy=0x7f0fd81cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.148:6789/0 conn(0x7f0fd809c660 legacy=0x7f0fd81cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 monclient: found mon.c 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd8121090 con 0x7f0fd809bb70 2024-09-16T04:12:30.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 5 monclient: authenticate success, global_id 74126 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 10 monclient: _renew_subs 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd811f960 con 0x7f0fd809bb70 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 10 monclient: _renew_subs 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0fd81342a0 con 0x7f0fd809bb70 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0fd0001c70 con 0x7f0fd809bb70 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fd0003a60 con 0x7f0fd809bb70 2024-09-16T04:12:30.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fdcbe0640 4 set_mon_vals no callback set 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.083+0000 7f0fd6ffd640 10 monclient: dump: 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.111 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.084+0000 7f0fe0647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.084+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0fb80065c0 con 0x7f0fb8002860 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.084+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0fd0003c80 con 0x7f0fd809bb70 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.084+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(234..234 src has 1..234) ==== 10124+0+0 (unknown 293951658 0 0) 0x7f0fd0048770 con 0x7f0fd809bb70 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.084+0000 7f0fdebe4640 10 monclient: get_auth_request con 0x7f0fc40390b0 auth_method 0 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059961680 0 0) 0x7f0fb80065c0 con 0x7f0fb8002860 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient(hunting): my global_id is 6289 2024-09-16T04:12:30.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fc4003520 con 0x7f0fb8002860 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2109799849 0 0) 0x7f0fc4003520 con 0x7f0fb8002860 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fb80065c0 con 0x7f0fb8002860 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fcc004d90 con 0x7f0fb8002860 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient: dump: 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.113 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1657037432 0 0) 0x7f0fb80065c0 con 0x7f0fb8002860 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fb800a150 con 0x7f0fb8002860 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0fcc006290 con 0x7f0fb8002860 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.171:6789/0 conn(0x7f0fb8002860 legacy=0x7f0fb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.085+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.171:6789/0 conn(0x7f0fb8002860 legacy=0x7f0fb8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.091+0000 7f0fe0647640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.091+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0fb8009e40 con 0x7f0fb8009100 2024-09-16T04:12:30.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741380453 0 0) 0x7f0fb8009e40 con 0x7f0fb8009100 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient(hunting): my global_id is 6292 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fb80065c0 con 0x7f0fb8009100 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 685277394 0 0) 0x7f0fb80065c0 con 0x7f0fb8009100 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0fb8009e40 con 0x7f0fb8009100 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0fcc004f80 con 0x7f0fb8009100 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient: dump: 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.115 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1207334718 0 0) 0x7f0fb8009e40 con 0x7f0fb8009100 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0fb800c3d0 con 0x7f0fb8009100 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 417479873) 0x7f0fcc006510 con 0x7f0fb8009100 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:30.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.171:6789/0 conn(0x7f0fb8009100 legacy=0x7f0fb8009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fd6ffd640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.171:6789/0 conn(0x7f0fb8009100 legacy=0x7f0fb8009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 28, 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 374130, 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:30.117 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:30.118 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:30.119 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:30.120 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:30.121 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:30.122 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.123 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.124 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:30.125 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:30.126 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:30.127 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:30.128 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:30.129 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:30.130 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:30.130 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:30.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.171:6812/7327273 conn(0x7f0fc40390b0 legacy=0x7f0fc403b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fe0647640 10 monclient: shutdown 2024-09-16T04:12:30.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fe0647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:30.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 >> v1:172.21.15.148:6790/0 conn(0x7f0fd809bb70 legacy=0x7f0fd81339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fdebe4640 1 -- 172.21.15.148:0/2569106764 reap_dead start 2024-09-16T04:12:30.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fdebe4640 1 -- 172.21.15.148:0/2569106764 reap_dead start 2024-09-16T04:12:30.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.092+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 shutdown_connections 2024-09-16T04:12:30.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.093+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 >> 172.21.15.148:0/2569106764 conn(0x7f0fd80a5640 msgr2=0x7f0fd80a2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:30.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.093+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 shutdown_connections 2024-09-16T04:12:30.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.093+0000 7f0fe0647640 1 -- 172.21.15.148:0/2569106764 wait complete. 2024-09-16T04:12:30.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:30.132 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:12:30.132 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:12:30.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.228+0000 7f7166c2c640 10 monclient: get_monmap_and_config 2024-09-16T04:12:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.228+0000 7f7166c2c640 10 monclient: build_initial_monmap 2024-09-16T04:12:30.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.228+0000 7f7166c2c640 10 monclient: monmap: 2024-09-16T04:12:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:30.229856+0000 2024-09-16T04:12:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:30.229856+0000 2024-09-16T04:12:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:30.232 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:30.233 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:30.233 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.229+0000 7f7166c2c640 1 Processor -- start 2024-09-16T04:12:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.229+0000 7f7166c2c640 1 -- start start 2024-09-16T04:12:30.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.229+0000 7f7166c2c640 10 monclient: init 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient(hunting): picked mon.noname-c con 0x7f71600a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient(hunting): picked mon.noname-b con 0x7f71600a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient(hunting): picked mon.noname-a con 0x7f716013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71600a0e00 con 0x7f716013d060 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71600a10e0 con 0x7f71600a2410 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71600a4970 con 0x7f71600a1b10 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient(hunting): authenticate will time out at 1170.855148s 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715ffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f71600a1b10 0x7f716013b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35658/0 (socket says 172.21.15.148:35658) 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f71651c9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f716013d060 0x7f716013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40424/0 (socket says 172.21.15.148:40424) 2024-09-16T04:12:30.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715ffff640 1 -- 172.21.15.148:0/2396080494 learned_addr learned my addr 172.21.15.148:0/2396080494 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2631118097 0 0) 0x7f71600a0e00 con 0x7f716013d060 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): my global_id is 64728 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7150003520 con 0x7f716013d060 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 959509098 0 0) 0x7f71600a10e0 con 0x7f71600a2410 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): my global_id is 6295 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71600a0e00 con 0x7f71600a2410 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881380683 0 0) 0x7f71600a4970 con 0x7f71600a1b10 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient(hunting): my global_id is 74129 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71600a10e0 con 0x7f71600a1b10 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 605792342 0 0) 0x7f71600a0e00 con 0x7f71600a2410 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 >> v1:172.21.15.148:6790/0 conn(0x7f71600a1b10 legacy=0x7f716013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 >> v1:172.21.15.148:6789/0 conn(0x7f716013d060 legacy=0x7f716013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:30.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716006dfd0 con 0x7f71600a2410 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 5 monclient: authenticate success, global_id 6295 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7154001570 con 0x7f71600a2410 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: dump: 2024-09-16T04:12:30.246 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.247 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7154001850 con 0x7f71600a2410 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 1 -- 172.21.15.148:0/2396080494 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7154001c60 con 0x7f71600a2410 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f7166c2c640 4 set_mon_vals no callback set 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.230+0000 7f715f7fe640 10 monclient: dump: 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.248 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 10 monclient: shutdown 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 1 -- 172.21.15.148:0/2396080494 >> v1:172.21.15.171:6789/0 conn(0x7f71600a2410 legacy=0x7f71600a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 1 -- 172.21.15.148:0/2396080494 shutdown_connections 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 1 -- 172.21.15.148:0/2396080494 >> 172.21.15.148:0/2396080494 conn(0x7f71600a97b0 msgr2=0x7f71600a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:30.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 1 -- 172.21.15.148:0/2396080494 shutdown_connections 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 1 -- 172.21.15.148:0/2396080494 wait complete. 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 10 monclient: build_initial_monmap 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.231+0000 7f7166c2c640 10 monclient: monmap: 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 1 Processor -- start 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 1 -- start start 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient: init 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient(hunting): picked mon.noname-c con 0x7f71600a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient(hunting): picked mon.noname-a con 0x7f71600a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient(hunting): picked mon.noname-b con 0x7f716013d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716013a8f0 con 0x7f71600a2410 2024-09-16T04:12:30.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7160137bc0 con 0x7f716013d060 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7160139440 con 0x7f71600a1b10 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f7166c2c640 10 monclient(hunting): authenticate will time out at 1170.857913s 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f71649c8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f71600a1b10 0x7f7160136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35672/0 (socket says 172.21.15.148:35672) 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f715ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f71600a2410 0x7f71601374b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40438/0 (socket says 172.21.15.148:40438) 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.232+0000 7f71649c8640 1 -- 172.21.15.148:0/448743089 learned_addr learned my addr 172.21.15.148:0/448743089 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251005617 0 0) 0x7f716013a8f0 con 0x7f71600a2410 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): my global_id is 64731 2024-09-16T04:12:30.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7144003520 con 0x7f71600a2410 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025515906 0 0) 0x7f7160139440 con 0x7f71600a1b10 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): my global_id is 74132 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f716013a8f0 con 0x7f71600a1b10 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589153227 0 0) 0x7f7160137bc0 con 0x7f716013d060 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): my global_id is 6298 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7160139440 con 0x7f716013d060 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2535652648 0 0) 0x7f7144003520 con 0x7f71600a2410 2024-09-16T04:12:30.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7160137bc0 con 0x7f71600a2410 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f714c001830 con 0x7f71600a2410 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): dump: 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.254 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2634904632 0 0) 0x7f716013a8f0 con 0x7f71600a1b10 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7144003520 con 0x7f71600a1b10 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7154001c60 con 0x7f71600a1b10 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): dump: 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.255 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1759278792 0 0) 0x7f7160139440 con 0x7f716013d060 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f716013a8f0 con 0x7f716013d060 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7158001db0 con 0x7f716013d060 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient(hunting): dump: 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.256 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 858648022 0 0) 0x7f7160137bc0 con 0x7f71600a2410 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 >> v1:172.21.15.148:6790/0 conn(0x7f71600a1b10 legacy=0x7f7160136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 >> v1:172.21.15.171:6789/0 conn(0x7f716013d060 legacy=0x7f71601d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 monclient: found mon.a 2024-09-16T04:12:30.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716013ca80 con 0x7f71600a2410 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 5 monclient: authenticate success, global_id 64731 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 10 monclient: _renew_subs 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7160140940 con 0x7f71600a2410 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 10 monclient: _renew_subs 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f716013abd0 con 0x7f71600a2410 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f714c0047e0 con 0x7f71600a2410 2024-09-16T04:12:30.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f714c004b30 con 0x7f71600a2410 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715effd640 4 set_mon_vals no callback set 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.233+0000 7f715d7fa640 10 monclient: dump: 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.259 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.234+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f714c00fdc0 con 0x7f71600a2410 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.234+0000 7f7166c2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.234+0000 7f7166c2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.234+0000 7f7166c2c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.234+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7140005080 con 0x7f71600a2410 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.235+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(234..234 src has 1..234) ==== 10124+0+0 (unknown 293951658 0 0) 0x7f714c0484d0 con 0x7f71600a2410 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.235+0000 7f71649c8640 10 monclient: get_auth_request con 0x7f7144039260 auth_method 0 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.239+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f714c015a00 con 0x7f71600a2410 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.239+0000 7f715d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.239+0000 7f715d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:30.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.385+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v235) ==== 120+0+0 (unknown 2095629348 0 0) 0x7fdd90015780 con 0x7fdd98132e60 2024-09-16T04:12:30.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.385+0000 7fdd8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T04:12:30.387 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.385+0000 7fdd8a7fc640 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-16T04:12:30.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f7166c2c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:12:30.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f7166c2c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:12:30.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f7166c2c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:30.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7140005b80 con 0x7f71600a2410 2024-09-16T04:12:30.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f715d7fa640 1 -- 172.21.15.148:0/448743089 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f714c015300 con 0x7f71600a2410 2024-09-16T04:12:30.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f715d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:12:30.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.474+0000 7f715d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:30.477 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.475+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.478 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.475+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=10,logm=63} crc {auth=595653432,config=692619927,health=1288994568,kv=2270513034,logm=3965852875}) 2024-09-16T04:12:30.478 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.475+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=8,logm=66} crc {auth=2390303440,config=692619927,health=1321841627,kv=4215109598,logm=1665467715}) 2024-09-16T04:12:30.478 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.475+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.478 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.475+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=10,logm=63} crc {auth=595653432,config=692619927,health=1288994568,kv=2270513034,logm=3965852875}) 2024-09-16T04:12:30.478 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.475+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=10,logm=63} crc {auth=595653432,config=692619927,health=2290190625,kv=2270513034,logm=3445538263}) 2024-09-16T04:12:30.479 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.479 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421199900}) 2024-09-16T04:12:30.479 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=460279906}) 2024-09-16T04:12:30.479 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.479 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421199900}) 2024-09-16T04:12:30.479 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3262325101}) 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 >> v1:172.21.15.171:6812/7327273 conn(0x7f7144039260 legacy=0x7f714403b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.477+0000 7f7166c2c640 10 monclient: shutdown 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f7166c2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 >> v1:172.21.15.148:6789/0 conn(0x7f71600a2410 legacy=0x7f71601374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 shutdown_connections 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 >> 172.21.15.148:0/448743089 conn(0x7f71600a97b0 msgr2=0x7f71600a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:30.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 shutdown_connections 2024-09-16T04:12:30.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f7166c2c640 1 -- 172.21.15.148:0/448743089 wait complete. 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613215050}) 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1343958919}) 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1613215050}) 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.478+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691042198}) 2024-09-16T04:12:30.482 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.479+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.479+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865796080}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.479+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837362152}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.479+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.479+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865796080}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.479+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063142856}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.480+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.480+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115090329}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.480+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=831053840}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.480+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.480+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115090329}) 2024-09-16T04:12:30.483 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.480+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697457668}) 2024-09-16T04:12:30.484 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.481+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.484 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.481+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504066633}) 2024-09-16T04:12:30.484 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.481+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523224945}) 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.481+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.481+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504066633}) 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.481+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825208731}) 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.482+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.482+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986814323}) 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.482+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700055850}) 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.482+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.482+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986814323}) 2024-09-16T04:12:30.485 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.482+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862106337}) 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.483+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.483+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006228037}) 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.483+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=842210638}) 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.483+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.483+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006228037}) 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.483+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2806637823}) 2024-09-16T04:12:30.486 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.484+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.487 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.484+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143123643}) 2024-09-16T04:12:30.487 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.484+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1999150805}) 2024-09-16T04:12:30.487 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.484+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.487 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.484+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143123643}) 2024-09-16T04:12:30.487 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.484+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323586514}) 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.485+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.485+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985122480}) 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.485+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529489910}) 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.485+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.485+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985122480}) 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.485+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2409196228}) 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.486+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.488 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.486+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763337270}) 2024-09-16T04:12:30.489 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.486+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836116797}) 2024-09-16T04:12:30.489 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.486+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.489 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.486+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763337270}) 2024-09-16T04:12:30.489 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.486+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=221825768}) 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.487+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.487+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214658266}) 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.487+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3618946903}) 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.487+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.487+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214658266}) 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.487+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1217443011}) 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.488+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.490 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.488+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358427871}) 2024-09-16T04:12:30.491 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.488+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1207450618}) 2024-09-16T04:12:30.491 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.488+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.491 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.488+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358427871}) 2024-09-16T04:12:30.491 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.488+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712296599}) 2024-09-16T04:12:30.492 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:12:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.489+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.489+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005324784}) 2024-09-16T04:12:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.489+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835806774}) 2024-09-16T04:12:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.489+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.489+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005324784}) 2024-09-16T04:12:30.493 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.489+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703721179}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.490+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.490+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176206649}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.490+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2326138547}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.490+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.490+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176206649}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.490+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882391165}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.491+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.491+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633816026}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.491+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484856373}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.491+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.491+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633816026}) 2024-09-16T04:12:30.494 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.491+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=661997683}) 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.492+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.492+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870632876}) 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.492+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71576840}) 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.492+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.492+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870632876}) 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.492+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014289586}) 2024-09-16T04:12:30.496 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.493+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.493+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133552450}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.493+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025097814}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.493+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.493+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133552450}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.493+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1274647736}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.494+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.494+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412199262}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.494+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711388241}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.494+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.494+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412199262}) 2024-09-16T04:12:30.497 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.494+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184698127}) 2024-09-16T04:12:30.502 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.495+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.502 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.495+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643584200}) 2024-09-16T04:12:30.502 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.495+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304754925}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.495+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.495+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643584200}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.495+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339765461}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.496+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.496+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891440951}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.496+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781964342}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.496+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.496+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891440951}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.496+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170281050}) 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.497+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.503 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.497+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547867341}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.497+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3743010561}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.497+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.497+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547867341}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.497+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457182515}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.498+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.498+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763341896}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.498+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3459268346}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.498+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.498+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763341896}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.498+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=478745225}) 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.499+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.504 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.499+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565419706}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.499+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119348257}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.499+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.499+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565419706}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.499+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797589792}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.500+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.500+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=217839107}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.500+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4231886406}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.500+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.500+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=217839107}) 2024-09-16T04:12:30.505 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.500+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254162900}) 2024-09-16T04:12:30.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.502+0000 7fdda0077640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.502+0000 7fdda0077640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.502+0000 7fdda0077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.502+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) -- 0x7fdd80002210 con 0x7fdd98132e60 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.504+0000 7fdd8a7fc640 1 -- 172.21.15.171:0/3029724566 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v235) ==== 120+0+0 (unknown 985644766 0 0) 0x7fdd90019ad0 con 0x7fdd98132e60 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.504+0000 7fdd8a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.504+0000 7fdd8a7fc640 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-16T04:12:30.507 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h20' does not appear in the crush map 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.501+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.501+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454235688}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.501+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1631880817}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.501+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.501+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454235688}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.501+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230629291}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.502+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.502+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070616764}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.502+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978846905}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.502+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.502+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070616764}) 2024-09-16T04:12:30.511 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.502+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3366914888}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358148443}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960585105}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358148443}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1960669754}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460006095}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500674980}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460006095}) 2024-09-16T04:12:30.512 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.503+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=257570762}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.504+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.504+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101034258}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.504+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672787724}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.504+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.504+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101034258}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.504+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020300716}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.505+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.505+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237493889}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.505+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491881683}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.505+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.505+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237493889}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.505+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3211469017}) 2024-09-16T04:12:30.513 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.506+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.506+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=3681229241,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=558787501,mon_config_key=4052540250,monmap=2633035914,osd_metadata=4007021953}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.506+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=3183705486,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=558787501,mon_config_key=718393118,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=2038605090,osd_snap=3823785925}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.506+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.506+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6} crc {logm=3681229241,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=558787501,mon_config_key=4052540250,monmap=2633035914,osd_metadata=4007021953}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.506+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1} crc {logm=2498024766,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=558787501,mon_config_key=888502937,monmap=2633035914,osd_metadata=1814933226,osd_pg_creating=2038605090,osd_snap=3823785925}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.508+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.508+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_pg_creating=1,osd_snap=2,osdmap=97} crc {osd_pg_creating=2038605090,osd_snap=841190635,osdmap=3505215673}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.508+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_pg_creating=1,osd_snap=2,osdmap=97} crc {osd_pg_creating=2038605090,osd_snap=841190635,osdmap=2716590636}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.508+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.508+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_pg_creating=1,osd_snap=2,osdmap=97} crc {osd_pg_creating=2038605090,osd_snap=841190635,osdmap=3505215673}) 2024-09-16T04:12:30.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.508+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_pg_creating=1,osd_snap=2,osdmap=97} crc {osd_pg_creating=2038605090,osd_snap=841190635,osdmap=442248898}) 2024-09-16T04:12:30.515 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.509+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.515 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.509+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=717239224}) 2024-09-16T04:12:30.515 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.509+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1996638937}) 2024-09-16T04:12:30.515 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.509+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.515 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.509+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=717239224}) 2024-09-16T04:12:30.515 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.509+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3056503534}) 2024-09-16T04:12:30.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.507+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 >> v1:172.21.15.171:6812/7327273 conn(0x7fdd7c039130 legacy=0x7fdd7c03b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.507+0000 7fdda0077640 10 monclient: shutdown 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.507+0000 7fdda0077640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.507+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 >> v1:172.21.15.171:6789/0 conn(0x7fdd98132e60 legacy=0x7fdd981cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.507+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 shutdown_connections 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.507+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 >> 172.21.15.171:0/3029724566 conn(0x7fdd980a52a0 msgr2=0x7fdd98131cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.508+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 shutdown_connections 2024-09-16T04:12:30.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.508+0000 7fdda0077640 1 -- 172.21.15.171:0/3029724566 wait complete. 2024-09-16T04:12:30.517 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.510+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.510+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=713166507}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.510+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2742400574}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.510+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.510+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=713166507}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.510+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=798614180}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.511+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.511+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=2584384476}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.511+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=936161183}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.511+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.511+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=2584384476}) 2024-09-16T04:12:30.518 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:12:30.511+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=50} crc {osdmap=208772217}) 2024-09-16T04:12:30.519 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:30.519 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h21 2024-09-16T04:12:30.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.597+0000 7fd1d5247640 10 monclient: get_monmap_and_config 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.597+0000 7fd1d5247640 10 monclient: build_initial_monmap 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.597+0000 7fd1d5247640 10 monclient: monmap: 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:30.597900+0000 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:30.597900+0000 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:30.599 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.597+0000 7fd1d5247640 1 Processor -- start 2024-09-16T04:12:30.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.597+0000 7fd1d5247640 1 -- start start 2024-09-16T04:12:30.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.597+0000 7fd1d5247640 10 monclient: init 2024-09-16T04:12:30.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:30.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:30.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d0136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:30.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d0139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d013d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d00a1340 con 0x7fd1d0136b60 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d0137650 con 0x7fd1d013d620 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d00a0aa0 con 0x7fd1d0139b10 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 10 monclient(hunting): authenticate will time out at 1126.103220s 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cf59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd1d013d620 0x7fd1d013fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59376/0 (socket says 172.21.15.171:59376) 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cf59e640 1 -- 172.21.15.171:0/2759502904 learned_addr learned my addr 172.21.15.171:0/2759502904 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213259220 0 0) 0x7fd1d0137650 con 0x7fd1d013d620 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): my global_id is 6301 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b8003520 con 0x7fd1d013d620 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108522975 0 0) 0x7fd1d00a1340 con 0x7fd1d0136b60 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): my global_id is 64734 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d0137650 con 0x7fd1d0136b60 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3780863005 0 0) 0x7fd1d00a0aa0 con 0x7fd1d0139b10 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient(hunting): my global_id is 74135 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d00a1340 con 0x7fd1d0139b10 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 380163059 0 0) 0x7fd1b8003520 con 0x7fd1d013d620 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 >> v1:172.21.15.148:6790/0 conn(0x7fd1d0139b10 legacy=0x7fd1d013bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 >> v1:172.21.15.148:6789/0 conn(0x7fd1d0136b60 legacy=0x7fd1d0136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 monclient: found mon.noname-c 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d00a0550 con 0x7fd1d013d620 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: _finish_auth 0 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:30.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1c8001990 con 0x7fd1d013d620 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 5 monclient: authenticate success, global_id 6301 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1cdd9b640 10 monclient: dump: 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.612 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.598+0000 7fd1d5247640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1c8001bf0 con 0x7fd1d013d620 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1cdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1cdd9b640 1 -- 172.21.15.171:0/2759502904 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1c8001fc0 con 0x7fd1d013d620 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 4 set_mon_vals no callback set 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1cdd9b640 10 monclient: dump: 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:30.613 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 10 monclient: shutdown 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 1 -- 172.21.15.171:0/2759502904 >> v1:172.21.15.171:6789/0 conn(0x7fd1d013d620 legacy=0x7fd1d013fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 1 -- 172.21.15.171:0/2759502904 shutdown_connections 2024-09-16T04:12:30.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 1 -- 172.21.15.171:0/2759502904 >> 172.21.15.171:0/2759502904 conn(0x7fd1d00a98b0 msgr2=0x7fd1d00a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 1 -- 172.21.15.171:0/2759502904 shutdown_connections 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.599+0000 7fd1d5247640 1 -- 172.21.15.171:0/2759502904 wait complete. 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.600+0000 7fd1d5247640 10 monclient: build_initial_monmap 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.600+0000 7fd1d5247640 10 monclient: monmap: 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:30.615 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.600+0000 7fd1d5247640 1 Processor -- start 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.600+0000 7fd1d5247640 1 -- start start 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.600+0000 7fd1d5247640 10 monclient: init 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d0136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d0139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d013d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d0120c00 con 0x7fd1d013d620 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d009f480 con 0x7fd1d0136b60 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d0120ee0 con 0x7fd1d0139b10 2024-09-16T04:12:30.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1d5247640 10 monclient(hunting): authenticate will time out at 1126.106378s 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1ced9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd1d0136b60 0x7fd1d01cba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59380/0 (socket says 172.21.15.171:59380) 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1ced9d640 1 -- 172.21.15.171:0/685040032 learned_addr learned my addr 172.21.15.171:0/685040032 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501241759 0 0) 0x7fd1d009f480 con 0x7fd1d0136b60 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): my global_id is 6304 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1bc003520 con 0x7fd1d0136b60 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 661340672 0 0) 0x7fd1d0120c00 con 0x7fd1d013d620 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): my global_id is 64737 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d009f480 con 0x7fd1d013d620 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 149857058 0 0) 0x7fd1d0120ee0 con 0x7fd1d0139b10 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:30.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): my global_id is 74138 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1d0120c00 con 0x7fd1d0139b10 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3273588000 0 0) 0x7fd1bc003520 con 0x7fd1d0136b60 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1d0120ee0 con 0x7fd1d0136b60 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1c4001970 con 0x7fd1d0136b60 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:30.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): dump: 2024-09-16T04:12:30.620 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:30.620 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.620 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.620 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.620 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 719389263 0 0) 0x7fd1d009f480 con 0x7fd1d013d620 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1bc003520 con 0x7fd1d013d620 2024-09-16T04:12:30.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1c80016f0 con 0x7fd1d013d620 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): dump: 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.622 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4113537679 0 0) 0x7fd1d0120c00 con 0x7fd1d0139b10 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1d009f480 con 0x7fd1d0139b10 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1c00018b0 con 0x7fd1d0139b10 2024-09-16T04:12:30.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.601+0000 7fd1b77fe640 10 monclient(hunting): dump: 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.625 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3962868406 0 0) 0x7fd1d0120ee0 con 0x7fd1d0136b60 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 >> v1:172.21.15.148:6790/0 conn(0x7fd1d0139b10 legacy=0x7fd1d01d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 >> v1:172.21.15.148:6789/0 conn(0x7fd1d013d620 legacy=0x7fd1d01d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 monclient: found mon.b 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d013d330 con 0x7fd1d0136b60 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 5 monclient: authenticate success, global_id 6304 2024-09-16T04:12:30.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 10 monclient: _renew_subs 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d013cf20 con 0x7fd1d0136b60 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 10 monclient: _renew_subs 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1d0121300 con 0x7fd1d0136b60 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1c40038c0 con 0x7fd1d0136b60 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1c4003af0 con 0x7fd1d0136b60 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1cd59a640 4 set_mon_vals no callback set 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 10 monclient: dump: 2024-09-16T04:12:30.627 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.602+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd1c4003af0 con 0x7fd1d0136b60 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.603+0000 7fd1d5247640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.603+0000 7fd1d5247640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.603+0000 7fd1d5247640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.603+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b0005080 con 0x7fd1d0136b60 2024-09-16T04:12:30.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.603+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(235..235 src has 1..235) ==== 9911+0+0 (unknown 1028469156 0 0) 0x7fd1c4048540 con 0x7fd1d0136b60 2024-09-16T04:12:30.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.603+0000 7fd1ce59c640 10 monclient: get_auth_request con 0x7fd1bc0390b0 auth_method 0 2024-09-16T04:12:30.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.607+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd1c4015bb0 con 0x7fd1d0136b60 2024-09-16T04:12:30.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.607+0000 7fd1b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:30.629 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.607+0000 7fd1b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:30.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.776+0000 7fd1d5247640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T04:12:30.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.776+0000 7fd1d5247640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T04:12:30.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.776+0000 7fd1d5247640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:30.778 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:30.776+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) -- 0x7fd1b0005370 con 0x7fd1d0136b60 2024-09-16T04:12:31.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.397+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v236) ==== 120+0+0 (unknown 122553328 0 0) 0x7fd1c40154b0 con 0x7fd1d0136b60 2024-09-16T04:12:31.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.397+0000 7fd1b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T04:12:31.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.397+0000 7fd1b77fe640 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-16T04:12:31.492 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:12:31.493 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:31.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.501+0000 7fd1d5247640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T04:12:31.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.501+0000 7fd1d5247640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T04:12:31.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.501+0000 7fd1d5247640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.503 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.501+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) -- 0x7fd1b0002210 con 0x7fd1d0136b60 2024-09-16T04:12:31.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.502+0000 7fd1b77fe640 1 -- 172.21.15.171:0/685040032 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v236) ==== 120+0+0 (unknown 2686244125 0 0) 0x7fd1c4015210 con 0x7fd1d0136b60 2024-09-16T04:12:31.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.502+0000 7fd1b77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-16T04:12:31.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.502+0000 7fd1b77fe640 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-16T04:12:31.504 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h21' does not appear in the crush map 2024-09-16T04:12:31.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 >> v1:172.21.15.171:6812/7327273 conn(0x7fd1bc0390b0 legacy=0x7fd1bc03b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 10 monclient: shutdown 2024-09-16T04:12:31.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:31.506 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 >> v1:172.21.15.171:6789/0 conn(0x7fd1d0136b60 legacy=0x7fd1d01cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 shutdown_connections 2024-09-16T04:12:31.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 >> 172.21.15.171:0/685040032 conn(0x7fd1d00a98b0 msgr2=0x7fd1d00a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:31.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 shutdown_connections 2024-09-16T04:12:31.507 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.504+0000 7fd1d5247640 1 -- 172.21.15.171:0/685040032 wait complete. 2024-09-16T04:12:31.515 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:31.515 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h22 2024-09-16T04:12:31.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.593+0000 7ff2e13fd640 10 monclient: get_monmap_and_config 2024-09-16T04:12:31.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.593+0000 7ff2e13fd640 10 monclient: build_initial_monmap 2024-09-16T04:12:31.606 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.593+0000 7ff2e13fd640 10 monclient: monmap: 2024-09-16T04:12:31.606 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:31.594071+0000 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:31.594071+0000 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.593+0000 7ff2e13fd640 1 Processor -- start 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.593+0000 7ff2e13fd640 1 -- start start 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.593+0000 7ff2e13fd640 10 monclient: init 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:31.607 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient(hunting): picked mon.noname-c con 0x7ff2dc136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient(hunting): picked mon.noname-b con 0x7ff2dc139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2dc13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc0a1340 con 0x7ff2dc13d620 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc137650 con 0x7ff2dc136b60 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc0a0aa0 con 0x7ff2dc139b10 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2e13fd640 10 monclient(hunting): authenticate will time out at 1127.099492s 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2daffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff2dc136b60 0x7ff2dc136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59388/0 (socket says 172.21.15.171:59388) 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2daffd640 1 -- 172.21.15.171:0/450935 learned_addr learned my addr 172.21.15.171:0/450935 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158035153 0 0) 0x7ff2dc137650 con 0x7ff2dc136b60 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): my global_id is 6307 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c4003520 con 0x7ff2dc136b60 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456852569 0 0) 0x7ff2dc0a0aa0 con 0x7ff2dc139b10 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 10 monclient(hunting): my global_id is 74141 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.594+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2dc137650 con 0x7ff2dc139b10 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075037669 0 0) 0x7ff2dc0a1340 con 0x7ff2dc13d620 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient(hunting): my global_id is 64740 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2dc0a0aa0 con 0x7ff2dc13d620 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2962066299 0 0) 0x7ff2c4003520 con 0x7ff2dc136b60 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 >> v1:172.21.15.148:6790/0 conn(0x7ff2dc139b10 legacy=0x7ff2dc13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 >> v1:172.21.15.148:6789/0 conn(0x7ff2dc13d620 legacy=0x7ff2dc13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 monclient: found mon.noname-c 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc0a0550 con 0x7ff2dc136b60 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:31.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 5 monclient: authenticate success, global_id 6307 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2cc001610 con 0x7ff2dc136b60 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: dump: 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.611 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2cc001930 con 0x7ff2dc136b60 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 1 -- 172.21.15.171:0/450935 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2cc001d00 con 0x7ff2dc136b60 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 4 set_mon_vals no callback set 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2d9ffb640 10 monclient: dump: 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.612 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 10 monclient: shutdown 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 1 -- 172.21.15.171:0/450935 >> v1:172.21.15.171:6789/0 conn(0x7ff2dc136b60 legacy=0x7ff2dc136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 1 -- 172.21.15.171:0/450935 shutdown_connections 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 1 -- 172.21.15.171:0/450935 >> 172.21.15.171:0/450935 conn(0x7ff2dc0a98b0 msgr2=0x7ff2dc0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:31.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 1 -- 172.21.15.171:0/450935 shutdown_connections 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.595+0000 7ff2e13fd640 1 -- 172.21.15.171:0/450935 wait complete. 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.596+0000 7ff2e13fd640 10 monclient: build_initial_monmap 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.596+0000 7ff2e13fd640 10 monclient: monmap: 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:31.614 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.596+0000 7ff2e13fd640 1 Processor -- start 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.596+0000 7ff2e13fd640 1 -- start start 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.596+0000 7ff2e13fd640 10 monclient: init 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2dc136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient(hunting): picked mon.noname-b con 0x7ff2dc139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient(hunting): picked mon.noname-c con 0x7ff2dc13d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc13d0b0 con 0x7ff2dc136b60 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc120fd0 con 0x7ff2dc139b10 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc0a2520 con 0x7ff2dc13d620 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:31.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2e13fd640 10 monclient(hunting): authenticate will time out at 1127.102273s 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2da7fc640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff2dc139b10 0x7ff2dc1d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:59398/0 (socket says 172.21.15.171:59398) 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2da7fc640 1 -- 172.21.15.171:0/2904111043 learned_addr learned my addr 172.21.15.171:0/2904111043 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3462861414 0 0) 0x7ff2dc120fd0 con 0x7ff2dc139b10 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): my global_id is 6310 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c8003550 con 0x7ff2dc139b10 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4116215471 0 0) 0x7ff2dc0a2520 con 0x7ff2dc13d620 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): my global_id is 74147 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2dc120fd0 con 0x7ff2dc13d620 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851310048 0 0) 0x7ff2dc0a0550 con 0x7ff2dc136b60 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): my global_id is 64746 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2dc0a2520 con 0x7ff2dc136b60 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2630071460 0 0) 0x7ff2c8003550 con 0x7ff2dc139b10 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2dc0a0550 con 0x7ff2dc139b10 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d00018f0 con 0x7ff2dc139b10 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): dump: 2024-09-16T04:12:31.617 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2761575546 0 0) 0x7ff2dc120fd0 con 0x7ff2dc13d620 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2c8003550 con 0x7ff2dc13d620 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d4001c30 con 0x7ff2dc13d620 2024-09-16T04:12:31.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 10 monclient(hunting): dump: 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 141739732 0 0) 0x7ff2dc0a0550 con 0x7ff2dc139b10 2024-09-16T04:12:31.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 >> v1:172.21.15.148:6790/0 conn(0x7ff2dc13d620 legacy=0x7ff2dc1d41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.597+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 >> v1:172.21.15.148:6789/0 conn(0x7ff2dc136b60 legacy=0x7ff2dc1358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 1 monclient: found mon.b 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc120cf0 con 0x7ff2dc139b10 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 5 monclient: authenticate success, global_id 6310 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 10 monclient: _renew_subs 2024-09-16T04:12:31.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2dc11f6a0 con 0x7ff2dc139b10 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 10 monclient: _renew_subs 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2dc136000 con 0x7ff2dc139b10 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff2d00035f0 con 0x7ff2dc139b10 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff2d0003940 con 0x7ff2dc139b10 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2d97fa640 4 set_mon_vals no callback set 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 10 monclient: dump: 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.621 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.598+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff2d0003940 con 0x7ff2dc139b10 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.599+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(236..236 src has 1..236) ==== 9698+0+0 (unknown 2642287693 0 0) 0x7ff2d0048340 con 0x7ff2dc139b10 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.599+0000 7ff2daffd640 10 monclient: get_auth_request con 0x7ff2c8038e10 auth_method 0 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.599+0000 7ff2e13fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.599+0000 7ff2e13fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.599+0000 7ff2e13fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.599+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2bc005080 con 0x7ff2dc139b10 2024-09-16T04:12:31.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.603+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff2d0015a40 con 0x7ff2dc139b10 2024-09-16T04:12:31.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.603+0000 7ff2c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.603+0000 7ff2c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.596+0000 7fc8ecec9640 10 monclient: get_monmap_and_config 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.596+0000 7fc8ecec9640 10 monclient: build_initial_monmap 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.596+0000 7fc8ecec9640 10 monclient: monmap: 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:31.597049+0000 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:31.597049+0000 2024-09-16T04:12:31.627 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.596+0000 7fc8ecec9640 1 Processor -- start 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.596+0000 7fc8ecec9640 1 -- start start 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.596+0000 7fc8ecec9640 10 monclient: init 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e80a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient(hunting): picked mon.noname-c con 0x7fc8e80a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:31.628 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient(hunting): picked mon.noname-a con 0x7fc8e813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e80a0e00 con 0x7fc8e813d060 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e80a10e0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e80a4970 con 0x7fc8e80a2410 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 10 monclient(hunting): authenticate will time out at 1172.222206s 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e659c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc8e80a2410 0x7fc8e80a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35678/0 (socket says 172.21.15.148:35678) 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e659c640 1 -- 172.21.15.148:0/2479992005 learned_addr learned my addr 172.21.15.148:0/2479992005 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578133161 0 0) 0x7fc8e80a10e0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): my global_id is 6313 2024-09-16T04:12:31.629 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8d4003520 con 0x7fc8e80a1b10 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007064934 0 0) 0x7fc8e80a0e00 con 0x7fc8e813d060 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): my global_id is 64743 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e80a10e0 con 0x7fc8e813d060 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31780031 0 0) 0x7fc8e80a4970 con 0x7fc8e80a2410 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient(hunting): my global_id is 74144 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e80a0e00 con 0x7fc8e80a2410 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1609479566 0 0) 0x7fc8e80a10e0 con 0x7fc8e813d060 2024-09-16T04:12:31.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 >> v1:172.21.15.148:6790/0 conn(0x7fc8e80a2410 legacy=0x7fc8e80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 >> v1:172.21.15.171:6789/0 conn(0x7fc8e80a1b10 legacy=0x7fc8e813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e806dfd0 con 0x7fc8e813d060 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc8e00018f0 con 0x7fc8e813d060 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8e5d9b640 10 monclient: dump: 2024-09-16T04:12:31.631 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 5 monclient: authenticate success, global_id 64743 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.597+0000 7fc8ecec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8e0001b50 con 0x7fc8e813d060 2024-09-16T04:12:31.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8e5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8e5d9b640 1 -- 172.21.15.148:0/2479992005 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc8e0001fa0 con 0x7fc8e813d060 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 4 set_mon_vals no callback set 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8e5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8e5d9b640 10 monclient: dump: 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.633 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 10 monclient: shutdown 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 1 -- 172.21.15.148:0/2479992005 >> v1:172.21.15.148:6789/0 conn(0x7fc8e813d060 legacy=0x7fc8e813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 1 -- 172.21.15.148:0/2479992005 shutdown_connections 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 1 -- 172.21.15.148:0/2479992005 >> 172.21.15.148:0/2479992005 conn(0x7fc8e80a97b0 msgr2=0x7fc8e80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 1 -- 172.21.15.148:0/2479992005 shutdown_connections 2024-09-16T04:12:31.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.598+0000 7fc8ecec9640 1 -- 172.21.15.148:0/2479992005 wait complete. 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.599+0000 7fc8ecec9640 10 monclient: build_initial_monmap 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.599+0000 7fc8ecec9640 10 monclient: monmap: 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.599+0000 7fc8ecec9640 1 Processor -- start 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.599+0000 7fc8ecec9640 1 -- start start 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.599+0000 7fc8ecec9640 10 monclient: init 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient(hunting): picked mon.noname-c con 0x7fc8e80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e80a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient(hunting): picked mon.noname-a con 0x7fc8e813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e8138030 con 0x7fc8e813d060 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e8137c10 con 0x7fc8e80a2410 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e813ccb0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 10 monclient(hunting): authenticate will time out at 1172.225300s 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8e659c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc8e80a1b10 0x7fc8e809ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35688/0 (socket says 172.21.15.148:35688) 2024-09-16T04:12:31.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8e659c640 1 -- 172.21.15.148:0/3532721465 learned_addr learned my addr 172.21.15.148:0/3532721465 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212160792 0 0) 0x7fc8e806dfd0 con 0x7fc8e813d060 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): my global_id is 64749 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8c8003520 con 0x7fc8e813d060 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131986384 0 0) 0x7fc8e813ccb0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): my global_id is 74150 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e806dfd0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1723585934 0 0) 0x7fc8c8003520 con 0x7fc8e813d060 2024-09-16T04:12:31.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8e813ccb0 con 0x7fc8e813d060 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc8e0001ea0 con 0x7fc8e813d060 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): dump: 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.639 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2000806420 0 0) 0x7fc8e806dfd0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc8c8003520 con 0x7fc8e80a1b10 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc8d80017b0 con 0x7fc8e80a1b10 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient(hunting): dump: 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.640 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1437645503 0 0) 0x7fc8e813ccb0 con 0x7fc8e813d060 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 >> v1:172.21.15.148:6790/0 conn(0x7fc8e80a1b10 legacy=0x7fc8e809ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 >> v1:172.21.15.171:6789/0 conn(0x7fc8e80a2410 legacy=0x7fc8e809d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 monclient: found mon.a 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e80a0130 con 0x7fc8e813d060 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:31.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.600+0000 7fc8ecec9640 5 monclient: authenticate success, global_id 64749 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8ecec9640 10 monclient: _renew_subs 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8ecec9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e8137930 con 0x7fc8e813d060 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8ecec9640 10 monclient: _renew_subs 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8ecec9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc8e809f590 con 0x7fc8e813d060 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc8e0001b50 con 0x7fc8e813d060 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc8e0003e60 con 0x7fc8e813d060 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 10 monclient: dump: 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8e559a640 4 set_mon_vals no callback set 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.601+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc8e000f290 con 0x7fc8e813d060 2024-09-16T04:12:31.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.602+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(236..236 src has 1..236) ==== 9698+0+0 (unknown 2642287693 0 0) 0x7fc8e0047560 con 0x7fc8e813d060 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.602+0000 7fc8e659c640 10 monclient: get_auth_request con 0x7fc8c8038f40 auth_method 0 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.602+0000 7fc8ecec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.602+0000 7fc8ecec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.602+0000 7fc8ecec9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.602+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc005080 con 0x7fc8e813d060 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.607+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc8e8138030 con 0x7fc8e813d060 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.607+0000 7fc8df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.607+0000 7fc8df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:31.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.772+0000 7ff2e13fd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T04:12:31.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.772+0000 7ff2e13fd640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T04:12:31.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.772+0000 7ff2e13fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:31.774 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:31.772+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) -- 0x7ff2bc005370 con 0x7ff2dc139b10 2024-09-16T04:12:31.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8ecec9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:31.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8ecec9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:31.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8ecec9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc8cc005370 con 0x7fc8e813d060 2024-09-16T04:12:31.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8df7fe640 1 -- 172.21.15.148:0/3532721465 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3713494085) 0x7fc8e0014da0 con 0x7fc8e813d060 2024-09-16T04:12:31.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:31.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.835+0000 7fc8df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:31.838 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 >> v1:172.21.15.171:6812/7327273 conn(0x7fc8c8038f40 legacy=0x7fc8c803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 10 monclient: shutdown 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 >> v1:172.21.15.148:6789/0 conn(0x7fc8e813d060 legacy=0x7fc8e81d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 shutdown_connections 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 >> 172.21.15.148:0/3532721465 conn(0x7fc8e80a97b0 msgr2=0x7fc8e8135880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:31.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 shutdown_connections 2024-09-16T04:12:31.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.837+0000 7fc8ecec9640 1 -- 172.21.15.148:0/3532721465 wait complete. 2024-09-16T04:12:31.850 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:12:31.850 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:12:31.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:12:31.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:12:31.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.946+0000 7fa616627640 10 monclient: get_monmap_and_config 2024-09-16T04:12:31.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.946+0000 7fa616627640 10 monclient: build_initial_monmap 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.946+0000 7fa616627640 10 monclient: monmap: 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:31.947811+0000 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:31.947811+0000 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:31.949 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.947+0000 7fa616627640 1 Processor -- start 2024-09-16T04:12:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.947+0000 7fa616627640 1 -- start start 2024-09-16T04:12:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.947+0000 7fa616627640 10 monclient: init 2024-09-16T04:12:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:31.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient(hunting): picked mon.noname-b con 0x7fa61009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient(hunting): picked mon.noname-c con 0x7fa61009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient(hunting): picked mon.noname-a con 0x7fa61009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6100a1010 con 0x7fa61009cc30 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61009f030 con 0x7fa61009c660 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61009dc60 con 0x7fa61009bb70 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 10 monclient(hunting): authenticate will time out at 1172.573206s 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60ffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa61009bb70 0x7fa61009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35690/0 (socket says 172.21.15.148:35690) 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60ffff640 1 -- 172.21.15.148:0/3261452683 learned_addr learned my addr 172.21.15.148:0/3261452683 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa614bc4640 1 --1- 172.21.15.148:0/3261452683 >> v1:172.21.15.148:6789/0 conn(0x7fa61009cc30 0x7fa61013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40460/0 (socket says 172.21.15.148:40460) 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2072214840 0 0) 0x7fa61009dc60 con 0x7fa61009bb70 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): my global_id is 74153 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f8003520 con 0x7fa61009bb70 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818814393 0 0) 0x7fa6100a1010 con 0x7fa61009cc30 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): my global_id is 64752 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61009dc60 con 0x7fa61009cc30 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672176570 0 0) 0x7fa61009f030 con 0x7fa61009c660 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient(hunting): my global_id is 6316 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6100a1010 con 0x7fa61009c660 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 876663401 0 0) 0x7fa61009dc60 con 0x7fa61009cc30 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 >> v1:172.21.15.148:6790/0 conn(0x7fa61009bb70 legacy=0x7fa61009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 >> v1:172.21.15.171:6789/0 conn(0x7fa61009c660 legacy=0x7fa610138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 monclient: found mon.noname-a 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61009df40 con 0x7fa61009cc30 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: _finish_auth 0 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa608001970 con 0x7fa61009cc30 2024-09-16T04:12:31.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 5 monclient: authenticate success, global_id 64752 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa60effd640 10 monclient: dump: 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.962 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.948+0000 7fa616627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa608001be0 con 0x7fa61009cc30 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa60effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa60effd640 1 -- 172.21.15.148:0/3261452683 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa608001ff0 con 0x7fa61009cc30 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa60effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 4 set_mon_vals no callback set 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa60effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa60effd640 10 monclient: dump: 2024-09-16T04:12:31.963 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 10 monclient: shutdown 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 1 -- 172.21.15.148:0/3261452683 >> v1:172.21.15.148:6789/0 conn(0x7fa61009cc30 legacy=0x7fa61013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 1 -- 172.21.15.148:0/3261452683 shutdown_connections 2024-09-16T04:12:31.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 1 -- 172.21.15.148:0/3261452683 >> 172.21.15.148:0/3261452683 conn(0x7fa6100a5640 msgr2=0x7fa6100a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 1 -- 172.21.15.148:0/3261452683 shutdown_connections 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.949+0000 7fa616627640 1 -- 172.21.15.148:0/3261452683 wait complete. 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient: build_initial_monmap 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient: monmap: 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:31.965 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 1 Processor -- start 2024-09-16T04:12:31.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 1 -- start start 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient: init 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient(hunting): picked mon.noname-b con 0x7fa61009bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient(hunting): picked mon.noname-a con 0x7fa61009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient(hunting): picked mon.noname-c con 0x7fa61009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61011c420 con 0x7fa61009c660 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61011c700 con 0x7fa61009bb70 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61011ccc0 con 0x7fa61009cc30 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:31.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.950+0000 7fa616627640 10 monclient(hunting): authenticate will time out at 1172.575997s 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa614bc4640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa61009cc30 0x7fa6101d2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35706/0 (socket says 172.21.15.148:35706) 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa614bc4640 1 -- 172.21.15.148:0/729391278 learned_addr learned my addr 172.21.15.148:0/729391278 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60f7fe640 1 --1- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6789/0 conn(0x7fa61009c660 0x7fa6101ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40464/0 (socket says 172.21.15.148:40464) 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799079564 0 0) 0x7fa61011ccc0 con 0x7fa61009cc30 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): my global_id is 74156 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5fc003550 con 0x7fa61009cc30 2024-09-16T04:12:31.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1350339901 0 0) 0x7fa61011c420 con 0x7fa61009c660 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): my global_id is 64755 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61011ccc0 con 0x7fa61009c660 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896691686 0 0) 0x7fa61011c700 con 0x7fa61009bb70 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): my global_id is 6319 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61011c420 con 0x7fa61009bb70 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3513291126 0 0) 0x7fa5fc003550 con 0x7fa61009cc30 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61011c700 con 0x7fa61009cc30 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa6080018b0 con 0x7fa61009cc30 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): dump: 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.970 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.971 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 542352387 0 0) 0x7fa61011ccc0 con 0x7fa61009c660 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5fc003550 con 0x7fa61009c660 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3305584895 0 0) 0x7fa61011c420 con 0x7fa61009bb70 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61011ccc0 con 0x7fa61009bb70 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa604001830 con 0x7fa61009c660 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): dump: 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.972 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa600001110 con 0x7fa61009bb70 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:31.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient(hunting): dump: 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1884276113 0 0) 0x7fa61011c700 con 0x7fa61009cc30 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.171:6789/0 conn(0x7fa61009bb70 legacy=0x7fa6101c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6789/0 conn(0x7fa61009c660 legacy=0x7fa6101ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 monclient: found mon.c 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61013c9c0 con 0x7fa61009cc30 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: _finish_auth 0 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 5 monclient: authenticate success, global_id 74156 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 10 monclient: _renew_subs 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6100740a0 con 0x7fa61009cc30 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 10 monclient: _renew_subs 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:31.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa61013b470 con 0x7fa61009cc30 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa608004f00 con 0x7fa61009cc30 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa6080050b0 con 0x7fa61009cc30 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60e7fc640 4 set_mon_vals no callback set 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.951+0000 7fa60cff9640 10 monclient: dump: 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.976 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa6080050b0 con 0x7fa61009cc30 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa616627640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5f00065c0 con 0x7fa5f0002860 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(236..236 src has 1..236) ==== 9698+0+0 (unknown 2642287693 0 0) 0x7fa608048770 con 0x7fa61009cc30 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4267361519 0 0) 0x7fa5f00065c0 con 0x7fa5f0002860 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60f7fe640 10 monclient: get_auth_request con 0x7fa5fc0391b0 auth_method 0 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 10 monclient(hunting): my global_id is 64758 2024-09-16T04:12:31.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.952+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61011c700 con 0x7fa5f0002860 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3665342267 0 0) 0x7fa61011c700 con 0x7fa5f0002860 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f00065c0 con 0x7fa5f0002860 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa6000047e0 con 0x7fa5f0002860 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 10 monclient: dump: 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.978 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4167191460 0 0) 0x7fa5f00065c0 con 0x7fa5f0002860 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5f000a150 con 0x7fa5f0002860 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa600006ac0 con 0x7fa5f0002860 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6789/0 conn(0x7fa5f0002860 legacy=0x7fa5f0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.953+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6789/0 conn(0x7fa5f0002860 legacy=0x7fa5f0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa616627640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5f0009e40 con 0x7fa5f0009100 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 439510113 0 0) 0x7fa5f0009e40 con 0x7fa5f0009100 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa60cff9640 10 monclient(hunting): my global_id is 64761 2024-09-16T04:12:31.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.959+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5f00065c0 con 0x7fa5f0009100 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 562468488 0 0) 0x7fa5f00065c0 con 0x7fa5f0009100 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5f0009e40 con 0x7fa5f0009100 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa600006030 con 0x7fa5f0009100 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 10 monclient: dump: 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:31.982 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2716657081 0 0) 0x7fa5f0009e40 con 0x7fa5f0009100 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5f000c3d0 con 0x7fa5f0009100 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4146 (unknown 0 0 1154253591) 0x7fa600005fa0 con 0x7fa5f0009100 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6789/0 conn(0x7fa5f0009100 legacy=0x7fa5f0009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa60cff9640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6789/0 conn(0x7fa5f0009100 legacy=0x7fa5f0009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:31.983 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 28, 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 131158, 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:31.984 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:31.985 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:31.986 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:31.987 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:31.988 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.989 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:31.990 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:31.991 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:31.992 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:31.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:31.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:31.995 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:31.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.171:6812/7327273 conn(0x7fa5fc0391b0 legacy=0x7fa5fc03b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa616627640 10 monclient: shutdown 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa616627640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 >> v1:172.21.15.148:6790/0 conn(0x7fa61009cc30 legacy=0x7fa6101d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa614bc4640 1 -- 172.21.15.148:0/729391278 reap_dead start 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa614bc4640 1 -- 172.21.15.148:0/729391278 reap_dead start 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 shutdown_connections 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.960+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 >> 172.21.15.148:0/729391278 conn(0x7fa6100a5640 msgr2=0x7fa610072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.961+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 shutdown_connections 2024-09-16T04:12:31.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:31.961+0000 7fa616627640 1 -- 172.21.15.148:0/729391278 wait complete. 2024-09-16T04:12:31.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:31.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:12:31.998 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:12:32.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.094+0000 7f8b3ff98640 10 monclient: get_monmap_and_config 2024-09-16T04:12:32.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.094+0000 7f8b3ff98640 10 monclient: build_initial_monmap 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.094+0000 7f8b3ff98640 10 monclient: monmap: 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:32.095807+0000 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:32.095807+0000 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:32.097 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.095+0000 7f8b3ff98640 1 Processor -- start 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.095+0000 7f8b3ff98640 1 -- start start 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.095+0000 7f8b3ff98640 10 monclient: init 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient(hunting): picked mon.noname-c con 0x7f8b3813af10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient(hunting): picked mon.noname-b con 0x7f8b3809d810 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient(hunting): picked mon.noname-a con 0x7f8b3809c220 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b3813ba00 con 0x7f8b3809c220 2024-09-16T04:12:32.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b380a0fd0 con 0x7f8b3813af10 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b3809cb50 con 0x7f8b3809d810 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 10 monclient(hunting): authenticate will time out at 1172.721238s 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3e535640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8b3809c220 0x7f8b38143530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40484/0 (socket says 172.21.15.148:40484) 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3e535640 1 -- 172.21.15.148:0/745310877 learned_addr learned my addr 172.21.15.148:0/745310877 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1642879943 0 0) 0x7f8b3813ba00 con 0x7f8b3809c220 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): my global_id is 64764 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b24003520 con 0x7f8b3809c220 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067462016 0 0) 0x7f8b3809cb50 con 0x7f8b3809d810 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): my global_id is 74159 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b3813ba00 con 0x7f8b3809d810 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417282463 0 0) 0x7f8b380a0fd0 con 0x7f8b3813af10 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient(hunting): my global_id is 6322 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b3809cb50 con 0x7f8b3813af10 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1800310565 0 0) 0x7f8b3813ba00 con 0x7f8b3809d810 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 >> v1:172.21.15.171:6789/0 conn(0x7f8b3813af10 legacy=0x7f8b3813b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 >> v1:172.21.15.148:6789/0 conn(0x7f8b3809c220 legacy=0x7f8b38143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: _finish_hunting 0 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 monclient: found mon.noname-b 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3809ce30 con 0x7f8b3809d810 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: _finish_auth 0 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: _check_auth_tickets 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b280014c0 con 0x7f8b3809d810 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 5 monclient: authenticate success, global_id 74159 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 10 monclient: dump: 2024-09-16T04:12:32.109 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3ff98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.096+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b280016f0 con 0x7f8b3809d810 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3cd32640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:32.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3cd32640 1 -- 172.21.15.148:0/745310877 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b28001b00 con 0x7f8b3809d810 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3cd32640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 4 set_mon_vals no callback set 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3cd32640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3cd32640 10 monclient: dump: 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 10 monclient: shutdown 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 1 -- 172.21.15.148:0/745310877 >> v1:172.21.15.148:6790/0 conn(0x7f8b3809d810 legacy=0x7f8b3813fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 1 -- 172.21.15.148:0/745310877 shutdown_connections 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 1 -- 172.21.15.148:0/745310877 >> 172.21.15.148:0/745310877 conn(0x7f8b380a5640 msgr2=0x7f8b380a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 1 -- 172.21.15.148:0/745310877 shutdown_connections 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.097+0000 7f8b3ff98640 1 -- 172.21.15.148:0/745310877 wait complete. 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient: build_initial_monmap 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient: monmap: 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:32.112 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 1 Processor -- start 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 1 -- start start 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient: init 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient(hunting): picked mon.noname-a con 0x7f8b3809c220 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient(hunting): picked mon.noname-b con 0x7f8b3809d810 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:32.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient(hunting): picked mon.noname-c con 0x7f8b3813af10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b38136050 con 0x7f8b3809c220 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b381370f0 con 0x7f8b3809d810 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b38072d30 con 0x7f8b3813af10 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3ff98640 10 monclient(hunting): authenticate will time out at 1172.723969s 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.098+0000 7f8b3dd34640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8b3809c220 0x7f8b38135230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40500/0 (socket says 172.21.15.148:40500) 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3dd34640 1 -- 172.21.15.148:0/1623342233 learned_addr learned my addr 172.21.15.148:0/1623342233 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992064918 0 0) 0x7f8b38072d30 con 0x7f8b3813af10 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): my global_id is 74162 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b1c003520 con 0x7f8b3813af10 2024-09-16T04:12:32.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 442009330 0 0) 0x7f8b38136050 con 0x7f8b3809c220 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): my global_id is 64767 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b38072d30 con 0x7f8b3809c220 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295431039 0 0) 0x7f8b381370f0 con 0x7f8b3809d810 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): my global_id is 6325 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b38136050 con 0x7f8b3809d810 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2241178575 0 0) 0x7f8b1c003520 con 0x7f8b3813af10 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b381370f0 con 0x7f8b3813af10 2024-09-16T04:12:32.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b30001490 con 0x7f8b3813af10 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 523788975 0 0) 0x7f8b38072d30 con 0x7f8b3809c220 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b1c003520 con 0x7f8b3809c220 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3834124601 0 0) 0x7f8b38136050 con 0x7f8b3809d810 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b38072d30 con 0x7f8b3809d810 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b34001940 con 0x7f8b3809c220 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.117 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b28001a40 con 0x7f8b3809d810 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.118 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1271455573 0 0) 0x7f8b381370f0 con 0x7f8b3813af10 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.171:6789/0 conn(0x7f8b3809d810 legacy=0x7f8b38135940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.148:6789/0 conn(0x7f8b3809c220 legacy=0x7f8b38135230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 monclient: found mon.c 2024-09-16T04:12:32.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b381446a0 con 0x7f8b3813af10 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 5 monclient: authenticate success, global_id 74162 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 10 monclient: _renew_subs 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b380740a0 con 0x7f8b3813af10 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 10 monclient: _renew_subs 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b38138f40 con 0x7f8b3813af10 2024-09-16T04:12:32.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b30004b60 con 0x7f8b3813af10 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b30004eb0 con 0x7f8b3813af10 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2ffff640 4 set_mon_vals no callback set 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.099+0000 7f8b2e7fc640 10 monclient: dump: 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.121 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8b300102e0 con 0x7f8b3813af10 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b3ff98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b200065c0 con 0x7f8b20002860 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(236..236 src has 1..236) ==== 9698+0+0 (unknown 2642287693 0 0) 0x7f8b30048760 con 0x7f8b3813af10 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932682003 0 0) 0x7f8b200065c0 con 0x7f8b20002860 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b3d533640 10 monclient: get_auth_request con 0x7f8b1c039220 auth_method 0 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 10 monclient(hunting): my global_id is 74165 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.100+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b381370f0 con 0x7f8b20002860 2024-09-16T04:12:32.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3696156793 0 0) 0x7f8b381370f0 con 0x7f8b20002860 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b200065c0 con 0x7f8b20002860 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b34003ae0 con 0x7f8b20002860 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 10 monclient: dump: 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.123 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1846609631 0 0) 0x7f8b200065c0 con 0x7f8b20002860 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b2000a150 con 0x7f8b20002860 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8b34003eb0 con 0x7f8b20002860 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.148:6790/0 conn(0x7f8b20002860 legacy=0x7f8b20004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.101+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.148:6790/0 conn(0x7f8b20002860 legacy=0x7f8b20004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:32.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b3ff98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:32.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b20009e20 con 0x7f8b200090e0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063489648 0 0) 0x7f8b20009e20 con 0x7f8b200090e0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient(hunting): my global_id is 74168 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b200065c0 con 0x7f8b200090e0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2957481932 0 0) 0x7f8b200065c0 con 0x7f8b200090e0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b20009e20 con 0x7f8b200090e0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8b340057c0 con 0x7f8b200090e0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.107+0000 7f8b2e7fc640 10 monclient: dump: 2024-09-16T04:12:32.126 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1460127625 0 0) 0x7f8b20009e20 con 0x7f8b200090e0 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8b2000c390 con 0x7f8b200090e0 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 2881795700) 0x7f8b34004330 con 0x7f8b200090e0 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:32.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.148:6790/0 conn(0x7f8b200090e0 legacy=0x7f8b20009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b2e7fc640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.148:6790/0 conn(0x7f8b200090e0 legacy=0x7f8b20009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 28, 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 17708, 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:32.128 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:32.129 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:32.130 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:32.131 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:32.132 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.133 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:32.135 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:32.136 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:32.137 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:32.138 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:32.139 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:12:32.139 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.139 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.139 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.139 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:32.139 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.171:6812/7327273 conn(0x7f8b1c039220 legacy=0x7f8b1c03b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 10 monclient: shutdown 2024-09-16T04:12:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:32.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 >> v1:172.21.15.148:6790/0 conn(0x7f8b3813af10 legacy=0x7f8b381d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3e535640 1 -- 172.21.15.148:0/1623342233 reap_dead start 2024-09-16T04:12:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3e535640 1 -- 172.21.15.148:0/1623342233 reap_dead start 2024-09-16T04:12:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 shutdown_connections 2024-09-16T04:12:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 >> 172.21.15.148:0/1623342233 conn(0x7f8b380a5640 msgr2=0x7f8b38133ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 shutdown_connections 2024-09-16T04:12:32.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.108+0000 7f8b3ff98640 1 -- 172.21.15.148:0/1623342233 wait complete. 2024-09-16T04:12:32.141 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:32.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:12:32.141 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.238+0000 7f8e223c4640 10 monclient: get_monmap_and_config 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.238+0000 7f8e223c4640 10 monclient: build_initial_monmap 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.238+0000 7f8e223c4640 10 monclient: monmap: 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:32.239168+0000 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:32.239168+0000 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:32.241 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:32.241 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 1 Processor -- start 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 1 -- start start 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient: init 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:32.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c0a0e00 con 0x7f8e1c0a2410 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c0a10e0 con 0x7f8e1c13d060 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c0a4970 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e223c4640 10 monclient(hunting): authenticate will time out at 1172.864704s 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e1bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8e1c0a2410 0x7f8e1c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:51694/0 (socket says 172.21.15.148:51694) 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.239+0000 7f8e1bfff640 1 -- 172.21.15.148:0/2868727297 learned_addr learned my addr 172.21.15.148:0/2868727297 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1518872762 0 0) 0x7f8e1c0a0e00 con 0x7f8e1c0a2410 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): my global_id is 64770 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e04003520 con 0x7f8e1c0a2410 2024-09-16T04:12:32.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403663185 0 0) 0x7f8e1c0a4970 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): my global_id is 74171 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c0a0e00 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 117511695 0 0) 0x7f8e1c0a10e0 con 0x7f8e1c13d060 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient(hunting): my global_id is 6328 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c0a4970 con 0x7f8e1c13d060 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4223038827 0 0) 0x7f8e04003520 con 0x7f8e1c0a2410 2024-09-16T04:12:32.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 >> v1:172.21.15.148:6790/0 conn(0x7f8e1c0a1b10 legacy=0x7f8e1c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 >> v1:172.21.15.171:6789/0 conn(0x7f8e1c13d060 legacy=0x7f8e1c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 monclient: found mon.noname-a 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c06dfd0 con 0x7f8e1c0a2410 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: _finish_auth 0 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e0c001610 con 0x7f8e1c0a2410 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 5 monclient: authenticate success, global_id 64770 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: dump: 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e0c001870 con 0x7f8e1c0a2410 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:32.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 1 -- 172.21.15.148:0/2868727297 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e0c001c80 con 0x7f8e1c0a2410 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 4 set_mon_vals no callback set 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e1affd640 10 monclient: dump: 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.254 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 10 monclient: shutdown 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 1 -- 172.21.15.148:0/2868727297 >> v1:172.21.15.148:6789/0 conn(0x7f8e1c0a2410 legacy=0x7f8e1c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 1 -- 172.21.15.148:0/2868727297 shutdown_connections 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.240+0000 7f8e223c4640 1 -- 172.21.15.148:0/2868727297 >> 172.21.15.148:0/2868727297 conn(0x7f8e1c0a97b0 msgr2=0x7f8e1c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.241+0000 7f8e223c4640 1 -- 172.21.15.148:0/2868727297 shutdown_connections 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.241+0000 7f8e223c4640 1 -- 172.21.15.148:0/2868727297 wait complete. 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.241+0000 7f8e223c4640 10 monclient: build_initial_monmap 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.241+0000 7f8e223c4640 10 monclient: monmap: 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:32.255 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.241+0000 7f8e223c4640 1 Processor -- start 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 1 -- start start 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient: init 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:32.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c137ef0 con 0x7f8e1c13d060 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c13cc20 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c0a0130 con 0x7f8e1c0a2410 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e223c4640 10 monclient(hunting): authenticate will time out at 1172.867653s 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e1b7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8e1c0a2410 0x7f8e1c1d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33526/0 (socket says 172.21.15.148:33526) 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e1b7fe640 1 -- 172.21.15.148:0/4210994816 learned_addr learned my addr 172.21.15.148:0/4210994816 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1156695725 0 0) 0x7f8e1c0a0130 con 0x7f8e1c0a2410 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): my global_id is 74174 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e08003520 con 0x7f8e1c0a2410 2024-09-16T04:12:32.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4206489158 0 0) 0x7f8e1c137ef0 con 0x7f8e1c13d060 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 10 monclient(hunting): my global_id is 64773 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c0a0130 con 0x7f8e1c13d060 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.242+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1348734393 0 0) 0x7f8e1c13cc20 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): my global_id is 6331 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c137ef0 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 102955660 0 0) 0x7f8e1c0a0130 con 0x7f8e1c13d060 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e1c13cc20 con 0x7f8e1c13d060 2024-09-16T04:12:32.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e14001bf0 con 0x7f8e1c13d060 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): dump: 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1322134611 0 0) 0x7f8e08003520 con 0x7f8e1c0a2410 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e1c0a0130 con 0x7f8e1c0a2410 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e100019b0 con 0x7f8e1c0a2410 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): dump: 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.260 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2945624110 0 0) 0x7f8e1c137ef0 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e08003520 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e0c001bc0 con 0x7f8e1c0a1b10 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient(hunting): dump: 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.261 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 874569717 0 0) 0x7f8e1c13cc20 con 0x7f8e1c13d060 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.148:6790/0 conn(0x7f8e1c0a2410 legacy=0x7f8e1c1d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.171:6789/0 conn(0x7f8e1c0a1b10 legacy=0x7f8e1c1cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 monclient: found mon.a 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c137c10 con 0x7f8e1c13d060 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: _finish_auth 0 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:32.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 5 monclient: authenticate success, global_id 64773 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 10 monclient: _renew_subs 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1c09f2b0 con 0x7f8e1c13d060 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 10 monclient: _renew_subs 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e1c09de60 con 0x7f8e1c13d060 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e14003990 con 0x7f8e1c13d060 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e14003ce0 con 0x7f8e1c13d060 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e1a7fc640 4 set_mon_vals no callback set 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:32.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.243+0000 7f8e18ff9640 10 monclient: dump: 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8e1400ef70 con 0x7f8e1c13d060 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e223c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dfc0065c0 con 0x7f8dfc002860 2024-09-16T04:12:32.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(236..236 src has 1..236) ==== 9698+0+0 (unknown 2642287693 0 0) 0x7f8e14047560 con 0x7f8e1c13d060 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e1b7fe640 10 monclient: get_auth_request con 0x7f8e08039210 auth_method 0 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2565702863 0 0) 0x7f8dfc0065c0 con 0x7f8dfc002860 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient(hunting): my global_id is 6334 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c13cc20 con 0x7f8dfc002860 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 986745339 0 0) 0x7f8e1c13cc20 con 0x7f8dfc002860 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dfc0065c0 con 0x7f8dfc002860 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e0c004940 con 0x7f8dfc002860 2024-09-16T04:12:32.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.244+0000 7f8e18ff9640 10 monclient: dump: 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3566019770 0 0) 0x7f8dfc0065c0 con 0x7f8dfc002860 2024-09-16T04:12:32.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8dfc00a150 con 0x7f8dfc002860 2024-09-16T04:12:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8e0c004cb0 con 0x7f8dfc002860 2024-09-16T04:12:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.171:6789/0 conn(0x7f8dfc002860 legacy=0x7f8dfc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.245+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.171:6789/0 conn(0x7f8dfc002860 legacy=0x7f8dfc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.251+0000 7f8e223c4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.251+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8dfc009e20 con 0x7f8dfc0090e0 2024-09-16T04:12:32.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021865394 0 0) 0x7f8dfc009e20 con 0x7f8dfc0090e0 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient(hunting): my global_id is 6337 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dfc0065c0 con 0x7f8dfc0090e0 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2130503042 0 0) 0x7f8dfc0065c0 con 0x7f8dfc0090e0 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dfc009e20 con 0x7f8dfc0090e0 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e0c005270 con 0x7f8dfc0090e0 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient: dump: 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:32.269 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1447959678 0 0) 0x7f8dfc009e20 con 0x7f8dfc0090e0 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8dfc00c390 con 0x7f8dfc0090e0 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 1407709632) 0x7f8e0c0065d0 con 0x7f8dfc0090e0 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:32.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.171:6789/0 conn(0x7f8dfc0090e0 legacy=0x7f8dfc009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.252+0000 7f8e18ff9640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.171:6789/0 conn(0x7f8dfc0090e0 legacy=0x7f8dfc009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:32.271 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:32.271 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:32.271 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:32.271 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:12:32.271 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 28, 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 376290, 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:32.272 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:32.273 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:32.274 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:32.275 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:32.276 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:32.277 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:32.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.279 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:32.280 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.281 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:32.282 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:32.283 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:32.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.171:6812/7327273 conn(0x7f8e08039210 legacy=0x7f8e0803b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 10 monclient: shutdown 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 >> v1:172.21.15.148:6789/0 conn(0x7f8e1c13d060 legacy=0x7f8e1c1d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e20961640 1 -- 172.21.15.148:0/4210994816 reap_dead start 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e20961640 1 -- 172.21.15.148:0/4210994816 reap_dead start 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 shutdown_connections 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 >> 172.21.15.148:0/4210994816 conn(0x7f8e1c0a97b0 msgr2=0x7f8e1c0a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 shutdown_connections 2024-09-16T04:12:32.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:32.253+0000 7f8e223c4640 1 -- 172.21.15.148:0/4210994816 wait complete. 2024-09-16T04:12:32.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:32.286 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:12:32.286 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'b'] 2024-09-16T04:12:32.286 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:12:32.286 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:12:32.286 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:12:32.286 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:12:32.296 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:12:32.296 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T04:12:32.296 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T04:12:32.296 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:12:32.296 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:12:32.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:12:34.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:34.597+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:34.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:34.597+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:34.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:34.597+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:34.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:34.597+0000 7ff2c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:35.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:35.597+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:35.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:35.597+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:35.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:35.597+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:36.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:36.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:36.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:36.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:36.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:36.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:37.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:37.597+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:37.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:37.597+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:37.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:37.597+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:38.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:38.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:38.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:38.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:38.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:38.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:39.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:39.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:39.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:39.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:39.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:39.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:40.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:40.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:40.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:40.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:40.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:40.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:41.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:41.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:41.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:41.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:41.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:41.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:42.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:42.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:42.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:42.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:43.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:43.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:43.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:43.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:43.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:43.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:44.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:44.598+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:44.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:44.598+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:44.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:44.598+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:44.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:44.598+0000 7ff2c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:45.599+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:45.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:45.599+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:45.599+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:46.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:46.599+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:46.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:46.599+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:46.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:46.599+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:47.297 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T04:12:47.298 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T04:12:47.298 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:12:47.298 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:12:47.298 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:12:47.299 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.405+0000 7f9a8812c640 10 monclient: get_monmap_and_config 2024-09-16T04:12:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.405+0000 7f9a8812c640 10 monclient: build_initial_monmap 2024-09-16T04:12:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.405+0000 7f9a8812c640 10 monclient: monmap: 2024-09-16T04:12:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:47.407 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:47.406582+0000 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:47.406582+0000 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:47.408 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 1 Processor -- start 2024-09-16T04:12:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 1 -- start start 2024-09-16T04:12:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 10 monclient: init 2024-09-16T04:12:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:47.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 10 monclient(hunting): picked mon.noname-c con 0x7f9a8009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.406+0000 7f9a8812c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a800a1010 con 0x7f9a8009bb70 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a8812c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8009f030 con 0x7f9a8009cc30 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a8812c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8009dc60 con 0x7f9a8009c660 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a8812c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a8812c640 10 monclient(hunting): authenticate will time out at 1188.032039s 2024-09-16T04:12:47.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a856c7640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a80138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a85ec8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9a8009bb70 0x7f9a8009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41482/0 (socket says 172.21.15.148:41482) 2024-09-16T04:12:47.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.407+0000 7f9a85ec8640 1 -- 172.21.15.148:0/83136084 learned_addr learned my addr 172.21.15.148:0/83136084 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:47.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:47.599+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:47.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:47.599+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:47.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:47.599+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:47.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:47.608+0000 7f9a856c7640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a80138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:48.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:48.009+0000 7f9a856c7640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a80138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:48.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:48.599+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:48.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:48.599+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:48.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:48.599+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:48.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:48.810+0000 7f9a856c7640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a80138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:49.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:49.599+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:49.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:49.599+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:49.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:49.599+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:50.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): tick 2024-09-16T04:12:50.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:12:50.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:12:50.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:12:50.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a80138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.171:6789/0 conn(0x7f9a8009cc30 legacy=0x7f9a8013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6789/0 conn(0x7f9a8009bb70 legacy=0x7f9a8009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): picked mon.noname-a con 0x7f9a64009060 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): picked mon.noname-c con 0x7f9a8009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 -- 172.21.15.148:0/83136084 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8009df40 con 0x7f9a64009060 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 -- 172.21.15.148:0/83136084 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8009dc60 con 0x7f9a8009cc30 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 1 -- 172.21.15.148:0/83136084 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a640125b0 con 0x7f9a8009c660 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a777fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:50.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.406+0000 7f9a85ec8640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a64007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:50.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:50.600+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:50.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:50.600+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:50.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:50.600+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:50.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:50.607+0000 7f9a85ec8640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a64007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:51.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:51.008+0000 7f9a85ec8640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a64007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:51.600+0000 7ff2c27fc640 10 monclient: tick 2024-09-16T04:12:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:51.600+0000 7ff2c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:51.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:51.600+0000 7ff2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:51.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:51.809+0000 7f9a85ec8640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a64007920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.306+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1952948961 0 0) 0x7f9a8009df40 con 0x7f9a64009060 2024-09-16T04:12:52.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.306+0000 7f9a84ec6640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.306+0000 7f9a84ec6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.306+0000 7f9a84ec6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.306+0000 7f9a84ec6640 10 monclient(hunting): my global_id is 64776 2024-09-16T04:12:52.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.306+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c003520 con 0x7f9a64009060 2024-09-16T04:12:52.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 671345703 0 0) 0x7f9a6c003520 con 0x7f9a64009060 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a64007920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.171:6789/0 conn(0x7f9a8009cc30 legacy=0x7f9a6400ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 monclient: found mon.noname-a 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a64013860 con 0x7f9a64009060 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9a700041a0 con 0x7f9a64009060 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a8812c640 5 monclient: authenticate success, global_id 64776 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a8812c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: dump: 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.313 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a8812c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a70004470 con 0x7f9a64009060 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 1 -- 172.21.15.148:0/83136084 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9a70004840 con 0x7f9a64009060 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a84ec6640 10 monclient: dump: 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.314 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a8812c640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.307+0000 7f9a8812c640 4 set_mon_vals no callback set 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.308+0000 7f9a8812c640 10 monclient: shutdown 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.308+0000 7f9a8812c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.308+0000 7f9a8812c640 1 -- 172.21.15.148:0/83136084 >> v1:172.21.15.148:6789/0 conn(0x7f9a64009060 legacy=0x7f9a6400b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.308+0000 7f9a8812c640 1 -- 172.21.15.148:0/83136084 shutdown_connections 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.308+0000 7f9a8812c640 1 -- 172.21.15.148:0/83136084 >> 172.21.15.148:0/83136084 conn(0x7f9a800a5640 msgr2=0x7f9a800a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.309+0000 7f9a8812c640 1 -- 172.21.15.148:0/83136084 shutdown_connections 2024-09-16T04:12:52.315 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.309+0000 7f9a8812c640 1 -- 172.21.15.148:0/83136084 wait complete. 2024-09-16T04:12:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.312+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v237) ==== 120+0+0 (unknown 1674451769 0 0) 0x7ff2d00196f0 con 0x7ff2dc139b10 2024-09-16T04:12:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.312+0000 7ff2c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T04:12:52.316 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.312+0000 7ff2c37fe640 10 monclient: _finish_command 2 = system:0 device 'h22' does not appear in the crush map 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.309+0000 7f9a8812c640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.309+0000 7f9a8812c640 10 monclient: monmap: 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:52.320 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 1 Processor -- start 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 1 -- start start 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient: init 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient(hunting): picked mon.noname-b con 0x7f9a64009060 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient(hunting): picked mon.noname-c con 0x7f9a8009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a800a4110 con 0x7f9a8009cc30 2024-09-16T04:12:52.321 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a80126130 con 0x7f9a64009060 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a80126410 con 0x7f9a8009c660 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a856c7640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a801d09b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a8812c640 10 monclient(hunting): authenticate will time out at 1192.936806s 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a85ec8640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9a64009060 0x7f9a801cb9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:48958/0 (socket says 172.21.15.148:48958) 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.310+0000 7f9a85ec8640 1 -- 172.21.15.148:0/3777051657 learned_addr learned my addr 172.21.15.148:0/3777051657 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.314+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044890045 0 0) 0x7f9a8009dc60 con 0x7f9a8009cc30 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.314+0000 7f9a767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.314+0000 7f9a767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.314+0000 7f9a767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.314+0000 7f9a767fc640 10 monclient(hunting): my global_id is 64779 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.314+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6400acb0 con 0x7f9a8009cc30 2024-09-16T04:12:52.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330613577 0 0) 0x7f9a80126130 con 0x7f9a64009060 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): my global_id is 6340 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a8009dc60 con 0x7f9a64009060 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 377344979 0 0) 0x7f9a6400acb0 con 0x7f9a8009cc30 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9a80126130 con 0x7f9a8009cc30 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9a780019c0 con 0x7f9a8009cc30 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): dump: 2024-09-16T04:12:52.323 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2641389581 0 0) 0x7f9a80126130 con 0x7f9a8009cc30 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 >> v1:172.21.15.148:6790/0 conn(0x7f9a8009c660 legacy=0x7f9a801d09b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:52.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 >> v1:172.21.15.171:6789/0 conn(0x7f9a64009060 legacy=0x7f9a801cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 monclient: found mon.a 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a800740a0 con 0x7f9a8009cc30 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 5 monclient: authenticate success, global_id 64779 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 10 monclient: _renew_subs 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9a801269d0 con 0x7f9a8009cc30 2024-09-16T04:12:52.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 10 monclient: _renew_subs 2024-09-16T04:12:52.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.315+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9a80123d20 con 0x7f9a8009cc30 2024-09-16T04:12:52.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.317+0000 7f9a8812c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.317+0000 7f9a8812c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.317+0000 7f9a8812c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.317+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a801246e0 con 0x7f9a8009cc30 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9a78002080 con 0x7f9a8009cc30 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9a78003b10 con 0x7f9a8009cc30 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a767fc640 10 monclient: dump: 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.327 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.318+0000 7f9a77fff640 4 set_mon_vals no callback set 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.319+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9a78003cb0 con 0x7f9a8009cc30 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.319+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(237..237 src has 1..237) ==== 9485+0+0 (unknown 2815424696 0 0) 0x7f9a78047360 con 0x7f9a8009cc30 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.320+0000 7f9a85ec8640 10 monclient: get_auth_request con 0x7f9a64039950 auth_method 0 2024-09-16T04:12:52.328 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.324+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9a800a4110 con 0x7f9a8009cc30 2024-09-16T04:12:52.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.324+0000 7f9a767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.324+0000 7f9a767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:52.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.420+0000 7ff2e13fd640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T04:12:52.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.420+0000 7ff2e13fd640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T04:12:52.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.420+0000 7ff2e13fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.420+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) -- 0x7ff2bc002210 con 0x7ff2dc139b10 2024-09-16T04:12:52.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.421+0000 7ff2c37fe640 1 -- 172.21.15.171:0/2904111043 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v237) ==== 120+0+0 (unknown 1674451769 0 0) 0x7ff2d00459a0 con 0x7ff2dc139b10 2024-09-16T04:12:52.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.421+0000 7ff2c37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-16T04:12:52.423 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.421+0000 7ff2c37fe640 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-16T04:12:52.423 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h22' does not appear in the crush map 2024-09-16T04:12:52.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 >> v1:172.21.15.171:6812/7327273 conn(0x7ff2c8038e10 legacy=0x7ff2c803b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.425 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 10 monclient: shutdown 2024-09-16T04:12:52.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 >> v1:172.21.15.171:6789/0 conn(0x7ff2dc139b10 legacy=0x7ff2dc1d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 shutdown_connections 2024-09-16T04:12:52.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 >> 172.21.15.171:0/2904111043 conn(0x7ff2dc0a98b0 msgr2=0x7ff2dc13bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 shutdown_connections 2024-09-16T04:12:52.426 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.423+0000 7ff2e13fd640 1 -- 172.21.15.171:0/2904111043 wait complete. 2024-09-16T04:12:52.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:52.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h23 2024-09-16T04:12:52.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.512+0000 7f30bc07b640 10 monclient: get_monmap_and_config 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.512+0000 7f30bc07b640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.512+0000 7f30bc07b640 10 monclient: monmap: 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:52.513566+0000 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:52.513566+0000 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.515 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.516 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:52.516 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:52.516 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 1 Processor -- start 2024-09-16T04:12:52.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 1 -- start start 2024-09-16T04:12:52.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 10 monclient: init 2024-09-16T04:12:52.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:52.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 10 monclient(hunting): picked mon.noname-b con 0x7f30b4132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 10 monclient(hunting): picked mon.noname-c con 0x7f30b409ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.513+0000 7f30bc07b640 10 monclient(hunting): picked mon.noname-a con 0x7f30b4132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b41332b0 con 0x7f30b4132e60 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b4134310 con 0x7f30b409ec20 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b409d980 con 0x7f30b4132a80 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 10 monclient(hunting): authenticate will time out at 1148.019054s 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b9e17640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f30b4132a80 legacy=0x7f30b41334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b9616640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f30b409ec20 0x7f30b4133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34364/0 (socket says 172.21.15.171:34364) 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b9616640 1 -- 172.21.15.171:0/1704367774 learned_addr learned my addr 172.21.15.171:0/1704367774 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713723810 0 0) 0x7f30b4134310 con 0x7f30b409ec20 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): my global_id is 6343 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30a0003520 con 0x7f30b409ec20 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650282995 0 0) 0x7f30b41332b0 con 0x7f30b4132e60 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient(hunting): my global_id is 64782 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30b4134310 con 0x7f30b4132e60 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1842769042 0 0) 0x7f30a0003520 con 0x7f30b409ec20 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 >> v1:172.21.15.148:6790/0 conn(0x7f30b4132a80 legacy=0x7f30b41334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:52.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 >> v1:172.21.15.148:6789/0 conn(0x7f30b4132e60 legacy=0x7f30b413c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 monclient: found mon.noname-c 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b409dc60 con 0x7f30b409ec20 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f30b0001610 con 0x7f30b409ec20 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 5 monclient: authenticate success, global_id 6343 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: dump: 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.528 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30b00018c0 con 0x7f30b409ec20 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 1 -- 172.21.15.171:0/1704367774 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f30b0001c90 con 0x7f30b409ec20 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30bc07b640 4 set_mon_vals no callback set 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.514+0000 7f30b8e15640 10 monclient: dump: 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:52.529 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 10 monclient: shutdown 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 1 -- 172.21.15.171:0/1704367774 >> v1:172.21.15.171:6789/0 conn(0x7f30b409ec20 legacy=0x7f30b4133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 1 -- 172.21.15.171:0/1704367774 shutdown_connections 2024-09-16T04:12:52.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 1 -- 172.21.15.171:0/1704367774 >> 172.21.15.171:0/1704367774 conn(0x7f30b40a52a0 msgr2=0x7f30b40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 1 -- 172.21.15.171:0/1704367774 shutdown_connections 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 1 -- 172.21.15.171:0/1704367774 wait complete. 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.515+0000 7f30bc07b640 10 monclient: monmap: 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.531 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 1 Processor -- start 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 1 -- start start 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient: init 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient(hunting): picked mon.noname-a con 0x7f30b409ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient(hunting): picked mon.noname-b con 0x7f30b4132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient(hunting): picked mon.noname-c con 0x7f30b4132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b411c0b0 con 0x7f30b409ec20 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b411d230 con 0x7f30b4132a80 2024-09-16T04:12:52.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b411cb70 con 0x7f30b4132e60 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30bc07b640 10 monclient(hunting): authenticate will time out at 1148.021654s 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30b9616640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f30b4132a80 0x7f30b411b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34378/0 (socket says 172.21.15.171:34378) 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30b9616640 1 -- 172.21.15.171:0/3294025876 learned_addr learned my addr 172.21.15.171:0/3294025876 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30ba618640 1 -- 172.21.15.171:0/3294025876 >> v1:172.21.15.148:6790/0 conn(0x7f30b4132e60 legacy=0x7f30b41cf630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4074874530 0 0) 0x7f30b411d230 con 0x7f30b4132a80 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): my global_id is 6346 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3098003520 con 0x7f30b4132a80 2024-09-16T04:12:52.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3335607891 0 0) 0x7f30b411c0b0 con 0x7f30b409ec20 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 10 monclient(hunting): my global_id is 64785 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.516+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30b411d230 con 0x7f30b409ec20 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4088517160 0 0) 0x7f3098003520 con 0x7f30b4132a80 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30b411c0b0 con 0x7f30b4132a80 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f30b0001c10 con 0x7f30b4132a80 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:52.534 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3549919919 0 0) 0x7f30b411d230 con 0x7f30b409ec20 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3098003520 con 0x7f30b409ec20 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f30a40040c0 con 0x7f30b409ec20 2024-09-16T04:12:52.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3317762822 0 0) 0x7f30b411c0b0 con 0x7f30b4132a80 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 >> v1:172.21.15.148:6790/0 conn(0x7f30b4132e60 legacy=0x7f30b41cf630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 >> v1:172.21.15.148:6789/0 conn(0x7f30b409ec20 legacy=0x7f30b411b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 monclient: found mon.b 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b411e6e0 con 0x7f30b4132a80 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 5 monclient: authenticate success, global_id 6346 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 10 monclient: _renew_subs 2024-09-16T04:12:52.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30b40a3db0 con 0x7f30b4132a80 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 10 monclient: _renew_subs 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30b411e400 con 0x7f30b4132a80 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30b00018c0 con 0x7f30b4132a80 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f30b0004de0 con 0x7f30b4132a80 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30aa7fc640 10 monclient: dump: 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:52.538 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.517+0000 7f30abfff640 4 set_mon_vals no callback set 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f30b0005000 con 0x7f30b4132a80 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30bc07b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30bc07b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30bc07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f309c005080 con 0x7f30b4132a80 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(237..237 src has 1..237) ==== 9485+0+0 (unknown 2815424696 0 0) 0x7f30b00486a0 con 0x7f30b4132a80 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.518+0000 7f30b9e17640 10 monclient: get_auth_request con 0x7f3098038f10 auth_method 0 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.522+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f30b0015f10 con 0x7f30b4132a80 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.522+0000 7f30aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.522+0000 7f30aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a8812c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a8812c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a8812c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9a8006bc20 con 0x7f9a8009cc30 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a767fc640 1 -- 172.21.15.148:0/3777051657 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2300205382) 0x7f9a78014af0 con 0x7f9a8009cc30 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:52.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.557+0000 7f9a767fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:52.561 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:52.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 >> v1:172.21.15.171:6812/7327273 conn(0x7f9a64039950 legacy=0x7f9a6403be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 10 monclient: shutdown 2024-09-16T04:12:52.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 >> v1:172.21.15.148:6789/0 conn(0x7f9a8009cc30 legacy=0x7f9a801d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 shutdown_connections 2024-09-16T04:12:52.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 >> 172.21.15.148:0/3777051657 conn(0x7f9a800a5640 msgr2=0x7f9a801338f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 shutdown_connections 2024-09-16T04:12:52.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.559+0000 7f9a8812c640 1 -- 172.21.15.148:0/3777051657 wait complete. 2024-09-16T04:12:52.573 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:12:52.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:12:52.573 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.668+0000 7f6f2c311640 10 monclient: get_monmap_and_config 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.668+0000 7f6f2c311640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.668+0000 7f6f2c311640 10 monclient: monmap: 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:52.670346+0000 2024-09-16T04:12:52.671 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:52.670346+0000 2024-09-16T04:12:52.672 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.672 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.672 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.672 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:52.672 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:52.672 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.668+0000 7f6f2c311640 1 Processor -- start 2024-09-16T04:12:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 1 -- start start 2024-09-16T04:12:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient: init 2024-09-16T04:12:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient(hunting): picked mon.noname-c con 0x7f6f2409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient(hunting): picked mon.noname-a con 0x7f6f2409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient(hunting): picked mon.noname-b con 0x7f6f2409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f240a1010 con 0x7f6f2409c660 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f2409f030 con 0x7f6f2409bb70 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f2409dc60 con 0x7f6f2409cc30 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2c311640 10 monclient(hunting): authenticate will time out at 1193.295669s 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f2a8ae640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6f2409cc30 legacy=0x7f6f2413c090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f298ac640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6f2409c660 0x7f6f24138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55226/0 (socket says 172.21.15.148:55226) 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.669+0000 7f6f298ac640 1 -- 172.21.15.148:0/2809731827 learned_addr learned my addr 172.21.15.148:0/2809731827 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323851988 0 0) 0x7f6f240a1010 con 0x7f6f2409c660 2024-09-16T04:12:52.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): my global_id is 64788 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f10003520 con 0x7f6f2409c660 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1939328931 0 0) 0x7f6f2409f030 con 0x7f6f2409bb70 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient(hunting): my global_id is 6349 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f240a1010 con 0x7f6f2409bb70 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1619287290 0 0) 0x7f6f10003520 con 0x7f6f2409c660 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 >> v1:172.21.15.148:6790/0 conn(0x7f6f2409cc30 legacy=0x7f6f2413c090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:52.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 >> v1:172.21.15.171:6789/0 conn(0x7f6f2409bb70 legacy=0x7f6f2409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 monclient: found mon.noname-a 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2409df40 con 0x7f6f2409c660 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f20001610 con 0x7f6f2409c660 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 5 monclient: authenticate success, global_id 64788 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: dump: 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f200018c0 con 0x7f6f2409c660 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 1 -- 172.21.15.148:0/2809731827 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f20001c90 con 0x7f6f2409c660 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 4 set_mon_vals no callback set 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f290ab640 10 monclient: dump: 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.686 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 10 monclient: shutdown 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 1 -- 172.21.15.148:0/2809731827 >> v1:172.21.15.148:6789/0 conn(0x7f6f2409c660 legacy=0x7f6f24138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 1 -- 172.21.15.148:0/2809731827 shutdown_connections 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 1 -- 172.21.15.148:0/2809731827 >> 172.21.15.148:0/2809731827 conn(0x7f6f240a5640 msgr2=0x7f6f240a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 1 -- 172.21.15.148:0/2809731827 shutdown_connections 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.670+0000 7f6f2c311640 1 -- 172.21.15.148:0/2809731827 wait complete. 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.671+0000 7f6f2c311640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.671+0000 7f6f2c311640 10 monclient: monmap: 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:52.688 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.671+0000 7f6f2c311640 1 Processor -- start 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.671+0000 7f6f2c311640 1 -- start start 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.671+0000 7f6f2c311640 10 monclient: init 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient(hunting): picked mon.noname-a con 0x7f6f2409bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient(hunting): picked mon.noname-c con 0x7f6f2409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient(hunting): picked mon.noname-b con 0x7f6f2409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f240740a0 con 0x7f6f2409bb70 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f2413bb80 con 0x7f6f2409cc30 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f241cc610 con 0x7f6f2409c660 2024-09-16T04:12:52.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient(hunting): authenticate will time out at 1193.298243s 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f298ac640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6f2409c660 legacy=0x7f6f241cbf00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2a0ad640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6f2409bb70 0x7f6f241cb7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55242/0 (socket says 172.21.15.148:55242) 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2a0ad640 1 -- 172.21.15.148:0/4103333836 learned_addr learned my addr 172.21.15.148:0/4103333836 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1419564520 0 0) 0x7f6f240a1010 con 0x7f6f2409bb70 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): my global_id is 64791 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f08003520 con 0x7f6f2409bb70 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019920929 0 0) 0x7f6f2413bb80 con 0x7f6f2409cc30 2024-09-16T04:12:52.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): my global_id is 6352 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f240a1010 con 0x7f6f2409cc30 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3943436894 0 0) 0x7f6f08003520 con 0x7f6f2409bb70 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f2413bb80 con 0x7f6f2409bb70 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f18001850 con 0x7f6f2409bb70 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.691 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 131121579 0 0) 0x7f6f240a1010 con 0x7f6f2409cc30 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f08003520 con 0x7f6f2409cc30 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f1c004330 con 0x7f6f2409cc30 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient(hunting): dump: 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1475091085 0 0) 0x7f6f2413bb80 con 0x7f6f2409bb70 2024-09-16T04:12:52.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.148:6790/0 conn(0x7f6f2409c660 legacy=0x7f6f241cbf00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.171:6789/0 conn(0x7f6f2409cc30 legacy=0x7f6f241d3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 monclient: found mon.a 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2413cac0 con 0x7f6f2409bb70 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 5 monclient: authenticate success, global_id 64791 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient: _renew_subs 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.672+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2413c140 con 0x7f6f2409bb70 2024-09-16T04:12:52.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f2c311640 10 monclient: _renew_subs 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f2c311640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f2413be60 con 0x7f6f2409bb70 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f18001c60 con 0x7f6f2409bb70 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f18003a60 con 0x7f6f2409bb70 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f288aa640 4 set_mon_vals no callback set 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 10 monclient: dump: 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.695 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6f18003c00 con 0x7f6f2409bb70 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f2c311640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f0c0065c0 con 0x7f6f0c002860 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.673+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(237..237 src has 1..237) ==== 9485+0+0 (unknown 2815424696 0 0) 0x7f6f180483d0 con 0x7f6f2409bb70 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f298ac640 10 monclient: get_auth_request con 0x7f6f08038e80 auth_method 0 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421554106 0 0) 0x7f6f0c0065c0 con 0x7f6f0c002860 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient(hunting): my global_id is 64794 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f2413bb80 con 0x7f6f0c002860 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1398351493 0 0) 0x7f6f2413bb80 con 0x7f6f0c002860 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f0c0065c0 con 0x7f6f0c002860 2024-09-16T04:12:52.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f1c004b90 con 0x7f6f0c002860 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient: dump: 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.698 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3061263356 0 0) 0x7f6f0c0065c0 con 0x7f6f0c002860 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f0c00a150 con 0x7f6f0c002860 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6f1c0059c0 con 0x7f6f0c002860 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.148:6789/0 conn(0x7f6f0c002860 legacy=0x7f6f0c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.674+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.148:6789/0 conn(0x7f6f0c002860 legacy=0x7f6f0c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:52.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.693+0000 7f30bc07b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T04:12:52.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.693+0000 7f30bc07b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T04:12:52.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.693+0000 7f30bc07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:52.700 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:52.693+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) -- 0x7f309c005370 con 0x7f30b4132a80 2024-09-16T04:12:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f0c009e40 con 0x7f6f0c009100 2024-09-16T04:12:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127866777 0 0) 0x7f6f0c009e40 con 0x7f6f0c009100 2024-09-16T04:12:52.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient(hunting): my global_id is 64797 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f0c0065c0 con 0x7f6f0c009100 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2749534020 0 0) 0x7f6f0c0065c0 con 0x7f6f0c009100 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f0c009e40 con 0x7f6f0c009100 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f1c005290 con 0x7f6f0c009100 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient: dump: 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.702 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3564025777 0 0) 0x7f6f0c009e40 con 0x7f6f0c009100 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6f0c00c3d0 con 0x7f6f0c009100 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 1407951089) 0x7f6f1c0050b0 con 0x7f6f0c009100 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:52.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.148:6789/0 conn(0x7f6f0c009100 legacy=0x7f6f0c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f16ffd640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.148:6789/0 conn(0x7f6f0c009100 legacy=0x7f6f0c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 32, 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 151880, 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:52.704 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:52.705 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:52.706 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:52.707 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:52.708 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.709 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:52.710 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.712 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.713 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.714 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:52.715 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:52.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.171:6812/7327273 conn(0x7f6f08038e80 legacy=0x7f6f0803b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 10 monclient: shutdown 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 >> v1:172.21.15.148:6789/0 conn(0x7f6f2409bb70 legacy=0x7f6f241cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2a8ae640 1 -- 172.21.15.148:0/4103333836 reap_dead start 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 shutdown_connections 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.681+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 >> 172.21.15.148:0/4103333836 conn(0x7f6f240a5640 msgr2=0x7f6f2406e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.682+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 shutdown_connections 2024-09-16T04:12:52.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.682+0000 7f6f2c311640 1 -- 172.21.15.148:0/4103333836 wait complete. 2024-09-16T04:12:52.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:12:52.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:12:52.718 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:12:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient: get_monmap_and_config 2024-09-16T04:12:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient: monmap: 2024-09-16T04:12:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:12:52.815393+0000 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:12:52.815393+0000 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 1 Processor -- start 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 1 -- start start 2024-09-16T04:12:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient: init 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient(hunting): picked mon.noname-a con 0x7f365c0a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient(hunting): picked mon.noname-c con 0x7f365c0a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f365c13d3c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c0a0590 con 0x7f365c0a3a90 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c0a1270 con 0x7f365c0a4580 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c0a4fd0 con 0x7f365c13d3c0 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f3660b5f640 10 monclient(hunting): authenticate will time out at 1193.440742s 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f365ad9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f365c13d3c0 legacy=0x7f365c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f365a59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f365c0a3a90 0x7f365c0a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55274/0 (socket says 172.21.15.148:55274) 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.814+0000 7f365a59c640 1 -- 172.21.15.148:0/538982425 learned_addr learned my addr 172.21.15.148:0/538982425 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61702039 0 0) 0x7f3654001110 con 0x7f365c0a3a90 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): my global_id is 64800 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3644003520 con 0x7f365c0a3a90 2024-09-16T04:12:52.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645978903 0 0) 0x7f365c0a1270 con 0x7f365c0a4580 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient(hunting): my global_id is 6355 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3654001110 con 0x7f365c0a4580 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3618728637 0 0) 0x7f3644003520 con 0x7f365c0a3a90 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 >> v1:172.21.15.148:6790/0 conn(0x7f365c13d3c0 legacy=0x7f365c13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 >> v1:172.21.15.171:6789/0 conn(0x7f365c0a4580 legacy=0x7f365c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 monclient: found mon.noname-a 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365c06e170 con 0x7f365c0a3a90 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f36540017e0 con 0x7f365c0a3a90 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 5 monclient: authenticate success, global_id 64800 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: dump: 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.836 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3654001b00 con 0x7f365c0a3a90 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 1 -- 172.21.15.148:0/538982425 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3654004a40 con 0x7f365c0a3a90 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 4 set_mon_vals no callback set 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f365959a640 10 monclient: dump: 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 10 monclient: shutdown 2024-09-16T04:12:52.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.815+0000 7f3660b5f640 1 -- 172.21.15.148:0/538982425 >> v1:172.21.15.148:6789/0 conn(0x7f365c0a3a90 legacy=0x7f365c0a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.816+0000 7f3660b5f640 1 -- 172.21.15.148:0/538982425 shutdown_connections 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.816+0000 7f3660b5f640 1 -- 172.21.15.148:0/538982425 >> 172.21.15.148:0/538982425 conn(0x7f365c0a9a40 msgr2=0x7f365c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.816+0000 7f3660b5f640 1 -- 172.21.15.148:0/538982425 shutdown_connections 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.816+0000 7f3660b5f640 1 -- 172.21.15.148:0/538982425 wait complete. 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.816+0000 7f3660b5f640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.816+0000 7f3660b5f640 10 monclient: monmap: 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.839 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:52.840 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:52.840 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:52.840 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 1 Processor -- start 2024-09-16T04:12:52.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 1 -- start start 2024-09-16T04:12:52.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient: init 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient(hunting): picked mon.noname-a con 0x7f365c0a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient(hunting): picked mon.noname-c con 0x7f365c0a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f365c13d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c13c730 con 0x7f365c0a3a90 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c09b910 con 0x7f365c13d3c0 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365c1ccbe0 con 0x7f365c0a4580 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3659d9b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f365c0a4580 legacy=0x7f365c1cc4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:52.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f3660b5f640 10 monclient(hunting): authenticate will time out at 1193.443668s 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f365a59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f365c0a3a90 0x7f365c1cbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:55278/0 (socket says 172.21.15.148:55278) 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f365a59c640 1 -- 172.21.15.148:0/689580238 learned_addr learned my addr 172.21.15.148:0/689580238 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450976218 0 0) 0x7f365c13c730 con 0x7f365c0a3a90 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 10 monclient(hunting): my global_id is 64803 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f363c003520 con 0x7f365c0a3a90 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.817+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555377191 0 0) 0x7f365c09b910 con 0x7f365c13d3c0 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): my global_id is 6358 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f365c13c730 con 0x7f365c13d3c0 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2458447516 0 0) 0x7f363c003520 con 0x7f365c0a3a90 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f365c09b910 con 0x7f365c0a3a90 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f36540017a0 con 0x7f365c0a3a90 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): dump: 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.843 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:52.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4125354785 0 0) 0x7f365c13c730 con 0x7f365c13d3c0 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f363c003520 con 0x7f365c13d3c0 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f364c003fb0 con 0x7f365c13d3c0 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient(hunting): dump: 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.845 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3846345978 0 0) 0x7f365c09b910 con 0x7f365c0a3a90 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.148:6790/0 conn(0x7f365c0a4580 legacy=0x7f365c1cc4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.171:6789/0 conn(0x7f365c13d3c0 legacy=0x7f365c1d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 monclient: found mon.a 2024-09-16T04:12:52.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365c0a8130 con 0x7f365c0a3a90 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 5 monclient: authenticate success, global_id 64803 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 10 monclient: _renew_subs 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f365c0a0280 con 0x7f365c0a3a90 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 10 monclient: _renew_subs 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f365c09b630 con 0x7f365c0a3a90 2024-09-16T04:12:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3654001b00 con 0x7f365c0a3a90 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3654004d50 con 0x7f365c0a3a90 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f3658d99640 4 set_mon_vals no callback set 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.818+0000 7f364affd640 10 monclient: dump: 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f3660b5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f36400065c0 con 0x7f3640002860 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3654010180 con 0x7f365c0a3a90 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(237..237 src has 1..237) ==== 9485+0+0 (unknown 2815424696 0 0) 0x7f36540485a0 con 0x7f365c0a3a90 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f365ad9d640 10 monclient: get_auth_request con 0x7f363c038e30 auth_method 0 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967308429 0 0) 0x7f36400065c0 con 0x7f3640002860 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient(hunting): my global_id is 6361 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f365c09b910 con 0x7f3640002860 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2858497931 0 0) 0x7f365c09b910 con 0x7f3640002860 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f36400065c0 con 0x7f3640002860 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3650001bf0 con 0x7f3640002860 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 10 monclient: dump: 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.850 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1360433481 0 0) 0x7f36400065c0 con 0x7f3640002860 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.819+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f364000a150 con 0x7f3640002860 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.820+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3650001950 con 0x7f3640002860 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.820+0000 7f364affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.820+0000 7f364affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:52.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.820+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.171:6789/0 conn(0x7f3640002860 legacy=0x7f3640004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.820+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.171:6789/0 conn(0x7f3640002860 legacy=0x7f3640004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f3660b5f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3640009e40 con 0x7f3640009100 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1926866915 0 0) 0x7f3640009e40 con 0x7f3640009100 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f364affd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f364affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f364affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f364affd640 10 monclient(hunting): my global_id is 6364 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.826+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36400065c0 con 0x7f3640009100 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3206660458 0 0) 0x7f36400065c0 con 0x7f3640009100 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3640009e40 con 0x7f3640009100 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3650004c90 con 0x7f3640009100 2024-09-16T04:12:52.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 10 monclient: dump: 2024-09-16T04:12:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:12:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.853 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2741020053 0 0) 0x7f3640009e40 con 0x7f3640009100 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f364000c3d0 con 0x7f3640009100 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4300 (unknown 0 0 307341782) 0x7f3650006180 con 0x7f3640009100 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.171:6789/0 conn(0x7f3640009100 legacy=0x7f3640009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f364affd640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.171:6789/0 conn(0x7f3640009100 legacy=0x7f3640009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 32, 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 396866, 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:12:52.855 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:52.856 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:52.857 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:52.858 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:12:52.859 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:12:52.860 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.861 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:52.862 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:12:52.863 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:12:52.864 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:12:52.865 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.866 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:12:52.867 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:12:52.868 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:12:52.868 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:12:52.868 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:12:52.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.171:6812/7327273 conn(0x7f363c038e30 legacy=0x7f363c03b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 10 monclient: shutdown 2024-09-16T04:12:52.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 >> v1:172.21.15.148:6789/0 conn(0x7f365c0a3a90 legacy=0x7f365c1cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f365ad9d640 1 -- 172.21.15.148:0/689580238 reap_dead start 2024-09-16T04:12:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f365ad9d640 1 -- 172.21.15.148:0/689580238 reap_dead start 2024-09-16T04:12:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 shutdown_connections 2024-09-16T04:12:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 >> 172.21.15.148:0/689580238 conn(0x7f365c0a9a40 msgr2=0x7f365c0a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.827+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 shutdown_connections 2024-09-16T04:12:52.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:12:52.828+0000 7f3660b5f640 1 -- 172.21.15.148:0/689580238 wait complete. 2024-09-16T04:12:52.870 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:12:52.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:12:53.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.323+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v238) ==== 120+0+0 (unknown 1539084720 0 0) 0x7f30b0015810 con 0x7f30b4132a80 2024-09-16T04:12:53.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.323+0000 7f30aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T04:12:53.325 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.323+0000 7f30aa7fc640 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-16T04:12:53.430 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.428+0000 7f30bc07b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T04:12:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.429+0000 7f30bc07b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T04:12:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.429+0000 7f30bc07b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.429+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) -- 0x7f309c001090 con 0x7f30b4132a80 2024-09-16T04:12:53.431 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.430+0000 7f30aa7fc640 1 -- 172.21.15.171:0/3294025876 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v238) ==== 120+0+0 (unknown 3447780018 0 0) 0x7f30b001e360 con 0x7f30b4132a80 2024-09-16T04:12:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.430+0000 7f30aa7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-16T04:12:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.430+0000 7f30aa7fc640 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-16T04:12:53.432 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h23' does not appear in the crush map 2024-09-16T04:12:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 >> v1:172.21.15.171:6812/7327273 conn(0x7f3098038f10 legacy=0x7f309803b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 10 monclient: shutdown 2024-09-16T04:12:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:53.434 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 >> v1:172.21.15.171:6789/0 conn(0x7f30b4132a80 legacy=0x7f30b411b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 shutdown_connections 2024-09-16T04:12:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 >> 172.21.15.171:0/3294025876 conn(0x7f30b40a52a0 msgr2=0x7f30b4131a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 shutdown_connections 2024-09-16T04:12:53.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.432+0000 7f30bc07b640 1 -- 172.21.15.171:0/3294025876 wait complete. 2024-09-16T04:12:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:53.443 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h24 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.521+0000 7f103ad9f640 10 monclient: get_monmap_and_config 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.521+0000 7f103ad9f640 10 monclient: build_initial_monmap 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.521+0000 7f103ad9f640 10 monclient: monmap: 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:53.522033+0000 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:53.522033+0000 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:53.523 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:53.524 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:53.524 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.521+0000 7f103ad9f640 1 Processor -- start 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 1 -- start start 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient: init 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient(hunting): picked mon.noname-c con 0x7f1034073020 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient(hunting): picked mon.noname-a con 0x7f1034137400 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient(hunting): picked mon.noname-b con 0x7f103413d270 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f103409fec0 con 0x7f1034137400 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10340a09c0 con 0x7f1034073020 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10340a0ca0 con 0x7f103413d270 2024-09-16T04:12:53.531 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103ad9f640 10 monclient(hunting): authenticate will time out at 1149.027641s 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1039d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f1034073020 0x7f1034136cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34386/0 (socket says 172.21.15.171:34386) 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1039d9d640 1 -- 172.21.15.171:0/2642352406 learned_addr learned my addr 172.21.15.171:0/2642352406 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f103a59e640 1 -- 172.21.15.171:0/2642352406 >> v1:172.21.15.148:6790/0 conn(0x7f103413d270 legacy=0x7f103413f660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936497853 0 0) 0x7f10340a09c0 con 0x7f1034073020 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1038d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1038d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1038d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1038d9b640 10 monclient(hunting): my global_id is 6367 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.522+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1020003520 con 0x7f1034073020 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2692446022 0 0) 0x7f1020003520 con 0x7f1034073020 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 >> v1:172.21.15.148:6790/0 conn(0x7f103413d270 legacy=0x7f103413f660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:53.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 >> v1:172.21.15.148:6789/0 conn(0x7f1034137400 legacy=0x7f103413bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 monclient: found mon.noname-c 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10340a0400 con 0x7f1034073020 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: _finish_auth 0 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 5 monclient: authenticate success, global_id 6367 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1028001610 con 0x7f1034073020 2024-09-16T04:12:53.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: dump: 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:53.534 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1028001970 con 0x7f1034073020 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 1 -- 172.21.15.171:0/2642352406 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10280048a0 con 0x7f1034073020 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 4 set_mon_vals no callback set 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f1038d9b640 10 monclient: dump: 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:53.535 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 10 monclient: shutdown 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 1 -- 172.21.15.171:0/2642352406 >> v1:172.21.15.171:6789/0 conn(0x7f1034073020 legacy=0x7f1034136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 1 -- 172.21.15.171:0/2642352406 shutdown_connections 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 1 -- 172.21.15.171:0/2642352406 >> 172.21.15.171:0/2642352406 conn(0x7f10340a9320 msgr2=0x7f10340a9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:53.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 1 -- 172.21.15.171:0/2642352406 shutdown_connections 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.523+0000 7f103ad9f640 1 -- 172.21.15.171:0/2642352406 wait complete. 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.524+0000 7f103ad9f640 10 monclient: build_initial_monmap 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.524+0000 7f103ad9f640 10 monclient: monmap: 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.524+0000 7f103ad9f640 1 Processor -- start 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.524+0000 7f103ad9f640 1 -- start start 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.524+0000 7f103ad9f640 10 monclient: init 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient(hunting): picked mon.noname-a con 0x7f1034073020 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient(hunting): picked mon.noname-c con 0x7f1034137400 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient(hunting): picked mon.noname-b con 0x7f103413d270 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f103409b840 con 0x7f1034073020 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f103413c900 con 0x7f103413d270 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10340a00f0 con 0x7f1034137400 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient(hunting): authenticate will time out at 1149.030229s 2024-09-16T04:12:53.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103959c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f1034137400 legacy=0x7f103409b130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103a59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f103413d270 0x7f10341d3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34388/0 (socket says 172.21.15.171:34388) 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103a59e640 1 -- 172.21.15.171:0/1668149642 learned_addr learned my addr 172.21.15.171:0/1668149642 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966950755 0 0) 0x7f103413c900 con 0x7f103413d270 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): my global_id is 6370 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1018003520 con 0x7f103413d270 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4173091167 0 0) 0x7f103409b840 con 0x7f1034073020 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:53.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): my global_id is 64809 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f103413c900 con 0x7f1034073020 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1822285804 0 0) 0x7f1018003520 con 0x7f103413d270 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f103409b840 con 0x7f103413d270 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f102c0043b0 con 0x7f103413d270 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): dump: 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.540 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:53.541 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.541 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:53.541 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:53.541 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:53.541 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:53.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 450039797 0 0) 0x7f103413c900 con 0x7f1034073020 2024-09-16T04:12:53.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1018003520 con 0x7f1034073020 2024-09-16T04:12:53.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1028001430 con 0x7f1034073020 2024-09-16T04:12:53.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:53.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:53.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient(hunting): dump: 2024-09-16T04:12:53.543 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:53.543 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:53.543 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.543 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.543 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:53.543 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4223036487 0 0) 0x7f103409b840 con 0x7f103413d270 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 >> v1:172.21.15.148:6790/0 conn(0x7f1034137400 legacy=0x7f103409b130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 >> v1:172.21.15.148:6789/0 conn(0x7f1034073020 legacy=0x7f103409aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 monclient: found mon.b 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103413cdc0 con 0x7f103413d270 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f10267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 5 monclient: authenticate success, global_id 6370 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient: _renew_subs 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.525+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f103409dde0 con 0x7f103413d270 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 10 monclient: _renew_subs 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f103409cb50 con 0x7f103413d270 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f102c004660 con 0x7f103413d270 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:53.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f102c004a30 con 0x7f103413d270 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f1027fff640 4 set_mon_vals no callback set 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 10 monclient: dump: 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:53.546 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f102c004cd0 con 0x7f103413d270 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.526+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c005080 con 0x7f103413d270 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.527+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(238..238 src has 1..238) ==== 9272+0+0 (unknown 2348892336 0 0) 0x7f102c048680 con 0x7f103413d270 2024-09-16T04:12:53.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.527+0000 7f1039d9d640 10 monclient: get_auth_request con 0x7f1018038ea0 auth_method 0 2024-09-16T04:12:53.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.531+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f102c015f10 con 0x7f103413d270 2024-09-16T04:12:53.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.531+0000 7f10267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:53.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.531+0000 7f10267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:53.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.703+0000 7f103ad9f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T04:12:53.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.703+0000 7f103ad9f640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T04:12:53.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.703+0000 7f103ad9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:53.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:53.703+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) -- 0x7f101c005370 con 0x7f103413d270 2024-09-16T04:12:54.327 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.324+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v239) ==== 120+0+0 (unknown 1464002232 0 0) 0x7f102c015810 con 0x7f103413d270 2024-09-16T04:12:54.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.324+0000 7f10267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T04:12:54.328 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.324+0000 7f10267fc640 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-16T04:12:54.435 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.433+0000 7f103ad9f640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T04:12:54.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.433+0000 7f103ad9f640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T04:12:54.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.433+0000 7f103ad9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.433+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) -- 0x7f101c001090 con 0x7f103413d270 2024-09-16T04:12:54.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.434+0000 7f10267fc640 1 -- 172.21.15.171:0/1668149642 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v239) ==== 120+0+0 (unknown 3783782784 0 0) 0x7f102c019b60 con 0x7f103413d270 2024-09-16T04:12:54.436 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.434+0000 7f10267fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-16T04:12:54.437 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.434+0000 7f10267fc640 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-16T04:12:54.437 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h24' does not appear in the crush map 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 >> v1:172.21.15.171:6812/7327273 conn(0x7f1018038ea0 legacy=0x7f101803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 10 monclient: shutdown 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 >> v1:172.21.15.171:6789/0 conn(0x7f103413d270 legacy=0x7f10341d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 shutdown_connections 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 >> 172.21.15.171:0/1668149642 conn(0x7f10340a9320 msgr2=0x7f10341400e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 shutdown_connections 2024-09-16T04:12:54.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.436+0000 7f103ad9f640 1 -- 172.21.15.171:0/1668149642 wait complete. 2024-09-16T04:12:54.448 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:54.448 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h25 2024-09-16T04:12:54.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.524+0000 7fe932be8640 10 monclient: get_monmap_and_config 2024-09-16T04:12:54.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.524+0000 7fe932be8640 10 monclient: build_initial_monmap 2024-09-16T04:12:54.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.524+0000 7fe932be8640 10 monclient: monmap: 2024-09-16T04:12:54.527 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:54.527 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:54.526406+0000 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:54.526406+0000 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:54.528 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:54.529 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:54.535 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 1 Processor -- start 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 1 -- start start 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient: init 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient(hunting): picked mon.noname-c con 0x7fe92c132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient(hunting): picked mon.noname-b con 0x7fe92c09ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient(hunting): picked mon.noname-a con 0x7fe92c132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92c1332b0 con 0x7fe92c132e60 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92c134310 con 0x7fe92c132a80 2024-09-16T04:12:54.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92c09d980 con 0x7fe92c09ec20 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.525+0000 7fe932be8640 10 monclient(hunting): authenticate will time out at 1150.031926s 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe930984640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe92c132a80 0x7fe92c1334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34396/0 (socket says 172.21.15.171:34396) 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe930984640 1 -- 172.21.15.171:0/4043926317 learned_addr learned my addr 172.21.15.171:0/4043926317 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92bfff640 1 -- 172.21.15.171:0/4043926317 >> v1:172.21.15.148:6790/0 conn(0x7fe92c09ec20 legacy=0x7fe92c133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607298776 0 0) 0x7fe92c134310 con 0x7fe92c132a80 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:54.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient(hunting): my global_id is 6373 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe91c003520 con 0x7fe92c132a80 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4136266720 0 0) 0x7fe91c003520 con 0x7fe92c132a80 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 >> v1:172.21.15.148:6790/0 conn(0x7fe92c09ec20 legacy=0x7fe92c133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 >> v1:172.21.15.148:6789/0 conn(0x7fe92c132e60 legacy=0x7fe92c13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:54.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c09dc60 con 0x7fe92c132a80 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe920001610 con 0x7fe92c132a80 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe932be8640 5 monclient: authenticate success, global_id 6373 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:54.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: dump: 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:54.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe932be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe920001900 con 0x7fe92c132a80 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 1 -- 172.21.15.171:0/4043926317 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe920001cd0 con 0x7fe92c132a80 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe932be8640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe932be8640 4 set_mon_vals no callback set 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.526+0000 7fe92b7fe640 10 monclient: dump: 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.541 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 10 monclient: shutdown 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 1 -- 172.21.15.171:0/4043926317 >> v1:172.21.15.171:6789/0 conn(0x7fe92c132a80 legacy=0x7fe92c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 1 -- 172.21.15.171:0/4043926317 shutdown_connections 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 1 -- 172.21.15.171:0/4043926317 >> 172.21.15.171:0/4043926317 conn(0x7fe92c0a52a0 msgr2=0x7fe92c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 1 -- 172.21.15.171:0/4043926317 shutdown_connections 2024-09-16T04:12:54.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 1 -- 172.21.15.171:0/4043926317 wait complete. 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 10 monclient: build_initial_monmap 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.527+0000 7fe932be8640 10 monclient: monmap: 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:54.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 1 Processor -- start 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 1 -- start start 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient: init 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient(hunting): picked mon.noname-a con 0x7fe92c09ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient(hunting): picked mon.noname-b con 0x7fe92c132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient(hunting): picked mon.noname-c con 0x7fe92c132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92c130f30 con 0x7fe92c09ec20 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92c1229e0 con 0x7fe92c132a80 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe92c121150 con 0x7fe92c132e60 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:54.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe932be8640 10 monclient(hunting): authenticate will time out at 1150.034696s 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe92bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe92c132a80 0x7fe92c1ce630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34402/0 (socket says 172.21.15.171:34402) 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe92bfff640 1 -- 172.21.15.171:0/2959448355 learned_addr learned my addr 172.21.15.171:0/2959448355 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe931185640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe92c132e60 legacy=0x7fe92c1d1d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453142075 0 0) 0x7fe92c1229e0 con 0x7fe92c132a80 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe9297fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe9297fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe9297fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe9297fa640 10 monclient(hunting): my global_id is 6376 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.528+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe910003520 con 0x7fe92c132a80 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625487646 0 0) 0x7fe92c130f30 con 0x7fe92c09ec20 2024-09-16T04:12:54.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): my global_id is 64815 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe92c1229e0 con 0x7fe92c09ec20 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3374505979 0 0) 0x7fe910003520 con 0x7fe92c132a80 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe92c130f30 con 0x7fe92c132a80 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe918004180 con 0x7fe92c132a80 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): dump: 2024-09-16T04:12:54.546 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1415334346 0 0) 0x7fe92c1229e0 con 0x7fe92c09ec20 2024-09-16T04:12:54.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe910003520 con 0x7fe92c09ec20 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3385397474 0 0) 0x7fe92c130f30 con 0x7fe92c132a80 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 >> v1:172.21.15.148:6790/0 conn(0x7fe92c132e60 legacy=0x7fe92c1d1d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 >> v1:172.21.15.148:6789/0 conn(0x7fe92c09ec20 legacy=0x7fe92c1c9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 monclient: found mon.b 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92c122cc0 con 0x7fe92c132a80 2024-09-16T04:12:54.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 5 monclient: authenticate success, global_id 6376 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 10 monclient: _renew_subs 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe92c09c980 con 0x7fe92c132a80 2024-09-16T04:12:54.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 10 monclient: _renew_subs 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe92c121590 con 0x7fe92c132a80 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe9180044a0 con 0x7fe92c132a80 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe918004870 con 0x7fe92c132a80 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:54.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe9297fa640 10 monclient: dump: 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.529+0000 7fe92affd640 4 set_mon_vals no callback set 2024-09-16T04:12:54.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe918004a90 con 0x7fe92c132a80 2024-09-16T04:12:54.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe932be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:54.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe932be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:54.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe932be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c005080 con 0x7fe92c132a80 2024-09-16T04:12:54.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(239..239 src has 1..239) ==== 9059+0+0 (unknown 3350830221 0 0) 0x7fe918048440 con 0x7fe92c132a80 2024-09-16T04:12:54.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.530+0000 7fe930984640 10 monclient: get_auth_request con 0x7fe910038e70 auth_method 0 2024-09-16T04:12:54.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.535+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe918015e00 con 0x7fe92c132a80 2024-09-16T04:12:54.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.535+0000 7fe9297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:54.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.535+0000 7fe9297fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:54.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.705+0000 7fe932be8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T04:12:54.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.705+0000 7fe932be8640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T04:12:54.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.705+0000 7fe932be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:54.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:54.705+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) -- 0x7fe90c005370 con 0x7fe92c132a80 2024-09-16T04:12:55.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.333+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v240) ==== 120+0+0 (unknown 1141000189 0 0) 0x7fe918015700 con 0x7fe92c132a80 2024-09-16T04:12:55.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.333+0000 7fe9297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T04:12:55.336 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.333+0000 7fe9297fa640 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-16T04:12:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.437+0000 7fe932be8640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T04:12:55.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.437+0000 7fe932be8640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T04:12:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.437+0000 7fe932be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.437+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) -- 0x7fe90c002210 con 0x7fe92c132a80 2024-09-16T04:12:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.438+0000 7fe9297fa640 1 -- 172.21.15.171:0/2959448355 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v240) ==== 120+0+0 (unknown 3242825715 0 0) 0x7fe91801e2b0 con 0x7fe92c132a80 2024-09-16T04:12:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.438+0000 7fe9297fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-16T04:12:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.438+0000 7fe9297fa640 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-16T04:12:55.441 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h25' does not appear in the crush map 2024-09-16T04:12:55.443 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.440+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 >> v1:172.21.15.171:6812/7327273 conn(0x7fe910038e70 legacy=0x7fe91003b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 10 monclient: shutdown 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 >> v1:172.21.15.171:6789/0 conn(0x7fe92c132a80 legacy=0x7fe92c1ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 shutdown_connections 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 >> 172.21.15.171:0/2959448355 conn(0x7fe92c0a52a0 msgr2=0x7fe92c0a4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 shutdown_connections 2024-09-16T04:12:55.444 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.441+0000 7fe932be8640 1 -- 172.21.15.171:0/2959448355 wait complete. 2024-09-16T04:12:55.453 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:55.453 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h26 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.529+0000 7ff1dc065640 10 monclient: get_monmap_and_config 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.529+0000 7ff1dc065640 10 monclient: build_initial_monmap 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.529+0000 7ff1dc065640 10 monclient: monmap: 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:55.531081+0000 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:55.531081+0000 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:55.532 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:55.533 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:55.533 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:55.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.529+0000 7ff1dc065640 1 Processor -- start 2024-09-16T04:12:55.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 1 -- start start 2024-09-16T04:12:55.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient: init 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d4132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d409ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d4132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d41332b0 con 0x7ff1d409ec20 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d4134310 con 0x7ff1d4132a80 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d409d980 con 0x7ff1d4132e60 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.530+0000 7ff1dc065640 10 monclient(hunting): authenticate will time out at 1151.036931s 2024-09-16T04:12:55.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1da602640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7ff1d4132e60 legacy=0x7ff1d413c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:55.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d9e01640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff1d4132a80 0x7ff1d41334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34412/0 (socket says 172.21.15.171:34412) 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d9e01640 1 -- 172.21.15.171:0/1485674089 learned_addr learned my addr 172.21.15.171:0/1485674089 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1353614 0 0) 0x7ff1d4134310 con 0x7ff1d4132a80 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): my global_id is 6379 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1c0003520 con 0x7ff1d4132a80 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125980254 0 0) 0x7ff1d41332b0 con 0x7ff1d409ec20 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:55.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient(hunting): my global_id is 64818 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d4134310 con 0x7ff1d409ec20 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1149435109 0 0) 0x7ff1c0003520 con 0x7ff1d4132a80 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 >> v1:172.21.15.148:6790/0 conn(0x7ff1d4132e60 legacy=0x7ff1d413c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 >> v1:172.21.15.148:6789/0 conn(0x7ff1d409ec20 legacy=0x7ff1d4133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 monclient: found mon.noname-c 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d409dc60 con 0x7ff1d4132a80 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: _finish_auth 0 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:55.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff1d0001610 con 0x7ff1d4132a80 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: dump: 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:55.545 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1dc065640 5 monclient: authenticate success, global_id 6379 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1dc065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1d00018c0 con 0x7ff1d4132a80 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 1 -- 172.21.15.171:0/1485674089 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff1d0001c90 con 0x7ff1d4132a80 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1dc065640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1dc065640 4 set_mon_vals no callback set 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.531+0000 7ff1d8dff640 10 monclient: dump: 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.546 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 10 monclient: shutdown 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 1 -- 172.21.15.171:0/1485674089 >> v1:172.21.15.171:6789/0 conn(0x7ff1d4132a80 legacy=0x7ff1d41334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 1 -- 172.21.15.171:0/1485674089 shutdown_connections 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 1 -- 172.21.15.171:0/1485674089 >> 172.21.15.171:0/1485674089 conn(0x7ff1d40a52a0 msgr2=0x7ff1d40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 1 -- 172.21.15.171:0/1485674089 shutdown_connections 2024-09-16T04:12:55.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.532+0000 7ff1dc065640 1 -- 172.21.15.171:0/1485674089 wait complete. 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient: build_initial_monmap 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient: monmap: 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:55.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 1 Processor -- start 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 1 -- start start 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient: init 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d409ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d4132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d4132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d412eb70 con 0x7ff1d4132e60 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1dc065640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d4120620 con 0x7ff1d4132a80 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.533+0000 7ff1d9e01640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7ff1d409ec20 legacy=0x7ff1d411dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d411eaf0 con 0x7ff1d409ec20 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:55.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 10 monclient(hunting): authenticate will time out at 1151.040029s 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1d9600640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff1d4132a80 0x7ff1d411e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34422/0 (socket says 172.21.15.171:34422) 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1d9600640 1 -- 172.21.15.171:0/3020264426 learned_addr learned my addr 172.21.15.171:0/3020264426 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741084114 0 0) 0x7ff1d4120620 con 0x7ff1d4132a80 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): my global_id is 6382 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b8003520 con 0x7ff1d4132a80 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3232048266 0 0) 0x7ff1b8003520 con 0x7ff1d4132a80 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1d4120620 con 0x7ff1d4132a80 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff1c4001910 con 0x7ff1d4132a80 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:55.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): dump: 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330567208 0 0) 0x7ff1d412eb70 con 0x7ff1d4132e60 2024-09-16T04:12:55.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient(hunting): my global_id is 64821 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b8003520 con 0x7ff1d4132e60 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1738954614 0 0) 0x7ff1d4120620 con 0x7ff1d4132a80 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 >> v1:172.21.15.148:6790/0 conn(0x7ff1d409ec20 legacy=0x7ff1d411dcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 >> v1:172.21.15.148:6789/0 conn(0x7ff1d4132e60 legacy=0x7ff1d41d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 monclient: found mon.b 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d411ed90 con 0x7ff1d4132a80 2024-09-16T04:12:55.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 5 monclient: authenticate success, global_id 6382 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 10 monclient: _renew_subs 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d4120900 con 0x7ff1d4132a80 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 10 monclient: _renew_subs 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1d4120340 con 0x7ff1d4132a80 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1c4003940 con 0x7ff1d4132a80 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff1c4003c90 con 0x7ff1d4132a80 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:55.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1cbfff640 4 set_mon_vals no callback set 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.534+0000 7ff1ca7fc640 10 monclient: dump: 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:55.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.535+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff1c4003eb0 con 0x7ff1d4132a80 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.535+0000 7ff1dc065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.535+0000 7ff1dc065640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.535+0000 7ff1dc065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.535+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1bc005080 con 0x7ff1d4132a80 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.536+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(240..240 src has 1..240) ==== 8846+0+0 (unknown 1596796342 0 0) 0x7ff1c4048400 con 0x7ff1d4132a80 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.536+0000 7ff1d9e01640 10 monclient: get_auth_request con 0x7ff1b8038fd0 auth_method 0 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.540+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff1c4015e30 con 0x7ff1d4132a80 2024-09-16T04:12:55.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.540+0000 7ff1ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:55.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.540+0000 7ff1ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:55.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.710+0000 7ff1dc065640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T04:12:55.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.710+0000 7ff1dc065640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T04:12:55.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.710+0000 7ff1dc065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:55.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:55.710+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) -- 0x7ff1bc005370 con 0x7ff1d4132a80 2024-09-16T04:12:56.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.338+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v241) ==== 120+0+0 (unknown 3591655590 0 0) 0x7ff1c4015730 con 0x7ff1d4132a80 2024-09-16T04:12:56.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.338+0000 7ff1ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T04:12:56.341 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.338+0000 7ff1ca7fc640 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-16T04:12:56.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.444+0000 7ff1dc065640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T04:12:56.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.444+0000 7ff1dc065640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T04:12:56.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.444+0000 7ff1dc065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.444+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) -- 0x7ff1bc001090 con 0x7ff1d4132a80 2024-09-16T04:12:56.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.445+0000 7ff1ca7fc640 1 -- 172.21.15.171:0/3020264426 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v241) ==== 120+0+0 (unknown 43735511 0 0) 0x7ff1c4019a80 con 0x7ff1d4132a80 2024-09-16T04:12:56.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.445+0000 7ff1ca7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-16T04:12:56.449 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.445+0000 7ff1ca7fc640 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-16T04:12:56.449 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h26' does not appear in the crush map 2024-09-16T04:12:56.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 >> v1:172.21.15.171:6812/7327273 conn(0x7ff1b8038fd0 legacy=0x7ff1b803b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 10 monclient: shutdown 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 >> v1:172.21.15.171:6789/0 conn(0x7ff1d4132a80 legacy=0x7ff1d411e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 shutdown_connections 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 >> 172.21.15.171:0/3020264426 conn(0x7ff1d40a52a0 msgr2=0x7ff1d40a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 shutdown_connections 2024-09-16T04:12:56.451 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.447+0000 7ff1dc065640 1 -- 172.21.15.171:0/3020264426 wait complete. 2024-09-16T04:12:56.459 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:56.459 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h27 2024-09-16T04:12:56.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.534+0000 7f3dd165e640 10 monclient: get_monmap_and_config 2024-09-16T04:12:56.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.534+0000 7f3dd165e640 10 monclient: build_initial_monmap 2024-09-16T04:12:56.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.534+0000 7f3dd165e640 10 monclient: monmap: 2024-09-16T04:12:56.538 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:56.538 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:56.536609+0000 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:56.536609+0000 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.535+0000 7f3dd165e640 1 Processor -- start 2024-09-16T04:12:56.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.535+0000 7f3dd165e640 1 -- start start 2024-09-16T04:12:56.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.535+0000 7f3dd165e640 10 monclient: init 2024-09-16T04:12:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient(hunting): picked mon.noname-a con 0x7f3dcc09f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:56.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient(hunting): picked mon.noname-b con 0x7f3dcc0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient(hunting): picked mon.noname-c con 0x7f3dcc09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc0a1260 con 0x7f3dcc09f640 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc09d0c0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc09d3a0 con 0x7f3dcc0a0130 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient(hunting): authenticate will time out at 1152.042177s 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dc3fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3dcc0a0130 legacy=0x7f3dcc138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:56.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dcb7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3dcc09e1d0 0x7f3dcc13bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34426/0 (socket says 172.21.15.171:34426) 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dcb7fe640 1 -- 172.21.15.171:0/3815392984 learned_addr learned my addr 172.21.15.171:0/3815392984 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 67289657 0 0) 0x7f3dcc09d0c0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient(hunting): my global_id is 6385 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db8003440 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1249802925 0 0) 0x7f3db8003440 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 >> v1:172.21.15.148:6790/0 conn(0x7f3dcc0a0130 legacy=0x7f3dcc138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 >> v1:172.21.15.148:6789/0 conn(0x7f3dcc09f640 legacy=0x7f3dcc09fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:56.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 monclient: found mon.noname-c 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc09d680 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:56.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3dc4001990 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 5 monclient: authenticate success, global_id 6385 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: dump: 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.552 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3dc4001c80 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 1 -- 172.21.15.171:0/3815392984 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3dc4002090 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:56.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dd165e640 4 set_mon_vals no callback set 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.536+0000 7f3dca7fc640 10 monclient: dump: 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:56.554 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 10 monclient: shutdown 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 1 -- 172.21.15.171:0/3815392984 >> v1:172.21.15.171:6789/0 conn(0x7f3dcc09e1d0 legacy=0x7f3dcc13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 1 -- 172.21.15.171:0/3815392984 shutdown_connections 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 1 -- 172.21.15.171:0/3815392984 >> 172.21.15.171:0/3815392984 conn(0x7f3dcc0a52a0 msgr2=0x7f3dcc0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 1 -- 172.21.15.171:0/3815392984 shutdown_connections 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.537+0000 7f3dd165e640 1 -- 172.21.15.171:0/3815392984 wait complete. 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient: build_initial_monmap 2024-09-16T04:12:56.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient: monmap: 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 1 Processor -- start 2024-09-16T04:12:56.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 1 -- start start 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient: init 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient(hunting): picked mon.noname-b con 0x7f3dcc09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient(hunting): picked mon.noname-c con 0x7f3dcc09f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient(hunting): picked mon.noname-a con 0x7f3dcc0a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc1a55e0 con 0x7f3dcc0a0130 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc1a5ba0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dcc0a3c70 con 0x7f3dcc09f640 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dd165e640 10 monclient(hunting): authenticate will time out at 1152.044862s 2024-09-16T04:12:56.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dc3fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3dcc09f640 legacy=0x7f3dcc1cc0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dcaffd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3dcc09e1d0 0x7f3dcc1b61c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:34430/0 (socket says 172.21.15.171:34430) 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.538+0000 7f3dcaffd640 1 -- 172.21.15.171:0/1999622276 learned_addr learned my addr 172.21.15.171:0/1999622276 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 658558471 0 0) 0x7f3dcc1a5ba0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 6388 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dac003520 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628698410 0 0) 0x7f3dcc1a55e0 con 0x7f3dcc0a0130 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): my global_id is 64827 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dcc1a5ba0 con 0x7f3dcc0a0130 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1123222181 0 0) 0x7f3dac003520 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3dcc1a55e0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3dbc0019e0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): dump: 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:56.559 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3694965792 0 0) 0x7f3dcc1a55e0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 >> v1:172.21.15.148:6790/0 conn(0x7f3dcc09f640 legacy=0x7f3dcc1cc0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:56.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 >> v1:172.21.15.148:6789/0 conn(0x7f3dcc0a0130 legacy=0x7f3dcc1cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 monclient: found mon.b 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc06ceb0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 5 monclient: authenticate success, global_id 6388 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 10 monclient: _renew_subs 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3dcc1a5300 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 10 monclient: _renew_subs 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3dcc1a4190 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3dbc0038c0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3dbc003c90 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc9ffb640 4 set_mon_vals no callback set 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.539+0000 7f3dc37fe640 10 monclient: dump: 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.562 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3dbc003eb0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dd165e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dd165e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dd165e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db0005080 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(241..241 src has 1..241) ==== 8633+0+0 (unknown 2113044720 0 0) 0x7f3dbc048310 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.540+0000 7f3dc3fff640 10 monclient: get_auth_request con 0x7f3dac038b90 auth_method 0 2024-09-16T04:12:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.545+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3dbc015db0 con 0x7f3dcc09e1d0 2024-09-16T04:12:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.545+0000 7f3dc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:56.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.545+0000 7f3dc37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:56.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.716+0000 7f3dd165e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T04:12:56.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.716+0000 7f3dd165e640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T04:12:56.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.716+0000 7f3dd165e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:56.719 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:56.716+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) -- 0x7f3db0005370 con 0x7f3dcc09e1d0 2024-09-16T04:12:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.337+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v242) ==== 120+0+0 (unknown 1223094776 0 0) 0x7f3dbc019a60 con 0x7f3dcc09e1d0 2024-09-16T04:12:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.337+0000 7f3dc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T04:12:57.340 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.337+0000 7f3dc37fe640 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-16T04:12:57.446 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.443+0000 7f3dd165e640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T04:12:57.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.443+0000 7f3dd165e640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T04:12:57.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.443+0000 7f3dd165e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.443+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) -- 0x7f3db0001090 con 0x7f3dcc09e1d0 2024-09-16T04:12:57.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.445+0000 7f3dc37fe640 1 -- 172.21.15.171:0/1999622276 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v242) ==== 120+0+0 (unknown 2899685468 0 0) 0x7f3dbc01e320 con 0x7f3dcc09e1d0 2024-09-16T04:12:57.447 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.445+0000 7f3dc37fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-16T04:12:57.448 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.445+0000 7f3dc37fe640 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-16T04:12:57.448 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h27' does not appear in the crush map 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 >> v1:172.21.15.171:6812/7327273 conn(0x7f3dac038b90 legacy=0x7f3dac03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 10 monclient: shutdown 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 >> v1:172.21.15.171:6789/0 conn(0x7f3dcc09e1d0 legacy=0x7f3dcc1b61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 shutdown_connections 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 >> 172.21.15.171:0/1999622276 conn(0x7f3dcc0a52a0 msgr2=0x7f3dcc1b8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 shutdown_connections 2024-09-16T04:12:57.450 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.447+0000 7f3dd165e640 1 -- 172.21.15.171:0/1999622276 wait complete. 2024-09-16T04:12:57.458 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:57.458 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h28 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.535+0000 7f8bd6fc4640 10 monclient: get_monmap_and_config 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.535+0000 7f8bd6fc4640 10 monclient: build_initial_monmap 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.535+0000 7f8bd6fc4640 10 monclient: monmap: 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:57.537313+0000 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:57.537313+0000 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:57.538 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:57.539 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:57.539 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:57.539 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:57.539 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:57.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 1 Processor -- start 2024-09-16T04:12:57.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 1 -- start start 2024-09-16T04:12:57.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient: init 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd00685d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd009bfc0 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd0136e10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd0099c00 con 0x7f8bd009bfc0 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd01373c0 con 0x7f8bd0136e10 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd009a150 con 0x7f8bd00685d0 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd6fc4640 10 monclient(hunting): authenticate will time out at 1153.042784s 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd4d60640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f8bd00685d0 legacy=0x7f8bd009b8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd5561640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8bd0136e10 0x7f8bd013f350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54246/0 (socket says 172.21.15.171:54246) 2024-09-16T04:12:57.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.536+0000 7f8bd5561640 1 -- 172.21.15.171:0/1164625070 learned_addr learned my addr 172.21.15.171:0/1164625070 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128871791 0 0) 0x7f8bd01373c0 con 0x7f8bd0136e10 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient(hunting): my global_id is 6391 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc0003520 con 0x7f8bd0136e10 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 387388483 0 0) 0x7f8bc0003520 con 0x7f8bd0136e10 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 >> v1:172.21.15.148:6790/0 conn(0x7f8bd00685d0 legacy=0x7f8bd009b8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 >> v1:172.21.15.148:6789/0 conn(0x7f8bd009bfc0 legacy=0x7f8bd013bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:57.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd0143aa0 con 0x7f8bd0136e10 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8bc8001990 con 0x7f8bd0136e10 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 5 monclient: authenticate success, global_id 6391 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:57.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:57.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: dump: 2024-09-16T04:12:57.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:57.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bc8001cf0 con 0x7f8bd0136e10 2024-09-16T04:12:57.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 1 -- 172.21.15.171:0/1164625070 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8bc80020c0 con 0x7f8bd0136e10 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 4 set_mon_vals no callback set 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bcf7fe640 10 monclient: dump: 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:57.553 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.554 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.554 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:57.554 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:57.554 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:57.554 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:57.554 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:57.555 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:57.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 10 monclient: shutdown 2024-09-16T04:12:57.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:57.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.537+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/1164625070 >> v1:172.21.15.171:6789/0 conn(0x7f8bd0136e10 legacy=0x7f8bd013f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:57.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/1164625070 shutdown_connections 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/1164625070 >> 172.21.15.171:0/1164625070 conn(0x7f8bd00a1e90 msgr2=0x7f8bd00a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/1164625070 shutdown_connections 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/1164625070 wait complete. 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 10 monclient: build_initial_monmap 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 10 monclient: monmap: 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:57.556 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:57.557 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.538+0000 7f8bd6fc4640 1 Processor -- start 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 1 -- start start 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient: init 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f8bd00685d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:57.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f8bd009bfc0 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:57.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f8bd0136e10 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:57.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd00a3290 con 0x7f8bd0136e10 2024-09-16T04:12:57.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd009c580 con 0x7f8bd009bfc0 2024-09-16T04:12:57.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd013d880 con 0x7f8bd00685d0 2024-09-16T04:12:57.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:57.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd6fc4640 10 monclient(hunting): authenticate will time out at 1153.045674s 2024-09-16T04:12:57.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bd4d60640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f8bd00685d0 legacy=0x7f8bd013a600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:57.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f8bd009bfc0 0x7f8bd013ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54250/0 (socket says 172.21.15.171:54250) 2024-09-16T04:12:57.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcffff640 1 -- 172.21.15.171:0/3072821338 learned_addr learned my addr 172.21.15.171:0/3072821338 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:57.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 443037820 0 0) 0x7f8bd009c580 con 0x7f8bd009bfc0 2024-09-16T04:12:57.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:57.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:57.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:57.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcd7fa640 10 monclient(hunting): my global_id is 6394 2024-09-16T04:12:57.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.539+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bb4003520 con 0x7f8bd009bfc0 2024-09-16T04:12:57.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1218997715 0 0) 0x7f8bb4003520 con 0x7f8bd009bfc0 2024-09-16T04:12:57.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8bd009c580 con 0x7f8bd009bfc0 2024-09-16T04:12:57.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120559618 0 0) 0x7f8bd0143aa0 con 0x7f8bd0136e10 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): my global_id is 64833 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bb4003520 con 0x7f8bd0136e10 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8bbc0018f0 con 0x7f8bd009bfc0 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): dump: 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:57.562 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4177345734 0 0) 0x7f8bd009c580 con 0x7f8bd009bfc0 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 >> v1:172.21.15.148:6790/0 conn(0x7f8bd00685d0 legacy=0x7f8bd013a600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 >> v1:172.21.15.148:6789/0 conn(0x7f8bd0136e10 legacy=0x7f8bd01d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:57.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 monclient: found mon.b 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd013d450 con 0x7f8bd009bfc0 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 5 monclient: authenticate success, global_id 6394 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 10 monclient: _renew_subs 2024-09-16T04:12:57.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd013b420 con 0x7f8bd009bfc0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 10 monclient: _renew_subs 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8bd01d5120 con 0x7f8bd009bfc0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8bbc001cb0 con 0x7f8bd009bfc0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8bbc003a60 con 0x7f8bd009bfc0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bceffd640 4 set_mon_vals no callback set 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.540+0000 7f8bcd7fa640 10 monclient: dump: 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:57.565 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8bbc003c80 con 0x7f8bd009bfc0 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bd6fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bd6fc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bd6fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb0005080 con 0x7f8bd009bfc0 2024-09-16T04:12:57.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(242..242 src has 1..242) ==== 8420+0+0 (unknown 4083151172 0 0) 0x7f8bbc0480c0 con 0x7f8bd009bfc0 2024-09-16T04:12:57.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.541+0000 7f8bd4d60640 10 monclient: get_auth_request con 0x7f8bb4038fc0 auth_method 0 2024-09-16T04:12:57.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.545+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8bbc015d00 con 0x7f8bd009bfc0 2024-09-16T04:12:57.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.545+0000 7f8bcd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:57.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.545+0000 7f8bcd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:57.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.714+0000 7f8bd6fc4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T04:12:57.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.714+0000 7f8bd6fc4640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T04:12:57.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.714+0000 7f8bd6fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:57.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:57.714+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) -- 0x7f8bb0005370 con 0x7f8bd009bfc0 2024-09-16T04:12:58.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.346+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v243) ==== 121+0+0 (unknown 2034464365 0 0) 0x7f8bbc015600 con 0x7f8bd009bfc0 2024-09-16T04:12:58.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.346+0000 7f8bcd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T04:12:58.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.346+0000 7f8bcd7fa640 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-16T04:12:58.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.451+0000 7f8bd6fc4640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T04:12:58.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.451+0000 7f8bd6fc4640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T04:12:58.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.451+0000 7f8bd6fc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.454 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.451+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) -- 0x7f8bb0001090 con 0x7f8bd009bfc0 2024-09-16T04:12:58.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.452+0000 7f8bcd7fa640 1 -- 172.21.15.171:0/3072821338 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v243) ==== 120+0+0 (unknown 1519738803 0 0) 0x7f8bbc01e1b0 con 0x7f8bd009bfc0 2024-09-16T04:12:58.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.452+0000 7f8bcd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-16T04:12:58.455 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.452+0000 7f8bcd7fa640 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-16T04:12:58.455 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h28' does not appear in the crush map 2024-09-16T04:12:58.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 >> v1:172.21.15.171:6812/7327273 conn(0x7f8bb4038fc0 legacy=0x7f8bb403b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:58.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 10 monclient: shutdown 2024-09-16T04:12:58.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:58.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 >> v1:172.21.15.171:6789/0 conn(0x7f8bd009bfc0 legacy=0x7f8bd013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:58.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 shutdown_connections 2024-09-16T04:12:58.457 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 >> 172.21.15.171:0/3072821338 conn(0x7f8bd00a1e90 msgr2=0x7f8bd01424a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:58.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 shutdown_connections 2024-09-16T04:12:58.458 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.454+0000 7f8bd6fc4640 1 -- 172.21.15.171:0/3072821338 wait complete. 2024-09-16T04:12:58.466 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:58.466 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h29 2024-09-16T04:12:58.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.541+0000 7fa255d98640 10 monclient: get_monmap_and_config 2024-09-16T04:12:58.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.541+0000 7fa255d98640 10 monclient: build_initial_monmap 2024-09-16T04:12:58.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.541+0000 7fa255d98640 10 monclient: monmap: 2024-09-16T04:12:58.544 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:58.544 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:58.544 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:58.543399+0000 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:58.543399+0000 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:58.545 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 1 Processor -- start 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 1 -- start start 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 10 monclient: init 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 10 monclient(hunting): picked mon.noname-a con 0x7fa250132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 10 monclient(hunting): picked mon.noname-c con 0x7fa25009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.542+0000 7fa255d98640 10 monclient(hunting): picked mon.noname-b con 0x7fa250132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2501332b0 con 0x7fa250132a80 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa250134310 con 0x7fa25009ec20 2024-09-16T04:12:58.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25009d980 con 0x7fa250132e60 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 10 monclient(hunting): authenticate will time out at 1154.049078s 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24ffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa250132e60 legacy=0x7fa25013c840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa25009ec20 0x7fa250133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54264/0 (socket says 172.21.15.171:54264) 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24effd640 1 -- 172.21.15.171:0/1176676687 learned_addr learned my addr 172.21.15.171:0/1176676687 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479062541 0 0) 0x7fa250134310 con 0x7fa25009ec20 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): my global_id is 6397 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa238003520 con 0x7fa25009ec20 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180692695 0 0) 0x7fa2501332b0 con 0x7fa250132a80 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:58.555 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient(hunting): my global_id is 64836 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa250134310 con 0x7fa250132a80 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2991463753 0 0) 0x7fa238003520 con 0x7fa25009ec20 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 >> v1:172.21.15.148:6790/0 conn(0x7fa250132e60 legacy=0x7fa25013c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 >> v1:172.21.15.148:6789/0 conn(0x7fa250132a80 legacy=0x7fa2501334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25009dc60 con 0x7fa25009ec20 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:58.556 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 5 monclient: authenticate success, global_id 6397 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa244001610 con 0x7fa25009ec20 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: dump: 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:58.557 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa2440018c0 con 0x7fa25009ec20 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 1 -- 172.21.15.171:0/1176676687 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa244001c90 con 0x7fa25009ec20 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa255d98640 4 set_mon_vals no callback set 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:58.558 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.543+0000 7fa24e7fc640 10 monclient: dump: 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 10 monclient: shutdown 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 1 -- 172.21.15.171:0/1176676687 >> v1:172.21.15.171:6789/0 conn(0x7fa25009ec20 legacy=0x7fa250133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:58.559 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 1 -- 172.21.15.171:0/1176676687 shutdown_connections 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 1 -- 172.21.15.171:0/1176676687 >> 172.21.15.171:0/1176676687 conn(0x7fa2500a52a0 msgr2=0x7fa2500a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 1 -- 172.21.15.171:0/1176676687 shutdown_connections 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 1 -- 172.21.15.171:0/1176676687 wait complete. 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 10 monclient: build_initial_monmap 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.544+0000 7fa255d98640 10 monclient: monmap: 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:58.560 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 1 Processor -- start 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 1 -- start start 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient: init 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient(hunting): picked mon.noname-a con 0x7fa25009ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient(hunting): picked mon.noname-b con 0x7fa250132a80 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient(hunting): picked mon.noname-c con 0x7fa250132e60 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa250120f30 con 0x7fa25009ec20 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa250122dc0 con 0x7fa250132a80 2024-09-16T04:12:58.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2500a33c0 con 0x7fa250132e60 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa255d98640 10 monclient(hunting): authenticate will time out at 1154.051711s 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa24ffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa250132e60 legacy=0x7fa2501d1f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa24effd640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa250132a80 0x7fa2501ce7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54276/0 (socket says 172.21.15.171:54276) 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa24effd640 1 -- 172.21.15.171:0/3101855334 learned_addr learned my addr 172.21.15.171:0/3101855334 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588759607 0 0) 0x7fa250122dc0 con 0x7fa250132a80 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa237fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa237fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa237fff640 10 monclient(hunting): my global_id is 6400 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.545+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa23c003520 con 0x7fa250132a80 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3673424804 0 0) 0x7fa250120f30 con 0x7fa25009ec20 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:58.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): my global_id is 64839 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa250122dc0 con 0x7fa25009ec20 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2188470858 0 0) 0x7fa23c003520 con 0x7fa250132a80 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa250120f30 con 0x7fa250132a80 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa244001c10 con 0x7fa250132a80 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:58.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): dump: 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:58.564 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4241482819 0 0) 0x7fa250122dc0 con 0x7fa25009ec20 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa23c003520 con 0x7fa25009ec20 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa2400018b0 con 0x7fa25009ec20 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient(hunting): dump: 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.565 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 156172255 0 0) 0x7fa250120f30 con 0x7fa250132a80 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 >> v1:172.21.15.148:6790/0 conn(0x7fa250132e60 legacy=0x7fa2501d1f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 >> v1:172.21.15.148:6789/0 conn(0x7fa25009ec20 legacy=0x7fa2501c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 monclient: found mon.b 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25006ceb0 con 0x7fa250132a80 2024-09-16T04:12:58.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: _finish_auth 0 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 5 monclient: authenticate success, global_id 6400 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 10 monclient: _renew_subs 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa250122800 con 0x7fa250132a80 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 10 monclient: _renew_subs 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa25011ff00 con 0x7fa250132a80 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa2440018c0 con 0x7fa250132a80 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa244004c40 con 0x7fa250132a80 2024-09-16T04:12:58.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa24dffb640 4 set_mon_vals no callback set 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.546+0000 7fa237fff640 10 monclient: dump: 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:58.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa244004de0 con 0x7fa250132a80 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa255d98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa255d98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa255d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa230005080 con 0x7fa250132a80 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(243..243 src has 1..243) ==== 8207+0+0 (unknown 3362072827 0 0) 0x7fa244047fa0 con 0x7fa250132a80 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.547+0000 7fa24f7fe640 10 monclient: get_auth_request con 0x7fa23c038ef0 auth_method 0 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.552+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa244015d50 con 0x7fa250132a80 2024-09-16T04:12:58.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.552+0000 7fa237fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:58.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.552+0000 7fa237fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:58.723 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.721+0000 7fa255d98640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T04:12:58.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.721+0000 7fa255d98640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T04:12:58.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.721+0000 7fa255d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:58.724 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:58.721+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) -- 0x7fa230005370 con 0x7fa250132a80 2024-09-16T04:12:59.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.353+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v244) ==== 121+0+0 (unknown 2949187316 0 0) 0x7fa244006230 con 0x7fa250132a80 2024-09-16T04:12:59.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.353+0000 7fa237fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T04:12:59.355 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.353+0000 7fa237fff640 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-16T04:12:59.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.457+0000 7fa255d98640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T04:12:59.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.457+0000 7fa255d98640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T04:12:59.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.457+0000 7fa255d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.460 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.457+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) -- 0x7fa230001090 con 0x7fa250132a80 2024-09-16T04:12:59.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.458+0000 7fa237fff640 1 -- 172.21.15.171:0/3101855334 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v244) ==== 120+0+0 (unknown 3224777840 0 0) 0x7fa24401e200 con 0x7fa250132a80 2024-09-16T04:12:59.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.458+0000 7fa237fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-16T04:12:59.461 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.458+0000 7fa237fff640 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-16T04:12:59.461 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h29' does not appear in the crush map 2024-09-16T04:12:59.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.460+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 >> v1:172.21.15.171:6812/7327273 conn(0x7fa23c038ef0 legacy=0x7fa23c03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:59.463 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.460+0000 7fa255d98640 10 monclient: shutdown 2024-09-16T04:12:59.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.460+0000 7fa255d98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:59.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.460+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 >> v1:172.21.15.171:6789/0 conn(0x7fa250132a80 legacy=0x7fa2501ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:59.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.460+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 shutdown_connections 2024-09-16T04:12:59.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.460+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 >> 172.21.15.171:0/3101855334 conn(0x7fa2500a52a0 msgr2=0x7fa25009fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:59.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.461+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 shutdown_connections 2024-09-16T04:12:59.464 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.461+0000 7fa255d98640 1 -- 172.21.15.171:0/3101855334 wait complete. 2024-09-16T04:12:59.472 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:12:59.472 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h30 2024-09-16T04:12:59.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.548+0000 7f4906c7b640 10 monclient: get_monmap_and_config 2024-09-16T04:12:59.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.548+0000 7f4906c7b640 10 monclient: build_initial_monmap 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.548+0000 7f4906c7b640 10 monclient: monmap: 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:12:59.549840+0000 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:12:59.549840+0000 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:12:59.551 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.548+0000 7f4906c7b640 1 Processor -- start 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.548+0000 7f4906c7b640 1 -- start start 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.548+0000 7f4906c7b640 10 monclient: init 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f4900136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f4900139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f490013d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49000a1340 con 0x7f4900136b60 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4900137650 con 0x7f490013d620 2024-09-16T04:12:59.560 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49000a0aa0 con 0x7f4900139b10 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4906c7b640 10 monclient(hunting): authenticate will time out at 1155.055492s 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48fffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f4900139b10 legacy=0x7f490013bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4905218640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f490013d620 0x7f490013fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54290/0 (socket says 172.21.15.171:54290) 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f4905218640 1 -- 172.21.15.171:0/1439217478 learned_addr learned my addr 172.21.15.171:0/1439217478 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2178759286 0 0) 0x7f4900137650 con 0x7f490013d620 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): my global_id is 6403 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48e8003520 con 0x7f490013d620 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240815735 0 0) 0x7f49000a1340 con 0x7f4900136b60 2024-09-16T04:12:59.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 10 monclient(hunting): my global_id is 64842 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4900137650 con 0x7f4900136b60 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.549+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1843412233 0 0) 0x7f48e8003520 con 0x7f490013d620 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 >> v1:172.21.15.148:6790/0 conn(0x7f4900139b10 legacy=0x7f490013bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 >> v1:172.21.15.148:6789/0 conn(0x7f4900136b60 legacy=0x7f4900136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49000a0550 con 0x7f490013d620 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:59.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 5 monclient: authenticate success, global_id 6403 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48f8001990 con 0x7f490013d620 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: dump: 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48f8001cb0 con 0x7f490013d620 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 1 -- 172.21.15.171:0/1439217478 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48f8002080 con 0x7f490013d620 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 4 set_mon_vals no callback set 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f48ff7fe640 10 monclient: dump: 2024-09-16T04:12:59.564 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:59.565 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 10 monclient: shutdown 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 1 -- 172.21.15.171:0/1439217478 >> v1:172.21.15.171:6789/0 conn(0x7f490013d620 legacy=0x7f490013fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 1 -- 172.21.15.171:0/1439217478 shutdown_connections 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 1 -- 172.21.15.171:0/1439217478 >> 172.21.15.171:0/1439217478 conn(0x7f49000a98b0 msgr2=0x7f49000a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 1 -- 172.21.15.171:0/1439217478 shutdown_connections 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.550+0000 7f4906c7b640 1 -- 172.21.15.171:0/1439217478 wait complete. 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.551+0000 7f4906c7b640 10 monclient: build_initial_monmap 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.551+0000 7f4906c7b640 10 monclient: monmap: 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:12:59.566 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.551+0000 7f4906c7b640 1 Processor -- start 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.551+0000 7f4906c7b640 1 -- start start 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.551+0000 7f4906c7b640 10 monclient: init 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient(hunting): picked mon.noname-a con 0x7f4900136b60 addr v1:172.21.15.148:6789/0 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient(hunting): picked mon.noname-b con 0x7f4900139b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient(hunting): picked mon.noname-c con 0x7f490013d620 addr v1:172.21.15.148:6790/0 2024-09-16T04:12:59.567 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49000a6f50 con 0x7f4900136b60 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49001208b0 con 0x7f4900139b10 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49001cc5c0 con 0x7f490013d620 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient(hunting): authenticate will time out at 1155.058125s 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f4900139b10 0x7f49001cbeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54300/0 (socket says 172.21.15.171:54300) 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fffff640 1 -- 172.21.15.171:0/410314809 learned_addr learned my addr 172.21.15.171:0/410314809 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4905218640 1 -- 172.21.15.171:0/410314809 >> v1:172.21.15.148:6790/0 conn(0x7f490013d620 legacy=0x7f49001d3a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3815003716 0 0) 0x7f49001208b0 con 0x7f4900139b10 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): my global_id is 6406 2024-09-16T04:12:59.568 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48ec003520 con 0x7f4900139b10 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637559549 0 0) 0x7f4900137650 con 0x7f4900136b60 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): my global_id is 64845 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49001208b0 con 0x7f4900136b60 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1110421475 0 0) 0x7f48ec003520 con 0x7f4900139b10 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4900137650 con 0x7f4900139b10 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48f40041d0 con 0x7f4900139b10 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): dump: 2024-09-16T04:12:59.569 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1486323623 0 0) 0x7f49001208b0 con 0x7f4900136b60 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48ec003520 con 0x7f4900136b60 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48f00018b0 con 0x7f4900136b60 2024-09-16T04:12:59.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient(hunting): dump: 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3988982885 0 0) 0x7f4900137650 con 0x7f4900139b10 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 >> v1:172.21.15.148:6790/0 conn(0x7f490013d620 legacy=0x7f49001d3a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 >> v1:172.21.15.148:6789/0 conn(0x7f4900136b60 legacy=0x7f49001cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 monclient: found mon.b 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49000a2220 con 0x7f4900139b10 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 5 monclient: authenticate success, global_id 6406 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient: _renew_subs 2024-09-16T04:12:59.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49000a0240 con 0x7f4900139b10 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient: _renew_subs 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f490009e300 con 0x7f4900139b10 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48f4004480 con 0x7f4900139b10 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48f4004850 con 0x7f4900139b10 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48feffd640 4 set_mon_vals no callback set 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.552+0000 7f48fd7fa640 10 monclient: dump: 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:12:59.573 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.553+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f48f4004ab0 con 0x7f4900139b10 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.553+0000 7f4906c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.553+0000 7f4906c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.553+0000 7f4906c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.553+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48e0005080 con 0x7f4900139b10 2024-09-16T04:12:59.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.553+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(244..244 src has 1..244) ==== 7994+0+0 (unknown 3091027361 0 0) 0x7f48f4047e60 con 0x7f4900139b10 2024-09-16T04:12:59.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.554+0000 7f4904a17640 10 monclient: get_auth_request con 0x7f48ec038ea0 auth_method 0 2024-09-16T04:12:59.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.558+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f48f4015d70 con 0x7f4900139b10 2024-09-16T04:12:59.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.558+0000 7f48fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:59.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.558+0000 7f48fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:59.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.727+0000 7f4906c7b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T04:12:59.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.727+0000 7f4906c7b640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T04:12:59.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.727+0000 7f4906c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:12:59.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:12:59.727+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) -- 0x7f48e0005370 con 0x7f4900139b10 2024-09-16T04:13:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.355+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v245) ==== 121+0+0 (unknown 1670820584 0 0) 0x7f48f4015670 con 0x7f4900139b10 2024-09-16T04:13:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.355+0000 7f48fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T04:13:00.358 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.355+0000 7f48fd7fa640 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-16T04:13:00.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.466+0000 7f4906c7b640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T04:13:00.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.466+0000 7f4906c7b640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T04:13:00.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.466+0000 7f4906c7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.469 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.466+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) -- 0x7f48e0002210 con 0x7f4900139b10 2024-09-16T04:13:00.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.468+0000 7f48fd7fa640 1 -- 172.21.15.171:0/410314809 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v245) ==== 120+0+0 (unknown 557577768 0 0) 0x7f48f401e220 con 0x7f4900139b10 2024-09-16T04:13:00.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.468+0000 7f48fd7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-16T04:13:00.470 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.468+0000 7f48fd7fa640 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-16T04:13:00.471 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h30' does not appear in the crush map 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 >> v1:172.21.15.171:6812/7327273 conn(0x7f48ec038ea0 legacy=0x7f48ec03b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 10 monclient: shutdown 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 >> v1:172.21.15.171:6789/0 conn(0x7f4900139b10 legacy=0x7f49001cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 shutdown_connections 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 >> 172.21.15.171:0/410314809 conn(0x7f49000a98b0 msgr2=0x7f49001392c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 shutdown_connections 2024-09-16T04:13:00.473 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.470+0000 7f4906c7b640 1 -- 172.21.15.171:0/410314809 wait complete. 2024-09-16T04:13:00.482 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:13:00.482 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h31 2024-09-16T04:13:00.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.558+0000 7fb577159640 10 monclient: get_monmap_and_config 2024-09-16T04:13:00.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.558+0000 7fb577159640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.558+0000 7fb577159640 10 monclient: monmap: 2024-09-16T04:13:00.561 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:00.561 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:00.560573+0000 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:00.560573+0000 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:00.562 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:00.569 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.559+0000 7fb577159640 1 Processor -- start 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.559+0000 7fb577159640 1 -- start start 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.559+0000 7fb577159640 10 monclient: init 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient(hunting): picked mon.noname-b con 0x7fb570132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient(hunting): picked mon.noname-c con 0x7fb57009ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient(hunting): picked mon.noname-a con 0x7fb570132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5701332b0 con 0x7fb570132e60 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb570134310 con 0x7fb57009ec20 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57009d980 con 0x7fb570132a80 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.570 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient(hunting): authenticate will time out at 1156.066200s 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb57009ec20 0x7fb570133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54316/0 (socket says 172.21.15.171:54316) 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56ffff640 1 -- 172.21.15.171:0/359257972 learned_addr learned my addr 172.21.15.171:0/359257972 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb574ef5640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb570132a80 legacy=0x7fb5701334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3160768002 0 0) 0x7fb570134310 con 0x7fb57009ec20 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): my global_id is 6409 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb558003520 con 0x7fb57009ec20 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236316643 0 0) 0x7fb5701332b0 con 0x7fb570132e60 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.571 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient(hunting): my global_id is 64848 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb570134310 con 0x7fb570132e60 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 802225336 0 0) 0x7fb558003520 con 0x7fb57009ec20 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 >> v1:172.21.15.148:6790/0 conn(0x7fb570132a80 legacy=0x7fb5701334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 >> v1:172.21.15.148:6789/0 conn(0x7fb570132e60 legacy=0x7fb57013c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57009dc60 con 0x7fb57009ec20 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 5 monclient: authenticate success, global_id 6409 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb564001610 con 0x7fb57009ec20 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: dump: 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:00.573 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb564001870 con 0x7fb57009ec20 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 1 -- 172.21.15.171:0/359257972 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb564001bc0 con 0x7fb57009ec20 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb577159640 4 set_mon_vals no callback set 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.560+0000 7fb56f7fe640 10 monclient: dump: 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:00.574 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 10 monclient: shutdown 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 1 -- 172.21.15.171:0/359257972 >> v1:172.21.15.171:6789/0 conn(0x7fb57009ec20 legacy=0x7fb570133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 1 -- 172.21.15.171:0/359257972 shutdown_connections 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 1 -- 172.21.15.171:0/359257972 >> 172.21.15.171:0/359257972 conn(0x7fb5700a52a0 msgr2=0x7fb5700a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 1 -- 172.21.15.171:0/359257972 shutdown_connections 2024-09-16T04:13:00.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 1 -- 172.21.15.171:0/359257972 wait complete. 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.561+0000 7fb577159640 10 monclient: monmap: 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:00.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 1 Processor -- start 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 1 -- start start 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient: init 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient(hunting): picked mon.noname-c con 0x7fb57009ec20 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient(hunting): picked mon.noname-a con 0x7fb570132a80 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient(hunting): picked mon.noname-b con 0x7fb570132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5701220e0 con 0x7fb570132a80 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb570120340 con 0x7fb570132e60 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb570120570 con 0x7fb57009ec20 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb577159640 10 monclient(hunting): authenticate will time out at 1156.068978s 2024-09-16T04:13:00.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.562+0000 7fb574ef5640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb57009ec20 legacy=0x7fb57011f520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb5756f6640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb570132e60 0x7fb5701d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54320/0 (socket says 172.21.15.171:54320) 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb5756f6640 1 -- 172.21.15.171:0/3997165083 learned_addr learned my addr 172.21.15.171:0/3997165083 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403627047 0 0) 0x7fb570120340 con 0x7fb570132e60 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): my global_id is 6412 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb55c003520 con 0x7fb570132e60 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724018373 0 0) 0x7fb5701220e0 con 0x7fb570132a80 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): my global_id is 64851 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb570120340 con 0x7fb570132a80 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2956418781 0 0) 0x7fb55c003520 con 0x7fb570132e60 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5701220e0 con 0x7fb570132e60 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb568001c50 con 0x7fb570132e60 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): dump: 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:00.579 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1315956809 0 0) 0x7fb5701220e0 con 0x7fb570132e60 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 >> v1:172.21.15.148:6790/0 conn(0x7fb57009ec20 legacy=0x7fb57011f520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 >> v1:172.21.15.148:6789/0 conn(0x7fb570132a80 legacy=0x7fb57011fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 monclient: found mon.b 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5701223c0 con 0x7fb570132e60 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 5 monclient: authenticate success, global_id 6412 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 10 monclient: _renew_subs 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5700a3350 con 0x7fb570132e60 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 10 monclient: _renew_subs 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb5701319e0 con 0x7fb570132e60 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb568003990 con 0x7fb570132e60 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb568003ce0 con 0x7fb570132e60 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56effd640 4 set_mon_vals no callback set 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.563+0000 7fb56d7fa640 10 monclient: dump: 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:00.582 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb568003f00 con 0x7fb570132e60 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb577159640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb577159640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb577159640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb550005080 con 0x7fb570132e60 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(245..245 src has 1..245) ==== 7781+0+0 (unknown 1999182661 0 0) 0x7fb568048070 con 0x7fb570132e60 2024-09-16T04:13:00.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.564+0000 7fb574ef5640 10 monclient: get_auth_request con 0x7fb55c038be0 auth_method 0 2024-09-16T04:13:00.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.569+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb568016070 con 0x7fb570132e60 2024-09-16T04:13:00.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.569+0000 7fb56d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.569+0000 7fb56d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:00.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.738+0000 7fb577159640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T04:13:00.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.738+0000 7fb577159640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T04:13:00.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.738+0000 7fb577159640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:00.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:00.738+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) -- 0x7fb550005370 con 0x7fb570132e60 2024-09-16T04:13:01.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.360+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v246) ==== 121+0+0 (unknown 2939391554 0 0) 0x7fb568015970 con 0x7fb570132e60 2024-09-16T04:13:01.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.360+0000 7fb56d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T04:13:01.363 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.360+0000 7fb56d7fa640 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-16T04:13:01.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.468+0000 7fb577159640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T04:13:01.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.468+0000 7fb577159640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T04:13:01.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.468+0000 7fb577159640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.471 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.468+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) -- 0x7fb550002210 con 0x7fb570132e60 2024-09-16T04:13:01.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.469+0000 7fb56d7fa640 1 -- 172.21.15.171:0/3997165083 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v246) ==== 120+0+0 (unknown 2406817699 0 0) 0x7fb56801e520 con 0x7fb570132e60 2024-09-16T04:13:01.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.469+0000 7fb56d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-16T04:13:01.472 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.469+0000 7fb56d7fa640 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-16T04:13:01.472 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h31' does not appear in the crush map 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 >> v1:172.21.15.171:6812/7327273 conn(0x7fb55c038be0 legacy=0x7fb55c03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 10 monclient: shutdown 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 >> v1:172.21.15.171:6789/0 conn(0x7fb570132e60 legacy=0x7fb5701d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 shutdown_connections 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 >> 172.21.15.171:0/3997165083 conn(0x7fb5700a52a0 msgr2=0x7fb570131f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:01.474 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 shutdown_connections 2024-09-16T04:13:01.475 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.471+0000 7fb577159640 1 -- 172.21.15.171:0/3997165083 wait complete. 2024-09-16T04:13:01.483 INFO:tasks.workunit.client.0.smithi171.stderr:+ for f in `seq 1 32` 2024-09-16T04:13:01.483 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm h32 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.560+0000 7f382c766640 10 monclient: get_monmap_and_config 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.560+0000 7f382c766640 10 monclient: build_initial_monmap 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.560+0000 7f382c766640 10 monclient: monmap: 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:01.562060+0000 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:01.562060+0000 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:01.563 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:01.564 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:01.564 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.560+0000 7f382c766640 1 Processor -- start 2024-09-16T04:13:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.560+0000 7f382c766640 1 -- start start 2024-09-16T04:13:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.560+0000 7f382c766640 10 monclient: init 2024-09-16T04:13:01.572 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient(hunting): picked mon.noname-b con 0x7f3824132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient(hunting): picked mon.noname-a con 0x7f382409ec20 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient(hunting): picked mon.noname-c con 0x7f3824132e60 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38241332b0 con 0x7f382409ec20 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3824134310 con 0x7f3824132e60 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382409d980 con 0x7f3824132a80 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382c766640 10 monclient(hunting): authenticate will time out at 1157.067627s 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382ad03640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f3824132e60 0x7f382413c840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54330/0 (socket says 172.21.15.171:54330) 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382ad03640 1 -- 172.21.15.171:0/1154400952 learned_addr learned my addr 172.21.15.171:0/1154400952 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:01.573 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f382a502640 1 -- 172.21.15.171:0/1154400952 >> v1:172.21.15.148:6790/0 conn(0x7f3824132a80 legacy=0x7f38241334f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3829325648 0 0) 0x7f3824134310 con 0x7f3824132e60 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f3829500640 10 monclient(hunting): _negotiate 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f3829500640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f3829500640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f3829500640 10 monclient(hunting): my global_id is 6415 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.561+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3810003520 con 0x7f3824132e60 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2743986222 0 0) 0x7f3810003520 con 0x7f3824132e60 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 >> v1:172.21.15.148:6790/0 conn(0x7f3824132a80 legacy=0x7f38241334f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 >> v1:172.21.15.148:6789/0 conn(0x7f382409ec20 legacy=0x7f3824133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: _finish_hunting 0 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 monclient: found mon.noname-c 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:01.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382409dc60 con 0x7f3824132e60 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: _finish_auth 0 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: _check_auth_tickets 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 5 monclient: authenticate success, global_id 6415 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f381c001990 con 0x7f3824132e60 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: dump: 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.575 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f381c001c80 con 0x7f3824132e60 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 1 -- 172.21.15.171:0/1154400952 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f381c002050 con 0x7f3824132e60 2024-09-16T04:13:01.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 4 set_mon_vals no callback set 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f3829500640 10 monclient: dump: 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:01.577 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 10 monclient: shutdown 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 1 -- 172.21.15.171:0/1154400952 >> v1:172.21.15.171:6789/0 conn(0x7f3824132e60 legacy=0x7f382413c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 1 -- 172.21.15.171:0/1154400952 shutdown_connections 2024-09-16T04:13:01.578 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 1 -- 172.21.15.171:0/1154400952 >> 172.21.15.171:0/1154400952 conn(0x7f38240a52a0 msgr2=0x7f38240a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 1 -- 172.21.15.171:0/1154400952 shutdown_connections 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.562+0000 7f382c766640 1 -- 172.21.15.171:0/1154400952 wait complete. 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.563+0000 7f382c766640 10 monclient: build_initial_monmap 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.563+0000 7f382c766640 10 monclient: monmap: 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:01.579 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.563+0000 7f382c766640 1 Processor -- start 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.563+0000 7f382c766640 1 -- start start 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.563+0000 7f382c766640 10 monclient: init 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient(hunting): picked mon.noname-b con 0x7f382409ec20 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:01.580 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient(hunting): picked mon.noname-c con 0x7f3824132a80 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient(hunting): picked mon.noname-a con 0x7f3824132e60 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382411c0b0 con 0x7f3824132e60 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382411e840 con 0x7f382409ec20 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38240a3d20 con 0x7f3824132a80 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient(hunting): authenticate will time out at 1157.070182s 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f3829d01640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f3824132a80 legacy=0x7f382411b9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382a502640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f382409ec20 0x7f382411b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54342/0 (socket says 172.21.15.171:54342) 2024-09-16T04:13:01.581 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382a502640 1 -- 172.21.15.171:0/1025992940 learned_addr learned my addr 172.21.15.171:0/1025992940 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221484737 0 0) 0x7f382411e840 con 0x7f382409ec20 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): my global_id is 6418 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3808003520 con 0x7f382409ec20 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2581066185 0 0) 0x7f382411c0b0 con 0x7f3824132e60 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): my global_id is 64857 2024-09-16T04:13:01.582 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f382411e840 con 0x7f3824132e60 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2752500467 0 0) 0x7f3808003520 con 0x7f382409ec20 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f382411c0b0 con 0x7f382409ec20 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f38200041d0 con 0x7f382409ec20 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): dump: 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.583 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:01.584 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:01.584 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:01.584 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 9884828 0 0) 0x7f382411c0b0 con 0x7f382409ec20 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 >> v1:172.21.15.148:6790/0 conn(0x7f3824132a80 legacy=0x7f382411b9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 >> v1:172.21.15.148:6789/0 conn(0x7f3824132e60 legacy=0x7f38241cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 monclient: found mon.b 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382409c910 con 0x7f382409ec20 2024-09-16T04:13:01.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: _finish_auth 0 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 5 monclient: authenticate success, global_id 6418 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient: _renew_subs 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f382406ceb0 con 0x7f382409ec20 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient: _renew_subs 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f382411e560 con 0x7f382409ec20 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3820004480 con 0x7f382409ec20 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3820004850 con 0x7f382409ec20 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:01.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f3828cff640 4 set_mon_vals no callback set 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.564+0000 7f381affd640 10 monclient: dump: 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:01.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.565+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f38200049f0 con 0x7f382409ec20 2024-09-16T04:13:01.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.565+0000 7f382c766640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:01.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.565+0000 7f382c766640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:01.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.565+0000 7f382c766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.565+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f380c005080 con 0x7f382409ec20 2024-09-16T04:13:01.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.566+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(246..246 src has 1..246) ==== 7568+0+0 (unknown 276540308 0 0) 0x7f3820047d30 con 0x7f382409ec20 2024-09-16T04:13:01.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.566+0000 7f3829d01640 10 monclient: get_auth_request con 0x7f3808038be0 auth_method 0 2024-09-16T04:13:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.570+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3820015cb0 con 0x7f382409ec20 2024-09-16T04:13:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.570+0000 7f381affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:01.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.570+0000 7f381affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:01.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.740+0000 7f382c766640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T04:13:01.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.740+0000 7f382c766640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T04:13:01.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.740+0000 7f382c766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:01.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:01.740+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) -- 0x7f380c005370 con 0x7f382409ec20 2024-09-16T04:13:02.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.303+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v247) ==== 121+0+0 (unknown 4085141497 0 0) 0x7f38200155b0 con 0x7f382409ec20 2024-09-16T04:13:02.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.303+0000 7f381affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T04:13:02.306 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.303+0000 7f381affd640 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-16T04:13:02.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.407+0000 7f382c766640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T04:13:02.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.407+0000 7f382c766640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T04:13:02.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.407+0000 7f382c766640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.410 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.407+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) -- 0x7f380c002210 con 0x7f382409ec20 2024-09-16T04:13:02.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.408+0000 7f381affd640 1 -- 172.21.15.171:0/1025992940 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v247) ==== 120+0+0 (unknown 1286066567 0 0) 0x7f3820019900 con 0x7f382409ec20 2024-09-16T04:13:02.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.408+0000 7f381affd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-16T04:13:02.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.408+0000 7f381affd640 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-16T04:13:02.411 INFO:tasks.workunit.client.0.smithi171.stderr:device 'h32' does not appear in the crush map 2024-09-16T04:13:02.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 >> v1:172.21.15.171:6812/7327273 conn(0x7f3808038be0 legacy=0x7f380803b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:02.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 10 monclient: shutdown 2024-09-16T04:13:02.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:02.413 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 >> v1:172.21.15.171:6789/0 conn(0x7f382409ec20 legacy=0x7f382411b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:02.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 shutdown_connections 2024-09-16T04:13:02.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 >> 172.21.15.171:0/1025992940 conn(0x7f38240a52a0 msgr2=0x7f38240a41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:02.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 shutdown_connections 2024-09-16T04:13:02.414 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.410+0000 7f382c766640 1 -- 172.21.15.171:0/1025992940 wait complete. 2024-09-16T04:13:02.423 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush rm r1 2024-09-16T04:13:02.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.498+0000 7fa7d8035640 10 monclient: get_monmap_and_config 2024-09-16T04:13:02.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.498+0000 7fa7d8035640 10 monclient: build_initial_monmap 2024-09-16T04:13:02.502 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.498+0000 7fa7d8035640 10 monclient: monmap: 2024-09-16T04:13:02.502 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:02.502 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:02.503 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:02.500383+0000 2024-09-16T04:13:02.503 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:02.500383+0000 2024-09-16T04:13:02.503 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:02.503 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:02.503 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:02.504 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:02.504 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:02.504 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:02.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 1 Processor -- start 2024-09-16T04:13:02.504 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 1 -- start start 2024-09-16T04:13:02.505 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient: init 2024-09-16T04:13:02.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:02.522 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:02.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d009eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:02.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d0134b20 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:02.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d0134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:02.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d009d060 con 0x7fa7d0134b20 2024-09-16T04:13:02.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d009d340 con 0x7fa7d0134f30 2024-09-16T04:13:02.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d0068840 con 0x7fa7d009eaa0 2024-09-16T04:13:02.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:02.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d8035640 10 monclient(hunting): authenticate will time out at 1158.005964s 2024-09-16T04:13:02.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.499+0000 7fa7d5dd1640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa7d009eaa0 legacy=0x7fa7d009ee80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:02.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d65d2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa7d0134f30 0x7fa7d013d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54344/0 (socket says 172.21.15.171:54344) 2024-09-16T04:13:02.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d65d2640 1 -- 172.21.15.171:0/4167686804 learned_addr learned my addr 172.21.15.171:0/4167686804 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:02.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341840331 0 0) 0x7fa7d009d340 con 0x7fa7d0134f30 2024-09-16T04:13:02.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): _negotiate 2024-09-16T04:13:02.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:02.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:02.525 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): my global_id is 6421 2024-09-16T04:13:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7bc003520 con 0x7fa7d0134f30 2024-09-16T04:13:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708288478 0 0) 0x7fa7d009d060 con 0x7fa7d0134b20 2024-09-16T04:13:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): _negotiate 2024-09-16T04:13:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:02.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient(hunting): my global_id is 64860 2024-09-16T04:13:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d009d340 con 0x7fa7d0134b20 2024-09-16T04:13:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2074831295 0 0) 0x7fa7bc003520 con 0x7fa7d0134f30 2024-09-16T04:13:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 >> v1:172.21.15.148:6790/0 conn(0x7fa7d009eaa0 legacy=0x7fa7d009ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 >> v1:172.21.15.148:6789/0 conn(0x7fa7d0134b20 legacy=0x7fa7d0139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:02.527 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: _finish_hunting 0 2024-09-16T04:13:02.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 monclient: found mon.noname-c 2024-09-16T04:13:02.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:02.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d0141800 con 0x7fa7d0134f30 2024-09-16T04:13:02.528 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: _finish_auth 0 2024-09-16T04:13:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: _check_auth_tickets 2024-09-16T04:13:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa7c8001990 con 0x7fa7d0134f30 2024-09-16T04:13:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d8035640 5 monclient: authenticate success, global_id 6421 2024-09-16T04:13:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d8035640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:02.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:02.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:02.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: dump: 2024-09-16T04:13:02.530 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:02.530 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:02.530 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.531 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.531 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:02.531 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:02.531 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:02.531 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:02.531 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:02.532 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:02.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d8035640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:02.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7c8001c40 con 0x7fa7d0134f30 2024-09-16T04:13:02.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:02.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 1 -- 172.21.15.171:0/4167686804 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa7c8002010 con 0x7fa7d0134f30 2024-09-16T04:13:02.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d8035640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:02.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d8035640 4 set_mon_vals no callback set 2024-09-16T04:13:02.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:02.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:02.534 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.500+0000 7fa7d4dcf640 10 monclient: dump: 2024-09-16T04:13:02.534 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:02.534 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:02.534 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.534 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.535 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:02.535 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:02.535 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:02.535 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:02.535 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:02.536 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:02.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 10 monclient: shutdown 2024-09-16T04:13:02.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:02.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 1 -- 172.21.15.171:0/4167686804 >> v1:172.21.15.171:6789/0 conn(0x7fa7d0134f30 legacy=0x7fa7d013d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:02.536 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 1 -- 172.21.15.171:0/4167686804 shutdown_connections 2024-09-16T04:13:02.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 1 -- 172.21.15.171:0/4167686804 >> 172.21.15.171:0/4167686804 conn(0x7fa7d00a1e90 msgr2=0x7fa7d00a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:02.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 1 -- 172.21.15.171:0/4167686804 shutdown_connections 2024-09-16T04:13:02.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 1 -- 172.21.15.171:0/4167686804 wait complete. 2024-09-16T04:13:02.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 10 monclient: build_initial_monmap 2024-09-16T04:13:02.537 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.501+0000 7fa7d8035640 10 monclient: monmap: 2024-09-16T04:13:02.537 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:02.538 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:02.538 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:02.538 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:02.538 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:02.538 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:02.538 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:02.539 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:02.539 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:02.539 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:02.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 1 Processor -- start 2024-09-16T04:13:02.539 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 1 -- start start 2024-09-16T04:13:02.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient: init 2024-09-16T04:13:02.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:02.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:02.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d009eaa0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:02.540 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d0134b20 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:02.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d0134f30 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:02.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d0141290 con 0x7fa7d0134b20 2024-09-16T04:13:02.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d009c580 con 0x7fa7d0134f30 2024-09-16T04:13:02.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d00a3400 con 0x7fa7d009eaa0 2024-09-16T04:13:02.541 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:02.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d8035640 10 monclient(hunting): authenticate will time out at 1158.008561s 2024-09-16T04:13:02.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d5dd1640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa7d009eaa0 legacy=0x7fa7d01cb810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:02.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d65d2640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa7d0134f30 0x7fa7d01d3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54354/0 (socket says 172.21.15.171:54354) 2024-09-16T04:13:02.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7d65d2640 1 -- 172.21.15.171:0/643312485 learned_addr learned my addr 172.21.15.171:0/643312485 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:02.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443834501 0 0) 0x7fa7d0141800 con 0x7fa7d0134f30 2024-09-16T04:13:02.542 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:13:02.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:02.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:02.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 10 monclient(hunting): my global_id is 6424 2024-09-16T04:13:02.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7b4003520 con 0x7fa7d0134f30 2024-09-16T04:13:02.543 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.502+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2747587319 0 0) 0x7fa7b4003520 con 0x7fa7d0134f30 2024-09-16T04:13:02.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7d0141800 con 0x7fa7d0134f30 2024-09-16T04:13:02.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa7c8001f90 con 0x7fa7d0134f30 2024-09-16T04:13:02.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:02.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:02.544 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): dump: 2024-09-16T04:13:02.545 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:02.545 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:02.545 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.545 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.545 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:02.545 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:02.546 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:02.546 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:02.546 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:02.546 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:02.546 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1284506323 0 0) 0x7fa7d009d340 con 0x7fa7d0134b20 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient(hunting): my global_id is 64863 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7b4003520 con 0x7fa7d0134b20 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2426478696 0 0) 0x7fa7d0141800 con 0x7fa7d0134f30 2024-09-16T04:13:02.547 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 >> v1:172.21.15.148:6790/0 conn(0x7fa7d009eaa0 legacy=0x7fa7d01cb810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 >> v1:172.21.15.148:6789/0 conn(0x7fa7d0134b20 legacy=0x7fa7d013daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 monclient: found mon.b 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d013bfb0 con 0x7fa7d0134f30 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 5 monclient: authenticate success, global_id 6424 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 10 monclient: _renew_subs 2024-09-16T04:13:02.548 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d013e1b0 con 0x7fa7d0134f30 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 10 monclient: _renew_subs 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7d01d5450 con 0x7fa7d0134f30 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa7c8001c40 con 0x7fa7d0134f30 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa7c80051e0 con 0x7fa7d0134f30 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c7fff640 4 set_mon_vals no callback set 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 10 monclient: dump: 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:02.549 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.503+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa7c8005380 con 0x7fa7d0134f30 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.504+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(247..247 src has 1..247) ==== 7355+0+0 (unknown 4077946237 0 0) 0x7fa7c80480c0 con 0x7fa7d0134f30 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.504+0000 7fa7d8035640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.504+0000 7fa7d5dd1640 10 monclient: get_auth_request con 0x7fa7b4038fc0 auth_method 0 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.504+0000 7fa7d8035640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:02.550 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.504+0000 7fa7d8035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.504+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b8005080 con 0x7fa7d0134f30 2024-09-16T04:13:02.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.509+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa7d009c580 con 0x7fa7d0134f30 2024-09-16T04:13:02.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.509+0000 7fa7c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:02.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.509+0000 7fa7c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:02.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.677+0000 7fa7d8035640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T04:13:02.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.677+0000 7fa7d8035640 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T04:13:02.680 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.677+0000 7fa7d8035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:02.681 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:02.677+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) -- 0x7fa7b8005370 con 0x7fa7d0134f30 2024-09-16T04:13:03.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.375+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v248) ==== 118+0+0 (unknown 3450644807 0 0) 0x7fa7c80161d0 con 0x7fa7d0134f30 2024-09-16T04:13:03.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.375+0000 7fa7c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T04:13:03.378 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.375+0000 7fa7c67fc640 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-16T04:13:03.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.480+0000 7fa7d8035640 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T04:13:03.483 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.480+0000 7fa7d8035640 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T04:13:03.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.480+0000 7fa7d8035640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.480+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) -- 0x7fa7b8002210 con 0x7fa7d0134f30 2024-09-16T04:13:03.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.481+0000 7fa7c67fc640 1 -- 172.21.15.171:0/643312485 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v248) ==== 118+0+0 (unknown 2396752965 0 0) 0x7fa7c8015ad0 con 0x7fa7d0134f30 2024-09-16T04:13:03.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.481+0000 7fa7c67fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-16T04:13:03.484 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.481+0000 7fa7c67fc640 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-16T04:13:03.484 INFO:tasks.workunit.client.0.smithi171.stderr:device 'r1' does not appear in the crush map 2024-09-16T04:13:03.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 >> v1:172.21.15.171:6812/7327273 conn(0x7fa7b4038fc0 legacy=0x7fa7b403b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:03.486 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 10 monclient: shutdown 2024-09-16T04:13:03.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:03.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 >> v1:172.21.15.171:6789/0 conn(0x7fa7d0134f30 legacy=0x7fa7d01d3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:03.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 shutdown_connections 2024-09-16T04:13:03.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 >> 172.21.15.171:0/643312485 conn(0x7fa7d00a1e90 msgr2=0x7fa7d01407d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:03.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 shutdown_connections 2024-09-16T04:13:03.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.483+0000 7fa7d8035640 1 -- 172.21.15.171:0/643312485 wait complete. 2024-09-16T04:13:03.496 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd crush weight-set rm-compat 2024-09-16T04:13:03.574 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.571+0000 7fb4b8e1e640 10 monclient: get_monmap_and_config 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.571+0000 7fb4b8e1e640 10 monclient: build_initial_monmap 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.571+0000 7fb4b8e1e640 10 monclient: monmap: 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:03.573562+0000 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:03.573562+0000 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:03.575 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:03.576 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:03.576 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 1 Processor -- start 2024-09-16T04:13:03.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 1 -- start start 2024-09-16T04:13:03.576 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient: init 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b409ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b40a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b409e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b409fbf0 con 0x7fb4b40a0750 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b40a0e10 con 0x7fb4b409e570 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b409f640 con 0x7fb4b409ba30 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:03.584 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.572+0000 7fb4b8e1e640 10 monclient(hunting): authenticate will time out at 1159.078927s 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b259c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb4b409ba30 legacy=0x7fb4b40a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b2d9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb4b409e570 0x7fb4b413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54358/0 (socket says 172.21.15.171:54358) 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b2d9d640 1 -- 172.21.15.171:0/575084071 learned_addr learned my addr 172.21.15.171:0/575084071 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936836821 0 0) 0x7fb4b40a0e10 con 0x7fb4b409e570 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): my global_id is 6427 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4a0003520 con 0x7fb4b409e570 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193880239 0 0) 0x7fb4b409fbf0 con 0x7fb4b40a0750 2024-09-16T04:13:03.585 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient(hunting): my global_id is 64866 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b40a0e10 con 0x7fb4b40a0750 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2733733793 0 0) 0x7fb4a0003520 con 0x7fb4b409e570 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 >> v1:172.21.15.148:6790/0 conn(0x7fb4b409ba30 legacy=0x7fb4b40a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 >> v1:172.21.15.148:6789/0 conn(0x7fb4b40a0750 legacy=0x7fb4b41399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 monclient: found mon.noname-c 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:03.586 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b409f810 con 0x7fb4b409e570 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: _finish_auth 0 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b8e1e640 5 monclient: authenticate success, global_id 6427 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4ac001990 con 0x7fb4b409e570 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: dump: 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:03.587 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b8e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb4ac001bf0 con 0x7fb4b409e570 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:03.588 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 1 -- 172.21.15.171:0/575084071 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4ac001fc0 con 0x7fb4b409e570 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b8e1e640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b8e1e640 4 set_mon_vals no callback set 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.573+0000 7fb4b1d9b640 10 monclient: dump: 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:03.589 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.590 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:03.590 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:03.590 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:03.590 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.590 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 10 monclient: shutdown 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/575084071 >> v1:172.21.15.171:6789/0 conn(0x7fb4b409e570 legacy=0x7fb4b413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/575084071 shutdown_connections 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/575084071 >> 172.21.15.171:0/575084071 conn(0x7fb4b40a52a0 msgr2=0x7fb4b40a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/575084071 shutdown_connections 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/575084071 wait complete. 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 10 monclient: build_initial_monmap 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.574+0000 7fb4b8e1e640 10 monclient: monmap: 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:03.591 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 1 Processor -- start 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 1 -- start start 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient: init 2024-09-16T04:13:03.592 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:03.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:03.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b409ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:03.593 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b409e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b40a0750 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b4072050 con 0x7fb4b409ba30 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b4130980 con 0x7fb4b409e570 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b41327a0 con 0x7fb4b40a0750 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b8e1e640 10 monclient(hunting): authenticate will time out at 1159.081942s 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.575+0000 7fb4b2d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb4b40a0750 legacy=0x7fb4b41d3d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4a9d9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb4b409e570 0x7fb4b4130270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54362/0 (socket says 172.21.15.171:54362) 2024-09-16T04:13:03.594 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4a9d9b640 1 -- 172.21.15.171:0/4250213220 learned_addr learned my addr 172.21.15.171:0/4250213220 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4218240109 0 0) 0x7fb4b4130980 con 0x7fb4b409e570 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): my global_id is 6430 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb494003520 con 0x7fb4b409e570 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168207964 0 0) 0x7fb4b4072050 con 0x7fb4b409ba30 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:03.595 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): my global_id is 64869 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4b4130980 con 0x7fb4b409ba30 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2697293729 0 0) 0x7fb494003520 con 0x7fb4b409e570 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb4b4072050 con 0x7fb4b409e570 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb49c001a00 con 0x7fb4b409e570 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): dump: 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.596 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1141652815 0 0) 0x7fb4b4130980 con 0x7fb4b409ba30 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb494003520 con 0x7fb4b409ba30 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb4a4004140 con 0x7fb4b409ba30 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient(hunting): dump: 2024-09-16T04:13:03.597 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 156636595 0 0) 0x7fb4b4072050 con 0x7fb4b409e570 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 >> v1:172.21.15.148:6790/0 conn(0x7fb4b40a0750 legacy=0x7fb4b41d3d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 >> v1:172.21.15.148:6789/0 conn(0x7fb4b409ba30 legacy=0x7fb4b412fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:03.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 monclient: found mon.b 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b409c920 con 0x7fb4b409e570 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 5 monclient: authenticate success, global_id 6430 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 10 monclient: _renew_subs 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b409d590 con 0x7fb4b409e570 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 10 monclient: _renew_subs 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb4b409cfd0 con 0x7fb4b409e570 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb49c0038c0 con 0x7fb4b409e570 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb49c003c10 con 0x7fb4b409e570 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4b159a640 4 set_mon_vals no callback set 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.576+0000 7fb4ab7fe640 10 monclient: dump: 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:03.600 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb49c003e30 con 0x7fb4b409e570 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4b8e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4b8e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4b8e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb498005080 con 0x7fb4b409e570 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(248..248 src has 1..248) ==== 7145+0+0 (unknown 866944751 0 0) 0x7fb49c047d10 con 0x7fb4b409e570 2024-09-16T04:13:03.601 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.577+0000 7fb4b259c640 10 monclient: get_auth_request con 0x7fb494038ea0 auth_method 0 2024-09-16T04:13:03.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.582+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb49c015df0 con 0x7fb4b409e570 2024-09-16T04:13:03.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.582+0000 7fb4ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:03.602 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.582+0000 7fb4ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:03.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.753+0000 7fb4b8e1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:13:03.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.753+0000 7fb4b8e1e640 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:13:03.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.753+0000 7fb4b8e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:03.756 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:03.753+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fb498005370 con 0x7fb4b409e570 2024-09-16T04:13:04.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.376+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v249) ==== 78+0+0 (unknown 3129046136 0 0) 0x7fb49c0156f0 con 0x7fb4b409e570 2024-09-16T04:13:04.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.376+0000 7fb4ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:13:04.379 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.376+0000 7fb4ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:04.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.485+0000 7fb4b8e1e640 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:13:04.487 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.485+0000 7fb4b8e1e640 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:13:04.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.485+0000 7fb4b8e1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:04.488 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:04.485+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) -- 0x7fb498002210 con 0x7fb4b409e570 2024-09-16T04:13:05.388 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.386+0000 7fb4ab7fe640 1 -- 172.21.15.171:0/4250213220 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v250) ==== 78+0+0 (unknown 2026023447 0 0) 0x7fb49c019a40 con 0x7fb4b409e570 2024-09-16T04:13:05.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.386+0000 7fb4ab7fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-16T04:13:05.389 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.386+0000 7fb4ab7fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:13:05.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.388+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 >> v1:172.21.15.171:6812/7327273 conn(0x7fb494038ea0 legacy=0x7fb49403b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:05.391 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.388+0000 7fb4b8e1e640 10 monclient: shutdown 2024-09-16T04:13:05.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.388+0000 7fb4b8e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:05.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.388+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 >> v1:172.21.15.171:6789/0 conn(0x7fb4b409e570 legacy=0x7fb4b4130270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:05.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.389+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 shutdown_connections 2024-09-16T04:13:05.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.389+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 >> 172.21.15.171:0/4250213220 conn(0x7fb4b40a52a0 msgr2=0x7fb4b4131c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:05.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.389+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 shutdown_connections 2024-09-16T04:13:05.392 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.389+0000 7fb4b8e1e640 1 -- 172.21.15.171:0/4250213220 wait complete. 2024-09-16T04:13:05.403 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo OK 2024-09-16T04:13:05.404 INFO:tasks.workunit.client.0.smithi171.stdout:OK 2024-09-16T04:13:05.404 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T04:13:05.404 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T04:13:05.474 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-16T04:13:05.475 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-16T04:13:05.475 DEBUG:teuthology.orchestra.run.smithi171:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-16T04:13:05.536 INFO:tasks.workunit.client.0.smithi171.stderr:+ set -e 2024-09-16T04:13:05.536 INFO:tasks.workunit.client.0.smithi171.stderr:++ uuidgen 2024-09-16T04:13:05.610 INFO:tasks.workunit.client.0.smithi171.stderr:+ ua=6df951c8-ca70-48bb-b32a-2b7a6ec9adff 2024-09-16T04:13:05.610 INFO:tasks.workunit.client.0.smithi171.stderr:++ uuidgen 2024-09-16T04:13:05.611 INFO:tasks.workunit.client.0.smithi171.stderr:+ ub=81bbd316-374e-4988-a68f-0955f1e1326d 2024-09-16T04:13:05.611 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 6df951c8-ca70-48bb-b32a-2b7a6ec9adff 2024-09-16T04:13:05.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.691+0000 7f82e4ed7640 10 monclient: get_monmap_and_config 2024-09-16T04:13:05.693 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.691+0000 7f82e4ed7640 10 monclient: build_initial_monmap 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.691+0000 7f82e4ed7640 10 monclient: monmap: 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:05.692776+0000 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:05.692776+0000 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:05.694 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:05.701 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.691+0000 7f82e4ed7640 1 Processor -- start 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.691+0000 7f82e4ed7640 1 -- start start 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.691+0000 7f82e4ed7640 10 monclient: init 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient(hunting): picked mon.noname-b con 0x7f82e009ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient(hunting): picked mon.noname-c con 0x7f82e00a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient(hunting): picked mon.noname-a con 0x7f82e009e570 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e009fbf0 con 0x7f82e009e570 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e00a0e10 con 0x7f82e00a0750 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e009f640 con 0x7f82e009ba30 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:05.702 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82e4ed7640 10 monclient(hunting): authenticate will time out at 1161.198470s 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82de59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f82e009ba30 legacy=0x7f82e00a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82ddd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f82e00a0750 0x7f82e01399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54368/0 (socket says 172.21.15.171:54368) 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82ddd9b640 1 -- 172.21.15.171:0/486060154 learned_addr learned my addr 172.21.15.171:0/486060154 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203649351 0 0) 0x7f82e00a0e10 con 0x7f82e00a0750 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 10 monclient(hunting): _negotiate 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 10 monclient(hunting): my global_id is 6433 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82c8003520 con 0x7f82e00a0750 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.692+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3043760526 0 0) 0x7f82c8003520 con 0x7f82e00a0750 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 >> v1:172.21.15.148:6790/0 conn(0x7f82e009ba30 legacy=0x7f82e00a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 >> v1:172.21.15.148:6789/0 conn(0x7f82e009e570 legacy=0x7f82e013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:05.703 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: _finish_hunting 0 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 monclient: found mon.noname-c 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e009f810 con 0x7f82e00a0750 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: _finish_auth 0 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: _check_auth_tickets 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f82d4001610 con 0x7f82e00a0750 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: dump: 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:05.704 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 5 monclient: authenticate success, global_id 6433 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f82d4001970 con 0x7f82e00a0750 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 1 -- 172.21.15.171:0/486060154 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f82d40048a0 con 0x7f82e00a0750 2024-09-16T04:13:05.705 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82dd59a640 10 monclient: dump: 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:05.706 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 4 set_mon_vals no callback set 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 10 monclient: shutdown 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 1 -- 172.21.15.171:0/486060154 >> v1:172.21.15.171:6789/0 conn(0x7f82e00a0750 legacy=0x7f82e01399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 1 -- 172.21.15.171:0/486060154 shutdown_connections 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 1 -- 172.21.15.171:0/486060154 >> 172.21.15.171:0/486060154 conn(0x7f82e00a52a0 msgr2=0x7f82e00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 1 -- 172.21.15.171:0/486060154 shutdown_connections 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.693+0000 7f82e4ed7640 1 -- 172.21.15.171:0/486060154 wait complete. 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.694+0000 7f82e4ed7640 10 monclient: build_initial_monmap 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.694+0000 7f82e4ed7640 10 monclient: monmap: 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:05.707 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.694+0000 7f82e4ed7640 1 Processor -- start 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.694+0000 7f82e4ed7640 1 -- start start 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.694+0000 7f82e4ed7640 10 monclient: init 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient(hunting): picked mon.noname-a con 0x7f82e009ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:05.708 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient(hunting): picked mon.noname-b con 0x7f82e009e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient(hunting): picked mon.noname-c con 0x7f82e00a0750 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e009d000 con 0x7f82e009ba30 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e012f8c0 con 0x7f82e009e570 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e009ddc0 con 0x7f82e00a0750 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82e4ed7640 10 monclient(hunting): authenticate will time out at 1161.201422s 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ded9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f82e00a0750 legacy=0x7f82e01d3b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ddd9b640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f82e009e570 0x7f82e009c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54376/0 (socket says 172.21.15.171:54376) 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ddd9b640 1 -- 172.21.15.171:0/1628064971 learned_addr learned my addr 172.21.15.171:0/1628064971 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791975087 0 0) 0x7f82e012f8c0 con 0x7f82e009e570 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:05.709 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): my global_id is 6436 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82c0003520 con 0x7f82e009e570 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835072102 0 0) 0x7f82e009d000 con 0x7f82e009ba30 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): my global_id is 64875 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e012f8c0 con 0x7f82e009ba30 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 263889078 0 0) 0x7f82c0003520 con 0x7f82e009e570 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82e009d000 con 0x7f82e009e570 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f82d4001430 con 0x7f82e009e570 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:05.710 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): dump: 2024-09-16T04:13:05.711 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:05.711 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:05.711 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.711 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.711 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:05.712 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:05.712 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:05.712 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:05.712 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:05.713 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:05.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:05.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3803079835 0 0) 0x7f82e009d000 con 0x7f82e009e570 2024-09-16T04:13:05.713 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 >> v1:172.21.15.148:6790/0 conn(0x7f82e00a0750 legacy=0x7f82e01d3b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:05.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.695+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 >> v1:172.21.15.148:6789/0 conn(0x7f82e009ba30 legacy=0x7f82e009c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:05.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:05.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 1 monclient: found mon.b 2024-09-16T04:13:05.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:05.714 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:05.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e0130d10 con 0x7f82e009e570 2024-09-16T04:13:05.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:05.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:05.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:05.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 5 monclient: authenticate success, global_id 6436 2024-09-16T04:13:05.715 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 monclient: _renew_subs 2024-09-16T04:13:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f82e012fe20 con 0x7f82e009e570 2024-09-16T04:13:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 monclient: _renew_subs 2024-09-16T04:13:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f82e01319d0 con 0x7f82e009e570 2024-09-16T04:13:05.716 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f82d4001690 con 0x7f82e009e570 2024-09-16T04:13:05.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:05.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f82d4004e60 con 0x7f82e009e570 2024-09-16T04:13:05.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:05.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:05.717 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82cffff640 4 set_mon_vals no callback set 2024-09-16T04:13:05.718 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 10 monclient: dump: 2024-09-16T04:13:05.718 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:05.718 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:05.718 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.719 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:05.719 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:05.719 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:05.719 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:05.719 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:05.719 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:05.720 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:05.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:05.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f82d4010290 con 0x7f82e009e570 2024-09-16T04:13:05.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:05.720 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:05.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.696+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c4005080 con 0x7f82e009e570 2024-09-16T04:13:05.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.697+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(250..250 src has 1..250) ==== 6817+0+0 (unknown 3412434972 0 0) 0x7f82d4047ab0 con 0x7f82e009e570 2024-09-16T04:13:05.721 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.697+0000 7f82de59c640 10 monclient: get_auth_request con 0x7f82c0039150 auth_method 0 2024-09-16T04:13:05.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.702+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f82d4015dd0 con 0x7f82e009e570 2024-09-16T04:13:05.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.702+0000 7f82ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:05.722 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.702+0000 7f82ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:05.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.879+0000 7f82e4ed7640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:05.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.879+0000 7f82e4ed7640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:05.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.879+0000 7f82e4ed7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:05.882 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:05.879+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"} v 0) -- 0x7f82c4005370 con 0x7f82e009e570 2024-09-16T04:13:06.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.387+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}]=0 v251) ==== 106+0+3 (unknown 1245935979 0 944643765) 0x7f82d40156d0 con 0x7f82e009e570 2024-09-16T04:13:06.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.387+0000 7f82ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.390 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.387+0000 7f82ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:06.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.506+0000 7f82e4ed7640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.508 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.506+0000 7f82e4ed7640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.506+0000 7f82e4ed7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.506+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"} v 0) -- 0x7f82c4005bd0 con 0x7f82e009e570 2024-09-16T04:13:06.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.507+0000 7f82ceffd640 1 -- 172.21.15.171:0/1628064971 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}]=0 v251) ==== 106+0+3 (unknown 1245935979 0 944643765) 0x7f82d401e220 con 0x7f82e009e570 2024-09-16T04:13:06.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.507+0000 7f82ceffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.507+0000 7f82ceffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 >> v1:172.21.15.171:6812/7327273 conn(0x7f82c0039150 legacy=0x7f82c003b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 10 monclient: shutdown 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 >> v1:172.21.15.171:6789/0 conn(0x7f82e009e570 legacy=0x7f82e009c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 shutdown_connections 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 >> 172.21.15.171:0/1628064971 conn(0x7f82e00a52a0 msgr2=0x7f82e0133fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 shutdown_connections 2024-09-16T04:13:06.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.510+0000 7f82e4ed7640 1 -- 172.21.15.171:0/1628064971 wait complete. 2024-09-16T04:13:06.521 INFO:tasks.workunit.client.0.smithi171.stderr:+ na=12 2024-09-16T04:13:06.521 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 6df951c8-ca70-48bb-b32a-2b7a6ec9adff 2024-09-16T04:13:06.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.597+0000 7fc87115f640 10 monclient: get_monmap_and_config 2024-09-16T04:13:06.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.597+0000 7fc87115f640 10 monclient: build_initial_monmap 2024-09-16T04:13:06.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.597+0000 7fc87115f640 10 monclient: monmap: 2024-09-16T04:13:06.599 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:06.599 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:06.598463+0000 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:06.598463+0000 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:06.600 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 1 Processor -- start 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 1 -- start start 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient: init 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c09f640 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c0a0130 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:06.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c0a1260 con 0x7fc86c09f640 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c09d0c0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c09d3a0 con 0x7fc86c0a0130 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc87115f640 10 monclient(hunting): authenticate will time out at 1162.103883s 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc86b59e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc86c09e1d0 0x7fc86c13bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54386/0 (socket says 172.21.15.171:54386) 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc86b59e640 1 -- 172.21.15.171:0/4232834379 learned_addr learned my addr 172.21.15.171:0/4232834379 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.598+0000 7fc86259c640 1 -- 172.21.15.171:0/4232834379 >> v1:172.21.15.148:6790/0 conn(0x7fc86c0a0130 legacy=0x7fc86c138440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810638586 0 0) 0x7fc86c09d0c0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient(hunting): _negotiate 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient(hunting): my global_id is 6439 2024-09-16T04:13:06.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc858004630 con 0x7fc86c09e1d0 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 946422414 0 0) 0x7fc858004630 con 0x7fc86c09e1d0 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 >> v1:172.21.15.148:6790/0 conn(0x7fc86c0a0130 legacy=0x7fc86c138440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 >> v1:172.21.15.148:6789/0 conn(0x7fc86c09f640 legacy=0x7fc86c09fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 monclient: found mon.noname-c 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c09d680 con 0x7fc86c09e1d0 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: _finish_auth 0 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: _check_auth_tickets 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:06.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc864001990 con 0x7fc86c09e1d0 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc87115f640 5 monclient: authenticate success, global_id 6439 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc87115f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: dump: 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:06.613 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc87115f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc864001cc0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 1 -- 172.21.15.171:0/4232834379 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc864002090 con 0x7fc86c09e1d0 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc87115f640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:06.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc87115f640 4 set_mon_vals no callback set 2024-09-16T04:13:06.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:06.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:06.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.599+0000 7fc86a59c640 10 monclient: dump: 2024-09-16T04:13:06.615 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:06.615 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:06.616 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:06.617 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 10 monclient: shutdown 2024-09-16T04:13:06.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:06.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 1 -- 172.21.15.171:0/4232834379 >> v1:172.21.15.171:6789/0 conn(0x7fc86c09e1d0 legacy=0x7fc86c13bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 1 -- 172.21.15.171:0/4232834379 shutdown_connections 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 1 -- 172.21.15.171:0/4232834379 >> 172.21.15.171:0/4232834379 conn(0x7fc86c0a52a0 msgr2=0x7fc86c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 1 -- 172.21.15.171:0/4232834379 shutdown_connections 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 1 -- 172.21.15.171:0/4232834379 wait complete. 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 10 monclient: build_initial_monmap 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.600+0000 7fc87115f640 10 monclient: monmap: 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:06.618 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 1 Processor -- start 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 1 -- start start 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient: init 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:06.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c09e1d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c09f640 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c0a0130 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c1c14c0 con 0x7fc86c0a0130 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c1a5c50 con 0x7fc86c09e1d0 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c1a8710 con 0x7fc86c09f640 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc87115f640 10 monclient(hunting): authenticate will time out at 1162.106865s 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc86ad9d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fc86c09e1d0 0x7fc86c1c0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54402/0 (socket says 172.21.15.171:54402) 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc86ad9d640 1 -- 172.21.15.171:0/925242994 learned_addr learned my addr 172.21.15.171:0/925242994 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.601+0000 7fc86259c640 1 -- 172.21.15.171:0/925242994 >> v1:172.21.15.148:6790/0 conn(0x7fc86c09f640 legacy=0x7fc86c1cc0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566418204 0 0) 0x7fc86c1a5c50 con 0x7fc86c09e1d0 2024-09-16T04:13:06.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _negotiate 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): my global_id is 6442 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84c003520 con 0x7fc86c09e1d0 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 175602257 0 0) 0x7fc84c003520 con 0x7fc86c09e1d0 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc86c1a5c50 con 0x7fc86c09e1d0 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc85c0018b0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): dump: 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:06.621 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172733816 0 0) 0x7fc86c1c14c0 con 0x7fc86c0a0130 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _negotiate 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:06.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient(hunting): my global_id is 64878 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84c003520 con 0x7fc86c0a0130 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1190228609 0 0) 0x7fc86c1a5c50 con 0x7fc86c09e1d0 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 >> v1:172.21.15.148:6790/0 conn(0x7fc86c09f640 legacy=0x7fc86c1cc0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 >> v1:172.21.15.148:6789/0 conn(0x7fc86c0a0130 legacy=0x7fc86c1cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 monclient: found mon.b 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c0a4140 con 0x7fc86c09e1d0 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: _finish_auth 0 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:06.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 5 monclient: authenticate success, global_id 6442 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 10 monclient: _renew_subs 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc86c06ceb0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 10 monclient: _renew_subs 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc86c1a8430 con 0x7fc86c09e1d0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc85c001ce0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc85c003ae0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:06.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc863fff640 10 monclient: dump: 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:06.625 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.602+0000 7fc869d9b640 4 set_mon_vals no callback set 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc85c003d00 con 0x7fc86c09e1d0 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc87115f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc87115f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc87115f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86c0008d0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(251..251 src has 1..251) ==== 6817+0+0 (unknown 319184596 0 0) 0x7fc85c047b20 con 0x7fc86c09e1d0 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.603+0000 7fc86259c640 10 monclient: get_auth_request con 0x7fc84c038f40 auth_method 0 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.608+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc85c015dc0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.608+0000 7fc863fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:06.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.608+0000 7fc863fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:06.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.786+0000 7fc87115f640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.786+0000 7fc87115f640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.786+0000 7fc87115f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.786+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"} v 0) -- 0x7fc86c1a71c0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.787+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}]=0 v251) ==== 106+0+3 (unknown 1245935979 0 944643765) 0x7fc85c0156c0 con 0x7fc86c09e1d0 2024-09-16T04:13:06.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.787+0000 7fc863fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.787+0000 7fc863fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:06.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.898+0000 7fc87115f640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.898+0000 7fc87115f640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.898+0000 7fc87115f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:06.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.898+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"} v 0) -- 0x7fc86c1c3820 con 0x7fc86c09e1d0 2024-09-16T04:13:06.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.899+0000 7fc863fff640 1 -- 172.21.15.171:0/925242994 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}]=0 v251) ==== 106+0+3 (unknown 1245935979 0 944643765) 0x7fc85c01e270 con 0x7fc86c09e1d0 2024-09-16T04:13:06.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.899+0000 7fc863fff640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:06.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.899+0000 7fc863fff640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:13:06.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 >> v1:172.21.15.171:6812/7327273 conn(0x7fc84c038f40 legacy=0x7fc84c03b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 10 monclient: shutdown 2024-09-16T04:13:06.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:06.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 >> v1:172.21.15.171:6789/0 conn(0x7fc86c09e1d0 legacy=0x7fc86c1c0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:06.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 shutdown_connections 2024-09-16T04:13:06.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 >> 172.21.15.171:0/925242994 conn(0x7fc86c0a52a0 msgr2=0x7fc86c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:06.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.901+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 shutdown_connections 2024-09-16T04:13:06.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.902+0000 7fc87115f640 1 -- 172.21.15.171:0/925242994 wait complete. 2024-09-16T04:13:06.912 INFO:tasks.workunit.client.0.smithi171.stderr:+ test 12 -eq 12 2024-09-16T04:13:06.912 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 81bbd316-374e-4988-a68f-0955f1e1326d 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.988+0000 7f602281e640 10 monclient: get_monmap_and_config 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.988+0000 7f602281e640 10 monclient: build_initial_monmap 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.988+0000 7f602281e640 10 monclient: monmap: 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:06.989391+0000 2024-09-16T04:13:06.990 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:06.989391+0000 2024-09-16T04:13:06.991 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:06.991 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:06.991 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:06.991 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:06.991 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:06.991 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:06.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 1 Processor -- start 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 1 -- start start 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient: init 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient(hunting): picked mon.noname-c con 0x7f601c09ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient(hunting): picked mon.noname-a con 0x7f601c0a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient(hunting): picked mon.noname-b con 0x7f601c09e570 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c09fbf0 con 0x7f601c0a0750 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c0a0e10 con 0x7f601c09ba30 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c09f640 con 0x7f601c09e570 2024-09-16T04:13:06.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.989+0000 7f602281e640 10 monclient(hunting): authenticate will time out at 1162.494888s 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f601c09ba30 0x7f601c0a0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54416/0 (socket says 172.21.15.171:54416) 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f6020dbb640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f601c09e570 legacy=0x7f601c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601bfff640 1 -- 172.21.15.171:0/917304065 learned_addr learned my addr 172.21.15.171:0/917304065 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368435198 0 0) 0x7f601c0a0e10 con 0x7f601c09ba30 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient(hunting): my global_id is 6445 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6004003520 con 0x7f601c09ba30 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1909694626 0 0) 0x7f6004003520 con 0x7f601c09ba30 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 >> v1:172.21.15.148:6790/0 conn(0x7f601c09e570 legacy=0x7f601c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:07.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 >> v1:172.21.15.148:6789/0 conn(0x7f601c0a0750 legacy=0x7f601c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 monclient: found mon.noname-c 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c09f810 con 0x7f601c09ba30 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: _finish_auth 0 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f600c001610 con 0x7f601c09ba30 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f602281e640 5 monclient: authenticate success, global_id 6445 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f602281e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: dump: 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f602281e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:07.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f600c001970 con 0x7f601c09ba30 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f602281e640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 1 -- 172.21.15.171:0/917304065 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f600c0048a0 con 0x7f601c09ba30 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f602281e640 4 set_mon_vals no callback set 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.990+0000 7f601affd640 10 monclient: dump: 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.003 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 10 monclient: shutdown 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 1 -- 172.21.15.171:0/917304065 >> v1:172.21.15.171:6789/0 conn(0x7f601c09ba30 legacy=0x7f601c0a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 1 -- 172.21.15.171:0/917304065 shutdown_connections 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 1 -- 172.21.15.171:0/917304065 >> 172.21.15.171:0/917304065 conn(0x7f601c0a52a0 msgr2=0x7f601c0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 1 -- 172.21.15.171:0/917304065 shutdown_connections 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 1 -- 172.21.15.171:0/917304065 wait complete. 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 10 monclient: build_initial_monmap 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 10 monclient: monmap: 2024-09-16T04:13:07.004 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.991+0000 7f602281e640 1 Processor -- start 2024-09-16T04:13:07.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 1 -- start start 2024-09-16T04:13:07.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient: init 2024-09-16T04:13:07.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:07.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:07.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient(hunting): picked mon.noname-c con 0x7f601c09ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:07.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient(hunting): picked mon.noname-a con 0x7f601c09e570 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient(hunting): picked mon.noname-b con 0x7f601c0a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c1ca8b0 con 0x7f601c09e570 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c132610 con 0x7f601c0a0750 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f601c06ceb0 con 0x7f601c09ba30 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f602281e640 10 monclient(hunting): authenticate will time out at 1162.497622s 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6020dbb640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f601c0a0750 0x7f601c1d1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:54420/0 (socket says 172.21.15.171:54420) 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6020dbb640 1 -- 172.21.15.171:0/2161794896 learned_addr learned my addr 172.21.15.171:0/2161794896 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f601bfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f601c09ba30 legacy=0x7f601c1c9a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:07.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145707099 0 0) 0x7f601c132610 con 0x7f601c0a0750 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6018ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6018ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6018ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6018ff9640 10 monclient(hunting): my global_id is 6448 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.992+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6008003520 con 0x7f601c0a0750 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3461136961 0 0) 0x7f6008003520 con 0x7f601c0a0750 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f601c132610 con 0x7f601c0a0750 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60140043f0 con 0x7f601c0a0750 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): dump: 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.009 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2947676775 0 0) 0x7f601c1ca8b0 con 0x7f601c09e570 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient(hunting): my global_id is 64884 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6008003520 con 0x7f601c09e570 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3275768417 0 0) 0x7f601c132610 con 0x7f601c0a0750 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 >> v1:172.21.15.148:6790/0 conn(0x7f601c09ba30 legacy=0x7f601c1c9a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 >> v1:172.21.15.148:6789/0 conn(0x7f601c09e570 legacy=0x7f601c1ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 monclient: found mon.b 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c130880 con 0x7f601c0a0750 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: _finish_auth 0 2024-09-16T04:13:07.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 5 monclient: authenticate success, global_id 6448 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 10 monclient: _renew_subs 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f601c130b60 con 0x7f601c0a0750 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 10 monclient: _renew_subs 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f601c12dfc0 con 0x7f601c0a0750 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60140046e0 con 0x7f601c0a0750 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6014004ab0 con 0x7f601c0a0750 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f601a7fc640 4 set_mon_vals no callback set 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.993+0000 7f6018ff9640 10 monclient: dump: 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6014004c50 con 0x7f601c0a0750 2024-09-16T04:13:07.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f602281e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f602281e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f602281e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc005080 con 0x7f601c0a0750 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(251..251 src has 1..251) ==== 6817+0+0 (unknown 319184596 0 0) 0x7f6014047d20 con 0x7f601c0a0750 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.994+0000 7f601bfff640 10 monclient: get_auth_request con 0x7f6008038fc0 auth_method 0 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.999+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6014078d40 con 0x7f601c0a0750 2024-09-16T04:13:07.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.999+0000 7f6018ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:07.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:06.999+0000 7f6018ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:07.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.174+0000 7f602281e640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.174+0000 7f602281e640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.174+0000 7f602281e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.174+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"} v 0) -- 0x7f5ffc005370 con 0x7f601c0a0750 2024-09-16T04:13:07.398 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.397+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}]=0 v252) ==== 106+0+3 (unknown 532937314 0 1367428999) 0x7f6014015f20 con 0x7f601c0a0750 2024-09-16T04:13:07.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.397+0000 7f6018ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.399 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.397+0000 7f6018ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:07.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.507+0000 7f602281e640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.507+0000 7f602281e640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.507+0000 7f602281e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.509 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.507+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"} v 0) -- 0x7f5ffc005bd0 con 0x7f601c0a0750 2024-09-16T04:13:07.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.508+0000 7f6018ff9640 1 -- 172.21.15.171:0/2161794896 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}]=0 v252) ==== 106+0+3 (unknown 532937314 0 1367428999) 0x7f6014019bd0 con 0x7f601c0a0750 2024-09-16T04:13:07.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.508+0000 7f6018ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.510 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.508+0000 7f6018ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:13:07.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 >> v1:172.21.15.171:6812/7327273 conn(0x7f6008038fc0 legacy=0x7f600803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 10 monclient: shutdown 2024-09-16T04:13:07.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:07.512 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 >> v1:172.21.15.171:6789/0 conn(0x7f601c0a0750 legacy=0x7f601c1d1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 shutdown_connections 2024-09-16T04:13:07.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 >> 172.21.15.171:0/2161794896 conn(0x7f601c0a52a0 msgr2=0x7f601c131a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:07.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 shutdown_connections 2024-09-16T04:13:07.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.510+0000 7f602281e640 1 -- 172.21.15.171:0/2161794896 wait complete. 2024-09-16T04:13:07.521 INFO:tasks.workunit.client.0.smithi171.stderr:+ nb=14 2024-09-16T04:13:07.521 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 81bbd316-374e-4988-a68f-0955f1e1326d 2024-09-16T04:13:07.598 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.597+0000 7f48e2465640 10 monclient: get_monmap_and_config 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.597+0000 7f48e2465640 10 monclient: build_initial_monmap 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.597+0000 7f48e2465640 10 monclient: monmap: 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:07.597964+0000 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:07.597964+0000 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:07.599 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.597+0000 7f48e2465640 1 Processor -- start 2024-09-16T04:13:07.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.597+0000 7f48e2465640 1 -- start start 2024-09-16T04:13:07.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient: init 2024-09-16T04:13:07.608 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient(hunting): picked mon.noname-c con 0x7f48dc136b60 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient(hunting): picked mon.noname-b con 0x7f48dc139b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient(hunting): picked mon.noname-a con 0x7f48dc13d620 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc0a1340 con 0x7f48dc13d620 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc137650 con 0x7f48dc136b60 2024-09-16T04:13:07.609 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc0a0aa0 con 0x7f48dc139b10 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48e2465640 10 monclient(hunting): authenticate will time out at 1163.103512s 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48dbfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f48dc136b60 0x7f48dc136f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43866/0 (socket says 172.21.15.171:43866) 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48db7fe640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f48dc139b10 legacy=0x7f48dc13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48dbfff640 1 -- 172.21.15.171:0/67961616 learned_addr learned my addr 172.21.15.171:0/67961616 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.598+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309488077 0 0) 0x7f48dc137650 con 0x7f48dc136b60 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): my global_id is 6451 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48c4003520 con 0x7f48dc136b60 2024-09-16T04:13:07.610 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516609015 0 0) 0x7f48dc0a1340 con 0x7f48dc13d620 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient(hunting): my global_id is 64887 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48dc137650 con 0x7f48dc13d620 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 550694627 0 0) 0x7f48c4003520 con 0x7f48dc136b60 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 >> v1:172.21.15.148:6790/0 conn(0x7f48dc139b10 legacy=0x7f48dc13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 >> v1:172.21.15.148:6789/0 conn(0x7f48dc13d620 legacy=0x7f48dc13fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:07.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 monclient: found mon.noname-c 2024-09-16T04:13:07.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:07.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc0a0550 con 0x7f48dc136b60 2024-09-16T04:13:07.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48cc001610 con 0x7f48dc136b60 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 5 monclient: authenticate success, global_id 6451 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: dump: 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:07.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48cc001930 con 0x7f48dc136b60 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 1 -- 172.21.15.171:0/67961616 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48cc001d00 con 0x7f48dc136b60 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 4 set_mon_vals no callback set 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48daffd640 10 monclient: dump: 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.615 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 10 monclient: shutdown 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 1 -- 172.21.15.171:0/67961616 >> v1:172.21.15.171:6789/0 conn(0x7f48dc136b60 legacy=0x7f48dc136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 1 -- 172.21.15.171:0/67961616 shutdown_connections 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 1 -- 172.21.15.171:0/67961616 >> 172.21.15.171:0/67961616 conn(0x7f48dc0a98b0 msgr2=0x7f48dc0a5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 1 -- 172.21.15.171:0/67961616 shutdown_connections 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.599+0000 7f48e2465640 1 -- 172.21.15.171:0/67961616 wait complete. 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.600+0000 7f48e2465640 10 monclient: build_initial_monmap 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.600+0000 7f48e2465640 10 monclient: monmap: 2024-09-16T04:13:07.616 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.600+0000 7f48e2465640 1 Processor -- start 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 1 -- start start 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient: init 2024-09-16T04:13:07.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient(hunting): picked mon.noname-c con 0x7f48dc136b60 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient(hunting): picked mon.noname-a con 0x7f48dc139b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient(hunting): picked mon.noname-b con 0x7f48dc13d620 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc13c210 con 0x7f48dc139b10 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc09b080 con 0x7f48dc13d620 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48dbfff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f48dc136b60 legacy=0x7f48dc13a160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc0a6f50 con 0x7f48dc136b60 2024-09-16T04:13:07.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e2465640 10 monclient(hunting): authenticate will time out at 1163.106499s 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e0a02640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f48dc13d620 0x7f48dc1d37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43874/0 (socket says 172.21.15.171:43874) 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48e0a02640 1 -- 172.21.15.171:0/4244605619 learned_addr learned my addr 172.21.15.171:0/4244605619 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2682837091 0 0) 0x7f48dc09b080 con 0x7f48dc13d620 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): my global_id is 6454 2024-09-16T04:13:07.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48c8003520 con 0x7f48dc13d620 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 195130969 0 0) 0x7f48c8003520 con 0x7f48dc13d620 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f48dc09b080 con 0x7f48dc13d620 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2607971953 0 0) 0x7f48dc13c210 con 0x7f48dc139b10 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): my global_id is 64890 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48c8003520 con 0x7f48dc139b10 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48d4001c50 con 0x7f48dc13d620 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.601+0000 7f48d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:07.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient(hunting): dump: 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:07.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4021898166 0 0) 0x7f48dc09b080 con 0x7f48dc13d620 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 >> v1:172.21.15.148:6790/0 conn(0x7f48dc136b60 legacy=0x7f48dc13a160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 >> v1:172.21.15.148:6789/0 conn(0x7f48dc139b10 legacy=0x7f48dc13a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 monclient: found mon.b 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc0a2a80 con 0x7f48dc13d620 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:07.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 5 monclient: authenticate success, global_id 6454 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 10 monclient: _renew_subs 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc0a0240 con 0x7f48dc13d620 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 10 monclient: _renew_subs 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48dc09ada0 con 0x7f48dc13d620 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48d4002060 con 0x7f48dc13d620 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f48d4003de0 con 0x7f48dc13d620 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 10 monclient: dump: 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.623 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.624 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.624 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.624 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48da7fc640 4 set_mon_vals no callback set 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.602+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f48d4003f80 con 0x7f48dc13d620 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.603+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(252..252 src has 1..252) ==== 6817+0+0 (unknown 301288270 0 0) 0x7f48d4047d90 con 0x7f48dc13d620 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.603+0000 7f48e2465640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.603+0000 7f48e2465640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.603+0000 7f48e2465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.603+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48bc005080 con 0x7f48dc13d620 2024-09-16T04:13:07.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.603+0000 7f48dbfff640 10 monclient: get_auth_request con 0x7f48c8038fc0 auth_method 0 2024-09-16T04:13:07.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.607+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f48d4015ff0 con 0x7f48dc13d620 2024-09-16T04:13:07.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.607+0000 7f48d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:07.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.607+0000 7f48d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:07.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.786+0000 7f48e2465640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.786+0000 7f48e2465640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.786+0000 7f48e2465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.786+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"} v 0) -- 0x7f48bc005370 con 0x7f48dc13d620 2024-09-16T04:13:07.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.787+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}]=0 v252) ==== 106+0+3 (unknown 532937314 0 1367428999) 0x7f48d40158f0 con 0x7f48dc13d620 2024-09-16T04:13:07.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.787+0000 7f48d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.787+0000 7f48d8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:07.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48e2465640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48e2465640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48e2465640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"} v 0) -- 0x7f48bc005b50 con 0x7f48dc13d620 2024-09-16T04:13:07.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48d8ff9640 1 -- 172.21.15.171:0/4244605619 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}]=0 v252) ==== 106+0+3 (unknown 532937314 0 1367428999) 0x7f48d401e4a0 con 0x7f48dc13d620 2024-09-16T04:13:07.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48d8ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "81bbd316-374e-4988-a68f-0955f1e1326d"}] 2024-09-16T04:13:07.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.896+0000 7f48d8ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 >> v1:172.21.15.171:6812/7327273 conn(0x7f48c8038fc0 legacy=0x7f48c803b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 10 monclient: shutdown 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 >> v1:172.21.15.171:6789/0 conn(0x7f48dc13d620 legacy=0x7f48dc1d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 shutdown_connections 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 >> 172.21.15.171:0/4244605619 conn(0x7f48dc0a98b0 msgr2=0x7f48dc0a3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 shutdown_connections 2024-09-16T04:13:07.901 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.898+0000 7f48e2465640 1 -- 172.21.15.171:0/4244605619 wait complete. 2024-09-16T04:13:07.909 INFO:tasks.workunit.client.0.smithi171.stderr:+ test 14 -eq 14 2024-09-16T04:13:07.909 INFO:tasks.workunit.client.0.smithi171.stderr:+ test 14 -ne 12 2024-09-16T04:13:07.909 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm 12 2024-09-16T04:13:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.984+0000 7fa66c133640 10 monclient: get_monmap_and_config 2024-09-16T04:13:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.984+0000 7fa66c133640 10 monclient: build_initial_monmap 2024-09-16T04:13:07.986 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.984+0000 7fa66c133640 10 monclient: monmap: 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:07.985524+0000 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:07.985524+0000 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 1 Processor -- start 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 1 -- start start 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient: init 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient(hunting): picked mon.noname-a con 0x7fa664130930 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient(hunting): picked mon.noname-b con 0x7fa66409eb60 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:07.994 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient(hunting): picked mon.noname-c con 0x7fa664130d10 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6641343a0 con 0x7fa664130930 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66409c1b0 con 0x7fa664130d10 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66409d8e0 con 0x7fa66409eb60 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.985+0000 7fa66c133640 10 monclient(hunting): authenticate will time out at 1163.490965s 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa6696ce640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa66409eb60 legacy=0x7fa664133c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66a6d0640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa664130d10 0x7fa66413c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43882/0 (socket says 172.21.15.171:43882) 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66a6d0640 1 -- 172.21.15.171:0/2688069389 learned_addr learned my addr 172.21.15.171:0/2688069389 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164707980 0 0) 0x7fa66409c1b0 con 0x7fa664130d10 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.995 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): my global_id is 6457 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa650003520 con 0x7fa664130d10 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 247304507 0 0) 0x7fa6641343a0 con 0x7fa664130930 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient(hunting): my global_id is 64893 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa66409c1b0 con 0x7fa664130930 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 704673855 0 0) 0x7fa650003520 con 0x7fa664130d10 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 >> v1:172.21.15.148:6790/0 conn(0x7fa66409eb60 legacy=0x7fa664133c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 >> v1:172.21.15.148:6789/0 conn(0x7fa664130930 legacy=0x7fa664133580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 monclient: found mon.noname-c 2024-09-16T04:13:07.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66409dbc0 con 0x7fa664130d10 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: _finish_auth 0 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66c133640 5 monclient: authenticate success, global_id 6457 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa65c001990 con 0x7fa664130d10 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66c133640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: dump: 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.997 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66c133640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa65c001c40 con 0x7fa664130d10 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 1 -- 172.21.15.171:0/2688069389 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa65c002010 con 0x7fa664130d10 2024-09-16T04:13:07.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66c133640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa66c133640 4 set_mon_vals no callback set 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.986+0000 7fa668ecd640 10 monclient: dump: 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:07.999 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 10 monclient: shutdown 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 1 -- 172.21.15.171:0/2688069389 >> v1:172.21.15.171:6789/0 conn(0x7fa664130d10 legacy=0x7fa66413c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 1 -- 172.21.15.171:0/2688069389 shutdown_connections 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 1 -- 172.21.15.171:0/2688069389 >> 172.21.15.171:0/2688069389 conn(0x7fa6640a52a0 msgr2=0x7fa6640a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 1 -- 172.21.15.171:0/2688069389 shutdown_connections 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 1 -- 172.21.15.171:0/2688069389 wait complete. 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 10 monclient: build_initial_monmap 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.987+0000 7fa66c133640 10 monclient: monmap: 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:08.000 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 1 Processor -- start 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 1 -- start start 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient: init 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:08.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient(hunting): picked mon.noname-a con 0x7fa66409eb60 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient(hunting): picked mon.noname-b con 0x7fa664130930 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient(hunting): picked mon.noname-c con 0x7fa664130d10 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66411e6a0 con 0x7fa66409eb60 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66411d4d0 con 0x7fa664130930 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66411ce10 con 0x7fa664130d10 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66c133640 10 monclient(hunting): authenticate will time out at 1163.493679s 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa6696ce640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fa664130930 0x7fa6641cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43892/0 (socket says 172.21.15.171:43892) 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa6696ce640 1 -- 172.21.15.171:0/3202188343 learned_addr learned my addr 172.21.15.171:0/3202188343 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa66a6d0640 1 -- 172.21.15.171:0/3202188343 >> v1:172.21.15.148:6790/0 conn(0x7fa664130d10 legacy=0x7fa6641cffe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446264782 0 0) 0x7fa66411d4d0 con 0x7fa664130930 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa65a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa65a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa65a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa65a7fc640 10 monclient(hunting): my global_id is 6460 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.988+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa648003520 con 0x7fa664130930 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200258447 0 0) 0x7fa66411e6a0 con 0x7fa66409eb60 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): my global_id is 64896 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa66411d4d0 con 0x7fa66409eb60 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3110003058 0 0) 0x7fa648003520 con 0x7fa664130930 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa66411e6a0 con 0x7fa664130930 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa6540041d0 con 0x7fa664130930 2024-09-16T04:13:08.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): dump: 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.004 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1773910929 0 0) 0x7fa66411d4d0 con 0x7fa66409eb60 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa648003520 con 0x7fa66409eb60 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3965572082 0 0) 0x7fa66411e6a0 con 0x7fa664130930 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 >> v1:172.21.15.148:6790/0 conn(0x7fa664130d10 legacy=0x7fa6641cffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 >> v1:172.21.15.148:6789/0 conn(0x7fa66409eb60 legacy=0x7fa66412dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 monclient: found mon.b 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66411e980 con 0x7fa664130930 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:08.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:08.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:08.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 5 monclient: authenticate success, global_id 6460 2024-09-16T04:13:08.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 10 monclient: _renew_subs 2024-09-16T04:13:08.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa664130340 con 0x7fa664130930 2024-09-16T04:13:08.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 10 monclient: _renew_subs 2024-09-16T04:13:08.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa66411c2a0 con 0x7fa664130930 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa654004430 con 0x7fa664130930 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa654004780 con 0x7fa664130930 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65bfff640 4 set_mon_vals no callback set 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.989+0000 7fa65a7fc640 10 monclient: dump: 2024-09-16T04:13:08.009 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa654004920 con 0x7fa664130930 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa66c133640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa66c133640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa66c133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64c005080 con 0x7fa664130930 2024-09-16T04:13:08.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(252..252 src has 1..252) ==== 6817+0+0 (unknown 301288270 0 0) 0x7fa654047990 con 0x7fa664130930 2024-09-16T04:13:08.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.990+0000 7fa669ecf640 10 monclient: get_auth_request con 0x7fa648038f40 auth_method 0 2024-09-16T04:13:08.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.994+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa654015c50 con 0x7fa664130930 2024-09-16T04:13:08.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.994+0000 7fa65a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:08.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:07.994+0000 7fa65a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:08.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.168+0000 7fa66c133640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.168+0000 7fa66c133640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.168+0000 7fa66c133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.168+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7fa64c005370 con 0x7fa664130930 2024-09-16T04:13:08.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:08.169+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e252 definitely_dead 0 2024-09-16T04:13:08.404 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.402+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 removed osd.12 v253) ==== 83+0+0 (unknown 3002182018 0 0) 0x7fa654015550 con 0x7fa664130930 2024-09-16T04:13:08.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.402+0000 7fa65a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.405 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.402+0000 7fa65a7fc640 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-16T04:13:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.511+0000 7fa66c133640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.511+0000 7fa66c133640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.511+0000 7fa66c133640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.511+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7fa64c000be0 con 0x7fa664130930 2024-09-16T04:13:08.514 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:08.512+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e253 definitely_dead 0 2024-09-16T04:13:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.512+0000 7fa65a7fc640 1 -- 172.21.15.171:0/3202188343 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v253) ==== 92+0+0 (unknown 2474614984 0 0) 0x7fa654043290 con 0x7fa664130930 2024-09-16T04:13:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.512+0000 7fa65a7fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.512+0000 7fa65a7fc640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T04:13:08.514 INFO:tasks.workunit.client.0.smithi171.stderr:osd.12 does not exist. 2024-09-16T04:13:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 >> v1:172.21.15.171:6812/7327273 conn(0x7fa648038f40 legacy=0x7fa64803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 10 monclient: shutdown 2024-09-16T04:13:08.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 >> v1:172.21.15.171:6789/0 conn(0x7fa664130930 legacy=0x7fa6641cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 shutdown_connections 2024-09-16T04:13:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 >> 172.21.15.171:0/3202188343 conn(0x7fa6640a52a0 msgr2=0x7fa664133480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 shutdown_connections 2024-09-16T04:13:08.517 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.514+0000 7fa66c133640 1 -- 172.21.15.171:0/3202188343 wait complete. 2024-09-16T04:13:08.525 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm 12 2024-09-16T04:13:08.603 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.602+0000 7fef7aca6640 10 monclient: get_monmap_and_config 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.602+0000 7fef7aca6640 10 monclient: build_initial_monmap 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.602+0000 7fef7aca6640 10 monclient: monmap: 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:08.602884+0000 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:08.602884+0000 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:08.604 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:08.605 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.611 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.602+0000 7fef7aca6640 1 Processor -- start 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.602+0000 7fef7aca6640 1 -- start start 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.602+0000 7fef7aca6640 10 monclient: init 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient(hunting): picked mon.noname-b con 0x7fef74136cf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient(hunting): picked mon.noname-a con 0x7fef74139ad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient(hunting): picked mon.noname-c con 0x7fef7413d5e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef740a4f90 con 0x7fef74139ad0 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef740a0a30 con 0x7fef7413d5e0 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef740a0d10 con 0x7fef74136cf0 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:08.612 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 10 monclient(hunting): authenticate will time out at 1164.108333s 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef78a42640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fef74136cf0 legacy=0x7fef741370d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef79243640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fef7413d5e0 0x7fef7413f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43898/0 (socket says 172.21.15.171:43898) 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef79243640 1 -- 172.21.15.171:0/1781450486 learned_addr learned my addr 172.21.15.171:0/1781450486 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543382255 0 0) 0x7fef740a0a30 con 0x7fef7413d5e0 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient(hunting): my global_id is 6463 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef64003520 con 0x7fef7413d5e0 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2414896823 0 0) 0x7fef64003520 con 0x7fef7413d5e0 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 >> v1:172.21.15.148:6790/0 conn(0x7fef74136cf0 legacy=0x7fef741370d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:08.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 >> v1:172.21.15.148:6789/0 conn(0x7fef74139ad0 legacy=0x7fef7413bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 monclient: found mon.noname-c 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef740a0ff0 con 0x7fef7413d5e0 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef6c001990 con 0x7fef7413d5e0 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:08.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef737fe640 10 monclient: dump: 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 5 monclient: authenticate success, global_id 6463 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.603+0000 7fef7aca6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:08.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef6c001c80 con 0x7fef7413d5e0 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef737fe640 1 -- 172.21.15.171:0/1781450486 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef6c002050 con 0x7fef7413d5e0 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 4 set_mon_vals no callback set 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef737fe640 10 monclient: dump: 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.616 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 10 monclient: shutdown 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 1 -- 172.21.15.171:0/1781450486 >> v1:172.21.15.171:6789/0 conn(0x7fef7413d5e0 legacy=0x7fef7413f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 1 -- 172.21.15.171:0/1781450486 shutdown_connections 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 1 -- 172.21.15.171:0/1781450486 >> 172.21.15.171:0/1781450486 conn(0x7fef740a9830 msgr2=0x7fef740a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 1 -- 172.21.15.171:0/1781450486 shutdown_connections 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.604+0000 7fef7aca6640 1 -- 172.21.15.171:0/1781450486 wait complete. 2024-09-16T04:13:08.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient: build_initial_monmap 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient: monmap: 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 1 Processor -- start 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 1 -- start start 2024-09-16T04:13:08.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient: init 2024-09-16T04:13:08.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:08.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:13:08.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient(hunting): picked mon.noname-b con 0x7fef74136cf0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:08.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.605+0000 7fef7aca6640 10 monclient(hunting): picked mon.noname-a con 0x7fef74139ad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:08.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient(hunting): picked mon.noname-c con 0x7fef7413d5e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7413b8f0 con 0x7fef74139ad0 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7413c450 con 0x7fef74136cf0 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef7413c1d0 con 0x7fef7413d5e0 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient(hunting): authenticate will time out at 1164.111030s 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef78a42640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fef74136cf0 0x7fef741cb9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43902/0 (socket says 172.21.15.171:43902) 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef78a42640 1 -- 172.21.15.171:0/245776714 learned_addr learned my addr 172.21.15.171:0/245776714 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef79243640 1 -- 172.21.15.171:0/245776714 >> v1:172.21.15.148:6790/0 conn(0x7fef7413d5e0 legacy=0x7fef741d4140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626462744 0 0) 0x7fef7413c450 con 0x7fef74136cf0 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): my global_id is 6466 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef58003520 con 0x7fef74136cf0 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3491317622 0 0) 0x7fef7413b8f0 con 0x7fef74139ad0 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): my global_id is 64902 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef7413c450 con 0x7fef74139ad0 2024-09-16T04:13:08.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1407058971 0 0) 0x7fef58003520 con 0x7fef74136cf0 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef7413b8f0 con 0x7fef74136cf0 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef680041d0 con 0x7fef74136cf0 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): dump: 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.622 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 246746430 0 0) 0x7fef7413c450 con 0x7fef74139ad0 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef58003520 con 0x7fef74139ad0 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef60001830 con 0x7fef74139ad0 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient(hunting): dump: 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.623 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3147542836 0 0) 0x7fef7413b8f0 con 0x7fef74136cf0 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 >> v1:172.21.15.148:6790/0 conn(0x7fef7413d5e0 legacy=0x7fef741d4140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 >> v1:172.21.15.148:6789/0 conn(0x7fef74139ad0 legacy=0x7fef741d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: _finish_hunting 0 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 monclient: found mon.b 2024-09-16T04:13:08.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7409ac70 con 0x7fef74136cf0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: _finish_auth 0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: _check_auth_tickets 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 5 monclient: authenticate success, global_id 6466 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient: _renew_subs 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef7409af50 con 0x7fef74136cf0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient: _renew_subs 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef7413aec0 con 0x7fef74136cf0 2024-09-16T04:13:08.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef68004870 con 0x7fef74136cf0 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fef68004bc0 con 0x7fef74136cf0 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef717fa640 10 monclient: dump: 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:08.626 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.606+0000 7fef72ffd640 4 set_mon_vals no callback set 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fef6800fff0 con 0x7fef74136cf0 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(253..253 src has 1..253) ==== 6817+0+0 (unknown 3382097286 0 0) 0x7fef68047970 con 0x7fef74136cf0 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef7aca6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef7aca6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef7aca6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef54005080 con 0x7fef74136cf0 2024-09-16T04:13:08.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.607+0000 7fef73fff640 10 monclient: get_auth_request con 0x7fef58038ec0 auth_method 0 2024-09-16T04:13:08.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.612+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fef68015c50 con 0x7fef74136cf0 2024-09-16T04:13:08.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.612+0000 7fef717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:08.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.612+0000 7fef717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:08.787 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.786+0000 7fef7aca6640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.786+0000 7fef7aca6640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.786+0000 7fef7aca6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.788 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.786+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7fef54005370 con 0x7fef74136cf0 2024-09-16T04:13:08.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:08.787+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e253 definitely_dead 0 2024-09-16T04:13:08.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.787+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v253) ==== 92+0+0 (unknown 2474614984 0 0) 0x7fef68019900 con 0x7fef74136cf0 2024-09-16T04:13:08.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.787+0000 7fef717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.789 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.787+0000 7fef717fa640 10 monclient: _finish_command 2 = system:0 osd.12 does not exist. 2024-09-16T04:13:08.896 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.895+0000 7fef7aca6640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.895+0000 7fef7aca6640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.895+0000 7fef7aca6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:08.897 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.895+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["12"]} v 0) -- 0x7fef54000be0 con 0x7fef74136cf0 2024-09-16T04:13:08.897 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:08.895+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e253 definitely_dead 0 2024-09-16T04:13:08.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.896+0000 7fef717fa640 1 -- 172.21.15.171:0/245776714 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["12"]}]=0 osd.12 does not exist. v253) ==== 92+0+0 (unknown 2474614984 0 0) 0x7fef6801e1c0 con 0x7fef74136cf0 2024-09-16T04:13:08.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.896+0000 7fef717fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["12"]}] 2024-09-16T04:13:08.898 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.896+0000 7fef717fa640 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-16T04:13:08.898 INFO:tasks.workunit.client.0.smithi171.stderr:osd.12 does not exist. 2024-09-16T04:13:08.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.897+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 >> v1:172.21.15.171:6812/7327273 conn(0x7fef58038ec0 legacy=0x7fef5803b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.899 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.897+0000 7fef7aca6640 10 monclient: shutdown 2024-09-16T04:13:08.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.897+0000 7fef7aca6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:08.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.897+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 >> v1:172.21.15.171:6789/0 conn(0x7fef74136cf0 legacy=0x7fef741cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.897+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 shutdown_connections 2024-09-16T04:13:08.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.897+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 >> 172.21.15.171:0/245776714 conn(0x7fef740a9830 msgr2=0x7fef740a7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:08.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.898+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 shutdown_connections 2024-09-16T04:13:08.900 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.898+0000 7fef7aca6640 1 -- 172.21.15.171:0/245776714 wait complete. 2024-09-16T04:13:08.908 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm 14 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.985+0000 7f9d6e8f8640 10 monclient: get_monmap_and_config 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.985+0000 7f9d6e8f8640 10 monclient: build_initial_monmap 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.985+0000 7f9d6e8f8640 10 monclient: monmap: 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:08.986045+0000 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:08.986045+0000 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:08.987 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:08.988 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:08.988 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:08.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 1 Processor -- start 2024-09-16T04:13:08.996 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 1 -- start start 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient: init 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f9d68136cf0 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f9d68139ad0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6813d5e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d680a4f90 con 0x7f9d68136cf0 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d680a0a30 con 0x7f9d68139ad0 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d680a0d10 con 0x7f9d6813d5e0 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6e8f8640 10 monclient(hunting): authenticate will time out at 1164.491591s 2024-09-16T04:13:08.997 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d6ce95640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f9d6813d5e0 legacy=0x7f9d6813f9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d677fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9d68139ad0 0x7f9d6813bec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43912/0 (socket says 172.21.15.171:43912) 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d677fe640 1 -- 172.21.15.171:0/3991390966 learned_addr learned my addr 172.21.15.171:0/3991390966 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485269712 0 0) 0x7f9d680a0a30 con 0x7f9d68139ad0 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d66ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.998 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d66ffd640 10 monclient(hunting): my global_id is 6469 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.986+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d50003520 con 0x7f9d68139ad0 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039655166 0 0) 0x7f9d680a4f90 con 0x7f9d68136cf0 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient(hunting): my global_id is 64905 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d680a0a30 con 0x7f9d68136cf0 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3600978624 0 0) 0x7f9d50003520 con 0x7f9d68139ad0 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 >> v1:172.21.15.148:6790/0 conn(0x7f9d6813d5e0 legacy=0x7f9d6813f9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 >> v1:172.21.15.148:6789/0 conn(0x7f9d68136cf0 legacy=0x7f9d681370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:08.999 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 monclient: found mon.noname-c 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d680a0ff0 con 0x7f9d68139ad0 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9d5c001610 con 0x7f9d68139ad0 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 5 monclient: authenticate success, global_id 6469 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: dump: 2024-09-16T04:13:09.000 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:09.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d5c0018c0 con 0x7f9d68139ad0 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 1 -- 172.21.15.171:0/3991390966 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9d5c001c90 con 0x7f9d68139ad0 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 4 set_mon_vals no callback set 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d66ffd640 10 monclient: dump: 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.002 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 10 monclient: shutdown 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3991390966 >> v1:172.21.15.171:6789/0 conn(0x7f9d68139ad0 legacy=0x7f9d6813bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3991390966 shutdown_connections 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3991390966 >> 172.21.15.171:0/3991390966 conn(0x7f9d680a9830 msgr2=0x7f9d680a5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3991390966 shutdown_connections 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.987+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3991390966 wait complete. 2024-09-16T04:13:09.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.988+0000 7f9d6e8f8640 10 monclient: build_initial_monmap 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.988+0000 7f9d6e8f8640 10 monclient: monmap: 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.988+0000 7f9d6e8f8640 1 Processor -- start 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.988+0000 7f9d6e8f8640 1 -- start start 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.988+0000 7f9d6e8f8640 10 monclient: init 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient(hunting): picked mon.noname-c con 0x7f9d68136cf0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient(hunting): picked mon.noname-a con 0x7f9d68139ad0 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6813d5e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d68120920 con 0x7f9d68139ad0 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d68120c00 con 0x7f9d6813d5e0 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d680a6eb0 con 0x7f9d68136cf0 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:09.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6e8f8640 10 monclient(hunting): authenticate will time out at 1164.494453s 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d67fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f9d68136cf0 legacy=0x7f9d681cb9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6ce95640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f9d6813d5e0 0x7f9d681d40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43916/0 (socket says 172.21.15.171:43916) 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d6ce95640 1 -- 172.21.15.171:0/3921159839 learned_addr learned my addr 172.21.15.171:0/3921159839 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101621465 0 0) 0x7f9d68120c00 con 0x7f9d6813d5e0 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): my global_id is 6472 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d54003520 con 0x7f9d6813d5e0 2024-09-16T04:13:09.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235521096 0 0) 0x7f9d68120920 con 0x7f9d68139ad0 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): my global_id is 64908 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d68120c00 con 0x7f9d68139ad0 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2762477542 0 0) 0x7f9d54003520 con 0x7f9d6813d5e0 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d68120920 con 0x7f9d6813d5e0 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9d60004440 con 0x7f9d6813d5e0 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): dump: 2024-09-16T04:13:09.007 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.989+0000 7f9d64ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:09.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 624501902 0 0) 0x7f9d68120c00 con 0x7f9d68139ad0 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d54003520 con 0x7f9d68139ad0 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9d5c001c10 con 0x7f9d68139ad0 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient(hunting): dump: 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.009 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4092169651 0 0) 0x7f9d68120920 con 0x7f9d6813d5e0 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 >> v1:172.21.15.148:6790/0 conn(0x7f9d68136cf0 legacy=0x7f9d681cb9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 >> v1:172.21.15.148:6789/0 conn(0x7f9d68139ad0 legacy=0x7f9d681d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 monclient: found mon.b 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d680a5d80 con 0x7f9d6813d5e0 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: _finish_auth 0 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:13:09.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 5 monclient: authenticate success, global_id 6472 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 10 monclient: _renew_subs 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6809f480 con 0x7f9d6813d5e0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 10 monclient: _renew_subs 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d6809e450 con 0x7f9d6813d5e0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d60004760 con 0x7f9d6813d5e0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9d60004ab0 con 0x7f9d6813d5e0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:09.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d667fc640 4 set_mon_vals no callback set 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.990+0000 7f9d64ff9640 10 monclient: dump: 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9d60004c50 con 0x7f9d6813d5e0 2024-09-16T04:13:09.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d6e8f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:09.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d6e8f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:09.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d6e8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d48005080 con 0x7f9d6813d5e0 2024-09-16T04:13:09.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(253..253 src has 1..253) ==== 6817+0+0 (unknown 3382097286 0 0) 0x7f9d60047ce0 con 0x7f9d6813d5e0 2024-09-16T04:13:09.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.991+0000 7f9d67fff640 10 monclient: get_auth_request con 0x7f9d54038ef0 auth_method 0 2024-09-16T04:13:09.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.995+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9d60015f40 con 0x7f9d6813d5e0 2024-09-16T04:13:09.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.995+0000 7f9d64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:09.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:08.995+0000 7f9d64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:09.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.170+0000 7f9d6e8f8640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T04:13:09.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.170+0000 7f9d6e8f8640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T04:13:09.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.170+0000 7f9d6e8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.170+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) -- 0x7f9d48005370 con 0x7f9d6813d5e0 2024-09-16T04:13:09.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:09.170+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e253 definitely_dead 0 2024-09-16T04:13:09.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.407+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 removed osd.14 v254) ==== 83+0+0 (unknown 1609553255 0 0) 0x7f9d60015840 con 0x7f9d6813d5e0 2024-09-16T04:13:09.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.407+0000 7f9d64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T04:13:09.409 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.407+0000 7f9d64ff9640 10 monclient: _finish_command 2 = system:0 removed osd.14 2024-09-16T04:13:09.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.513+0000 7f9d6e8f8640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T04:13:09.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.513+0000 7f9d6e8f8640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T04:13:09.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.513+0000 7f9d6e8f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.513+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["14"]} v 0) -- 0x7f9d48000bb0 con 0x7f9d6813d5e0 2024-09-16T04:13:09.516 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:09.514+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e254 definitely_dead 0 2024-09-16T04:13:09.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.514+0000 7f9d64ff9640 1 -- 172.21.15.171:0/3921159839 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["14"]}]=0 osd.14 does not exist. v254) ==== 92+0+0 (unknown 3147415118 0 0) 0x7f9d6001e3f0 con 0x7f9d6813d5e0 2024-09-16T04:13:09.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.514+0000 7f9d64ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["14"]}] 2024-09-16T04:13:09.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.514+0000 7f9d64ff9640 10 monclient: _finish_command 3 = system:0 osd.14 does not exist. 2024-09-16T04:13:09.516 INFO:tasks.workunit.client.0.smithi171.stderr:osd.14 does not exist. 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 >> v1:172.21.15.171:6812/7327273 conn(0x7f9d54038ef0 legacy=0x7f9d5403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 10 monclient: shutdown 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 >> v1:172.21.15.171:6789/0 conn(0x7f9d6813d5e0 legacy=0x7f9d681d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 shutdown_connections 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 >> 172.21.15.171:0/3921159839 conn(0x7f9d680a9830 msgr2=0x7f9d681392d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 shutdown_connections 2024-09-16T04:13:09.518 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.516+0000 7f9d6e8f8640 1 -- 172.21.15.171:0/3921159839 wait complete. 2024-09-16T04:13:09.527 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph osd rm 1000 2024-09-16T04:13:09.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.603+0000 7f0ecf76f640 10 monclient: get_monmap_and_config 2024-09-16T04:13:09.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.603+0000 7f0ecf76f640 10 monclient: build_initial_monmap 2024-09-16T04:13:09.605 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.603+0000 7f0ecf76f640 10 monclient: monmap: 2024-09-16T04:13:09.605 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:09.605 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:09.604488+0000 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:09.604488+0000 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:09.606 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.613 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 1 Processor -- start 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 1 -- start start 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 10 monclient: init 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec809ba30 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec8073570 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec809dcb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.604+0000 7f0ecf76f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec80a0460 con 0x7f0ec809ba30 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec80a0db0 con 0x7f0ec8073570 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec80a11b0 con 0x7f0ec809dcb0 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:09.614 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 10 monclient(hunting): authenticate will time out at 1165.110044s 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0eccd0a640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0ec8073570 0x7f0ec809fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43918/0 (socket says 172.21.15.171:43918) 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecdd0c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0ec809dcb0 legacy=0x7f0ec813b460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0eccd0a640 1 -- 172.21.15.171:0/3948889983 learned_addr learned my addr 172.21.15.171:0/3948889983 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068639189 0 0) 0x7f0ec80a0db0 con 0x7f0ec8073570 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): _negotiate 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): my global_id is 6475 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb4003520 con 0x7f0ec8073570 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519547681 0 0) 0x7f0ec80a0460 con 0x7f0ec809ba30 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): _negotiate 2024-09-16T04:13:09.615 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient(hunting): my global_id is 64911 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec80a0db0 con 0x7f0ec809ba30 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1614654763 0 0) 0x7f0eb4003520 con 0x7f0ec8073570 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 >> v1:172.21.15.148:6790/0 conn(0x7f0ec809dcb0 legacy=0x7f0ec813b460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 >> v1:172.21.15.148:6789/0 conn(0x7f0ec809ba30 legacy=0x7f0ec809f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 monclient: found mon.noname-c 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec809cff0 con 0x7f0ec8073570 2024-09-16T04:13:09.616 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: _finish_auth 0 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0eb8001610 con 0x7f0ec8073570 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 5 monclient: authenticate success, global_id 6475 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: dump: 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.617 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0eb8001930 con 0x7f0ec8073570 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 1 -- 172.21.15.171:0/3948889983 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0eb8001d00 con 0x7f0ec8073570 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ecf76f640 4 set_mon_vals no callback set 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:09.618 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.605+0000 7f0ebffff640 10 monclient: dump: 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 10 monclient: shutdown 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:09.619 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3948889983 >> v1:172.21.15.171:6789/0 conn(0x7f0ec8073570 legacy=0x7f0ec809fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3948889983 shutdown_connections 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3948889983 >> 172.21.15.171:0/3948889983 conn(0x7f0ec80a52a0 msgr2=0x7f0ec80a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3948889983 shutdown_connections 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3948889983 wait complete. 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 10 monclient: build_initial_monmap 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.606+0000 7f0ecf76f640 10 monclient: monmap: 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:09.620 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 1 Processor -- start 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 1 -- start start 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient: init 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec8073570 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec809ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec809dcb0 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:09.621 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec81a7650 con 0x7f0ec8073570 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec81cc750 con 0x7f0ec809dcb0 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec80721e0 con 0x7f0ec809ba30 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecf76f640 10 monclient(hunting): authenticate will time out at 1165.112764s 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0eccd0a640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0ec809ba30 legacy=0x7f0ec81cc040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecdd0c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0ec809dcb0 0x7f0ec81d3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43928/0 (socket says 172.21.15.171:43928) 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ecdd0c640 1 -- 172.21.15.171:0/3448420097 learned_addr learned my addr 172.21.15.171:0/3448420097 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.607+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916667581 0 0) 0x7f0ec81cc750 con 0x7f0ec809dcb0 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): my global_id is 6478 2024-09-16T04:13:09.622 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eac003520 con 0x7f0ec809dcb0 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3631193415 0 0) 0x7f0ec81a7650 con 0x7f0ec8073570 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): my global_id is 64914 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec81cc750 con 0x7f0ec8073570 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3820621626 0 0) 0x7f0eac003520 con 0x7f0ec809dcb0 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ec81a7650 con 0x7f0ec809dcb0 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0ec00043f0 con 0x7f0ec809dcb0 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:09.623 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): dump: 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1254302382 0 0) 0x7f0ec81a7650 con 0x7f0ec809dcb0 2024-09-16T04:13:09.624 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 >> v1:172.21.15.148:6790/0 conn(0x7f0ec809ba30 legacy=0x7f0ec81cc040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 >> v1:172.21.15.148:6789/0 conn(0x7f0ec8073570 legacy=0x7f0ec81cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 monclient: found mon.b 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec80a4260 con 0x7f0ec809dcb0 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: _finish_auth 0 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 5 monclient: authenticate success, global_id 6478 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 10 monclient: _renew_subs 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.625 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec81a8f00 con 0x7f0ec809dcb0 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ec00046a0 con 0x7f0ec809dcb0 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0ec0004a70 con 0x7f0ec809dcb0 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebdffb640 10 monclient: dump: 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.626 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ebf7fe640 4 set_mon_vals no callback set 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 10 monclient: _renew_subs 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.608+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ec81a6540 con 0x7f0ec809dcb0 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0ec0004c10 con 0x7f0ec809dcb0 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(254..254 src has 1..254) ==== 6817+0+0 (unknown 2766061615 0 0) 0x7f0ec0047d00 con 0x7f0ec809dcb0 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ecd50b640 10 monclient: get_auth_request con 0x7f0eac038b90 auth_method 0 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ecf76f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:09.627 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ecf76f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:09.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ecf76f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.609+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec80008d0 con 0x7f0ec809dcb0 2024-09-16T04:13:09.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.614+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0ec0015f00 con 0x7f0ec809dcb0 2024-09-16T04:13:09.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.614+0000 7f0ebdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:09.628 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.614+0000 7f0ebdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:09.792 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.791+0000 7f0ecf76f640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T04:13:09.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.791+0000 7f0ecf76f640 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T04:13:09.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.791+0000 7f0ecf76f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.793 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.791+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) -- 0x7f0ec81a67d0 con 0x7f0ec809dcb0 2024-09-16T04:13:09.793 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:09.791+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e254 definitely_dead 0 2024-09-16T04:13:09.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.792+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v254) ==== 96+0+0 (unknown 3172144995 0 0) 0x7f0ec0015800 con 0x7f0ec809dcb0 2024-09-16T04:13:09.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.792+0000 7f0ebdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T04:13:09.794 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.792+0000 7f0ebdffb640 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-16T04:13:09.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.901+0000 7f0ecf76f640 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T04:13:09.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.901+0000 7f0ecf76f640 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T04:13:09.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.901+0000 7f0ecf76f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:09.903 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.901+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) -- 0x7f0ec81d52b0 con 0x7f0ec809dcb0 2024-09-16T04:13:09.903 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:13:09.902+0000 7f3aabf6b640 -1 mon.a@0(leader).osd e254 definitely_dead 0 2024-09-16T04:13:09.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.902+0000 7f0ebdffb640 1 -- 172.21.15.171:0/3448420097 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v254) ==== 96+0+0 (unknown 3172144995 0 0) 0x7f0ec0019b50 con 0x7f0ec809dcb0 2024-09-16T04:13:09.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.902+0000 7f0ebdffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-16T04:13:09.904 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.902+0000 7f0ebdffb640 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-16T04:13:09.904 INFO:tasks.workunit.client.0.smithi171.stderr:osd.1000 does not exist. 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 >> v1:172.21.15.171:6812/7327273 conn(0x7f0eac038b90 legacy=0x7f0eac03b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 10 monclient: shutdown 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 >> v1:172.21.15.171:6789/0 conn(0x7f0ec809dcb0 legacy=0x7f0ec81d3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 shutdown_connections 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 >> 172.21.15.171:0/3448420097 conn(0x7f0ec80a52a0 msgr2=0x7f0ec81ba530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 shutdown_connections 2024-09-16T04:13:09.906 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.904+0000 7f0ecf76f640 1 -- 172.21.15.171:0/3448420097 wait complete. 2024-09-16T04:13:09.915 INFO:tasks.workunit.client.0.smithi171.stderr:++ ceph osd create 6df951c8-ca70-48bb-b32a-2b7a6ec9adff 2024-09-16T04:13:09.992 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.990+0000 7f6ec51bd640 10 monclient: get_monmap_and_config 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.990+0000 7f6ec51bd640 10 monclient: build_initial_monmap 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.990+0000 7f6ec51bd640 10 monclient: monmap: 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:13:09.991726+0000 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:13:09.991726+0000 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:13:09.993 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.991+0000 7f6ec51bd640 1 Processor -- start 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.991+0000 7f6ec51bd640 1 -- start start 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.991+0000 7f6ec51bd640 10 monclient: init 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient(hunting): picked mon.noname-b con 0x7f6ec009ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ec00a0750 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient(hunting): picked mon.noname-a con 0x7f6ec009e570 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:10.001 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec009fbf0 con 0x7f6ec009e570 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec00a0e10 con 0x7f6ec00a0750 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec009f640 con 0x7f6ec009ba30 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient(hunting): authenticate will time out at 1165.497085s 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebe59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6ec00a0750 0x7f6ec01399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43934/0 (socket says 172.21.15.171:43934) 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebe59c640 1 -- 172.21.15.171:0/3361180609 learned_addr learned my addr 172.21.15.171:0/3361180609 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebed9d640 1 -- 172.21.15.171:0/3361180609 >> v1:172.21.15.148:6790/0 conn(0x7f6ec009ba30 legacy=0x7f6ec00a0040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1517572130 0 0) 0x7f6ec00a0e10 con 0x7f6ec00a0750 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient(hunting): my global_id is 6481 2024-09-16T04:13:10.002 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eac003520 con 0x7f6ec00a0750 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 533856710 0 0) 0x7f6eac003520 con 0x7f6ec00a0750 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 >> v1:172.21.15.148:6790/0 conn(0x7f6ec009ba30 legacy=0x7f6ec00a0040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 >> v1:172.21.15.148:6789/0 conn(0x7f6ec009e570 legacy=0x7f6ec013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 monclient: found mon.noname-c 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec009f810 con 0x7f6ec00a0750 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: _finish_auth 0 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ea8001610 con 0x7f6ec00a0750 2024-09-16T04:13:10.003 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 5 monclient: authenticate success, global_id 6481 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: dump: 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:10.004 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ea8001970 con 0x7f6ec00a0750 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ec51bd640 4 set_mon_vals no callback set 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 1 -- 172.21.15.171:0/3361180609 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ea80048a0 con 0x7f6ec00a0750 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:10.005 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.992+0000 7f6ebdd9b640 10 monclient: dump: 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 10 monclient: shutdown 2024-09-16T04:13:10.006 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 1 -- 172.21.15.171:0/3361180609 >> v1:172.21.15.171:6789/0 conn(0x7f6ec00a0750 legacy=0x7f6ec01399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 1 -- 172.21.15.171:0/3361180609 shutdown_connections 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 1 -- 172.21.15.171:0/3361180609 >> 172.21.15.171:0/3361180609 conn(0x7f6ec00a52a0 msgr2=0x7f6ec00a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 1 -- 172.21.15.171:0/3361180609 shutdown_connections 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.993+0000 7f6ec51bd640 1 -- 172.21.15.171:0/3361180609 wait complete. 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient: build_initial_monmap 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient: monmap: 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:10.007 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 1 Processor -- start 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 1 -- start start 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient: init 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ec009ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient(hunting): picked mon.noname-b con 0x7f6ec009e570 addr v1:172.21.15.171:6789/0 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 10 monclient(hunting): picked mon.noname-a con 0x7f6ec00a0750 addr v1:172.21.15.148:6789/0 2024-09-16T04:13:10.008 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec00a3250 con 0x7f6ec00a0750 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.994+0000 7f6ec51bd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec013a100 con 0x7f6ec009e570 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec006ceb0 con 0x7f6ec009ba30 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 10 monclient(hunting): authenticate will time out at 1165.500016s 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ebed9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f6ec009ba30 legacy=0x7f6ec01392e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ebe59c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f6ec009e570 0x7f6ec01399f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:43950/0 (socket says 172.21.15.171:43950) 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ebe59c640 1 -- 172.21.15.171:0/1655520621 learned_addr learned my addr 172.21.15.171:0/1655520621 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899405090 0 0) 0x7f6ec013a100 con 0x7f6ec009e570 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): my global_id is 6484 2024-09-16T04:13:10.009 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea0003520 con 0x7f6ec009e570 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743440822 0 0) 0x7f6ec00a3250 con 0x7f6ec00a0750 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): my global_id is 64920 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ec013a100 con 0x7f6ec00a0750 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3437871981 0 0) 0x7f6ea0003520 con 0x7f6ec009e570 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ec00a3250 con 0x7f6ec009e570 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ea8001430 con 0x7f6ec009e570 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:10.010 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): dump: 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1141622141 0 0) 0x7f6ec00a3250 con 0x7f6ec009e570 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 >> v1:172.21.15.148:6790/0 conn(0x7f6ec009ba30 legacy=0x7f6ec01392e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:10.011 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 >> v1:172.21.15.148:6789/0 conn(0x7f6ec00a0750 legacy=0x7f6ec01d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 monclient: found mon.b 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec013aef0 con 0x7f6ec009e570 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 5 monclient: authenticate success, global_id 6484 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 10 monclient: _renew_subs 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.012 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec013d1e0 con 0x7f6ec009e570 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 10 monclient: _renew_subs 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ec013cf00 con 0x7f6ec009e570 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ea80016e0 con 0x7f6ec009e570 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ea8005000 con 0x7f6ec009e570 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6ebd59a640 4 set_mon_vals no callback set 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.995+0000 7f6eb77fe640 10 monclient: dump: 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:13:10.013 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.996+0000 7f6ec51bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.996+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6ea8010430 con 0x7f6ec009e570 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.996+0000 7f6ec51bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.996+0000 7f6ec51bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.996+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea4005080 con 0x7f6ec009e570 2024-09-16T04:13:10.014 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.997+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(254..254 src has 1..254) ==== 6817+0+0 (unknown 2766061615 0 0) 0x7f6ea80479d0 con 0x7f6ec009e570 2024-09-16T04:13:10.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:09.997+0000 7f6ebed9d640 10 monclient: get_auth_request con 0x7f6ea0038b90 auth_method 0 2024-09-16T04:13:10.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.001+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6ea8015dd0 con 0x7f6ec009e570 2024-09-16T04:13:10.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.001+0000 7f6eb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:10.015 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.001+0000 7f6eb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:10.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.177+0000 7f6ec51bd640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:10.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.177+0000 7f6ec51bd640 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:10.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.177+0000 7f6ec51bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.177+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"} v 0) -- 0x7f6ea4005370 con 0x7f6ec009e570 2024-09-16T04:13:10.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.409+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}]=0 v255) ==== 106+0+3 (unknown 340606004 0 944643765) 0x7f6ea80156d0 con 0x7f6ec009e570 2024-09-16T04:13:10.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.409+0000 7f6eb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:10.411 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.409+0000 7f6eb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:10.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.521+0000 7f6ec51bd640 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:10.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.521+0000 7f6ec51bd640 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:10.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.521+0000 7f6ec51bd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:13:10.523 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.521+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"} v 0) -- 0x7f6ea4005bd0 con 0x7f6ec009e570 2024-09-16T04:13:10.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.522+0000 7f6eb77fe640 1 -- 172.21.15.171:0/1655520621 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}]=0 v255) ==== 106+0+3 (unknown 340606004 0 944643765) 0x7f6ea801e220 con 0x7f6ec009e570 2024-09-16T04:13:10.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.522+0000 7f6eb77fe640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "6df951c8-ca70-48bb-b32a-2b7a6ec9adff"}] 2024-09-16T04:13:10.524 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.522+0000 7f6eb77fe640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 >> v1:172.21.15.171:6812/7327273 conn(0x7f6ea0038b90 legacy=0x7f6ea003b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 10 monclient: shutdown 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 >> v1:172.21.15.171:6789/0 conn(0x7f6ec009e570 legacy=0x7f6ec01399f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 shutdown_connections 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 >> 172.21.15.171:0/1655520621 conn(0x7f6ec00a52a0 msgr2=0x7f6ec0071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 shutdown_connections 2024-09-16T04:13:10.526 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:13:10.524+0000 7f6ec51bd640 1 -- 172.21.15.171:0/1655520621 wait complete. 2024-09-16T04:13:10.535 INFO:tasks.workunit.client.0.smithi171.stderr:+ na2=12 2024-09-16T04:13:10.535 INFO:tasks.workunit.client.0.smithi171.stdout:OK 2024-09-16T04:13:10.536 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo OK 2024-09-16T04:13:10.537 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T04:13:10.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T04:13:10.602 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-16T04:13:10.603 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-16T04:13:10.604 DEBUG:teuthology.orchestra.run.smithi171:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-16T04:13:10.663 INFO:tasks.workunit.client.0.smithi171.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ exit_on_error=1 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ ! -z '' ]] 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:++ uname 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ '[' Linux = FreeBSD ']' 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ ETIMEDOUT=110 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-16T04:13:10.664 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=0 2024-09-16T04:13:10.668 INFO:tasks.workunit.client.0.smithi171.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T04:13:10.668 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T04:13:10.668 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-16T04:13:11.167 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=0 2024-09-16T04:13:11.167 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T04:13:11.167 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:13:11.167 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-16T04:13:11.167 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-16T04:13:11.167 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:13:11.168 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T04:13:11.168 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T04:13:11.168 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-16T04:14:22.872 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:14:22.872 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:14:22.872 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:14:22.873 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:14:22.877 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:14:22.877 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T04:14:22.877 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T04:14:22.878 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:14:22.878 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:14:22.878 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:14:37.879 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T04:14:37.879 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T04:14:37.880 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:14:37.880 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:14:37.880 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:14:37.880 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:37.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.986+0000 7fa94c63c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:37.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.986+0000 7fa94c63c640 10 monclient: build_initial_monmap 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.986+0000 7fa94c63c640 10 monclient: monmap: 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:37.987830+0000 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:37.987830+0000 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:37.989 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.987+0000 7fa94c63c640 1 Processor -- start 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.987+0000 7fa94c63c640 1 -- start start 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.987+0000 7fa94c63c640 10 monclient: init 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient(hunting): picked mon.noname-b con 0x7fa94409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient(hunting): picked mon.noname-a con 0x7fa94409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient(hunting): picked mon.noname-c con 0x7fa94409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9440a1010 con 0x7fa94409c660 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94409f030 con 0x7fa94409cc30 2024-09-16T04:14:37.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94409dc60 con 0x7fa94409bb70 2024-09-16T04:14:37.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:37.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94c63c640 10 monclient(hunting): authenticate will time out at 1298.613253s 2024-09-16T04:14:37.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa949bd7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa94409c660 0x7fa944138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40532/0 (socket says 172.21.15.148:40532) 2024-09-16T04:14:37.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa94a3d8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa94409bb70 0x7fa94409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:35248/0 (socket says 172.21.15.148:35248) 2024-09-16T04:14:37.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:37.988+0000 7fa949bd7640 1 -- 172.21.15.148:0/1208281287 learned_addr learned my addr 172.21.15.148:0/1208281287 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): tick 2024-09-16T04:14:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:14:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 monclient(hunting): continuing hunt 2024-09-16T04:14:40.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 -- 172.21.15.148:0/1208281287 >> v1:172.21.15.148:6790/0 conn(0x7fa94409bb70 legacy=0x7fa94409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 -- 172.21.15.148:0/1208281287 >> v1:172.21.15.171:6789/0 conn(0x7fa94409cc30 legacy=0x7fa94413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 -- 172.21.15.148:0/1208281287 >> v1:172.21.15.148:6789/0 conn(0x7fa94409c660 legacy=0x7fa944138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fa9280036e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fa94409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fa94409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 -- 172.21.15.148:0/1208281287 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94409df40 con 0x7fa9280036e0 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 -- 172.21.15.148:0/1208281287 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92800f340 con 0x7fa94409cc30 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 1 -- 172.21.15.148:0/1208281287 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa928010590 con 0x7fa94409bb70 2024-09-16T04:14:40.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:40.988+0000 7fa93bfff640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:42.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611570991 0 0) 0x7fa92800f340 con 0x7fa94409cc30 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient(hunting): _negotiate 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient(hunting): my global_id is 6565 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa930003520 con 0x7fa94409cc30 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4007813817 0 0) 0x7fa930003520 con 0x7fa94409cc30 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 >> v1:172.21.15.148:6790/0 conn(0x7fa94409bb70 legacy=0x7fa9280093c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 >> v1:172.21.15.148:6789/0 conn(0x7fa9280036e0 legacy=0x7fa928005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: _finish_hunting 0 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 monclient: found mon.noname-c 2024-09-16T04:14:42.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa928011880 con 0x7fa94409cc30 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: _finish_auth 0 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: _check_auth_tickets 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa93c001650 con 0x7fa94409cc30 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.886+0000 7fa9493d6640 10 monclient: dump: 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:42.891 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa94c63c640 5 monclient: authenticate success, global_id 6565 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa94c63c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa93c001970 con 0x7fa94409cc30 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa9493d6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa9493d6640 1 -- 172.21.15.148:0/1208281287 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa93c001d40 con 0x7fa94409cc30 2024-09-16T04:14:42.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa9493d6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa9493d6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa9493d6640 10 monclient: dump: 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:42.893 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa94c63c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.887+0000 7fa94c63c640 4 set_mon_vals no callback set 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 10 monclient: shutdown 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 1 -- 172.21.15.148:0/1208281287 >> v1:172.21.15.171:6789/0 conn(0x7fa94409cc30 legacy=0x7fa92800cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 1 -- 172.21.15.148:0/1208281287 shutdown_connections 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 1 -- 172.21.15.148:0/1208281287 >> 172.21.15.148:0/1208281287 conn(0x7fa9440a5640 msgr2=0x7fa9440a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 1 -- 172.21.15.148:0/1208281287 shutdown_connections 2024-09-16T04:14:42.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.888+0000 7fa94c63c640 1 -- 172.21.15.148:0/1208281287 wait complete. 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.889+0000 7fa94c63c640 10 monclient: build_initial_monmap 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.889+0000 7fa94c63c640 10 monclient: monmap: 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:42.899 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.889+0000 7fa94c63c640 1 Processor -- start 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.889+0000 7fa94c63c640 1 -- start start 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.889+0000 7fa94c63c640 10 monclient: init 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient(hunting): picked mon.noname-b con 0x7fa9280036e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient(hunting): picked mon.noname-c con 0x7fa94409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient(hunting): picked mon.noname-a con 0x7fa94409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:42.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa944120f20 con 0x7fa94409cc30 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa944072bd0 con 0x7fa9280036e0 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa944120c40 con 0x7fa94409bb70 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94c63c640 10 monclient(hunting): authenticate will time out at 1303.515683s 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94abd9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa94409cc30 0x7fa9441d1470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49576/0 (socket says 172.21.15.148:49576) 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa94abd9640 1 -- 172.21.15.148:0/736576637 learned_addr learned my addr 172.21.15.148:0/736576637 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.890+0000 7fa949bd7640 1 --1- 172.21.15.148:0/736576637 >> v1:172.21.15.148:6790/0 conn(0x7fa94409bb70 0x7fa944133840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57470/0 (socket says 172.21.15.148:57470) 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573788160 0 0) 0x7fa944120f20 con 0x7fa94409cc30 2024-09-16T04:14:42.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): my global_id is 65007 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa92800eac0 con 0x7fa94409cc30 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4001463745 0 0) 0x7fa92800eac0 con 0x7fa94409cc30 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa944120f20 con 0x7fa94409cc30 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa93c001650 con 0x7fa94409cc30 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:42.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): dump: 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:42.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.893+0000 7fa93affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 900407862 0 0) 0x7fa944120f20 con 0x7fa94409cc30 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 >> v1:172.21.15.148:6790/0 conn(0x7fa94409bb70 legacy=0x7fa944133840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 >> v1:172.21.15.171:6789/0 conn(0x7fa9280036e0 legacy=0x7fa9441c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 monclient: found mon.a 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94411f7f0 con 0x7fa94409cc30 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: _finish_auth 0 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 5 monclient: authenticate success, global_id 65007 2024-09-16T04:14:42.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 10 monclient: _renew_subs 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa944133f50 con 0x7fa94409cc30 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 10 monclient: _renew_subs 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9441d2d70 con 0x7fa94409cc30 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa93c001970 con 0x7fa94409cc30 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa93c003c10 con 0x7fa94409cc30 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:42.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa948bd5640 4 set_mon_vals no callback set 2024-09-16T04:14:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.894+0000 7fa93affd640 10 monclient: dump: 2024-09-16T04:14:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:42.908 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.895+0000 7fa94c63c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.895+0000 7fa94c63c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.895+0000 7fa94c63c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:42.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.895+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa92c005080 con 0x7fa94409cc30 2024-09-16T04:14:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.895+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa93c003db0 con 0x7fa94409cc30 2024-09-16T04:14:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.896+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fa93c047a00 con 0x7fa94409cc30 2024-09-16T04:14:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.896+0000 7fa94a3d8640 10 monclient: get_auth_request con 0x7fa928039b20 auth_method 0 2024-09-16T04:14:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.901+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa93c015ca0 con 0x7fa94409cc30 2024-09-16T04:14:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.901+0000 7fa93affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:42.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:42.901+0000 7fa93affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:42.940 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:14:42.941 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:14:42.941 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:42.941 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph auth del client.bazar 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.024+0000 7f01286d9640 10 monclient: get_monmap_and_config 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.024+0000 7f01286d9640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.024+0000 7f01286d9640 10 monclient: monmap: 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:14:43.026249+0000 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:14:43.026249+0000 2024-09-16T04:14:43.027 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.028 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.028 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.028 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:43.028 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:43.028 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 1 Processor -- start 2024-09-16T04:14:43.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 1 -- start start 2024-09-16T04:14:43.036 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient: init 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient(hunting): picked mon.noname-c con 0x7f012009ba30 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient(hunting): picked mon.noname-b con 0x7f0120073570 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient(hunting): picked mon.noname-a con 0x7f012009dcb0 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01200a0460 con 0x7f012009dcb0 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01200a0db0 con 0x7f012009ba30 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01200a11b0 con 0x7f0120073570 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.025+0000 7f01286d9640 10 monclient(hunting): authenticate will time out at 1258.531932s 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0126475640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f012009ba30 0x7f012009f640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47086/0 (socket says 172.21.15.171:47086) 2024-09-16T04:14:43.037 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0126475640 1 -- 172.21.15.171:0/1589135916 learned_addr learned my addr 172.21.15.171:0/1589135916 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139568574 0 0) 0x7f01200a0db0 con 0x7f012009ba30 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): my global_id is 6571 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c004630 con 0x7f012009ba30 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102325120 0 0) 0x7f01200a0460 con 0x7f012009dcb0 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): my global_id is 65010 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01200a0db0 con 0x7f012009dcb0 2024-09-16T04:14:43.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253717243 0 0) 0x7f01200a11b0 con 0x7f0120073570 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): my global_id is 84104 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01200a0460 con 0x7f0120073570 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125c74640 1 -- 172.21.15.171:0/1589135916 >> v1:172.21.15.148:6790/0 conn(0x7f0120073570 legacy=0x7f012009fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125c74640 1 -- 172.21.15.171:0/1589135916 >> v1:172.21.15.148:6790/0 conn(0x7f0120073570 legacy=0x7f012009fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125c74640 1 --1- 172.21.15.171:0/1589135916 >> v1:172.21.15.148:6790/0 conn(0x7f0120073570 0x7f012009fd50 :-1 s=OPENED pgs=618 cs=1 l=1).handle_message read tag failed 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125c74640 1 --1- 172.21.15.171:0/1589135916 >> v1:172.21.15.148:6790/0 conn(0x7f0120073570 0x7f012009fd50 :-1 s=OPENED pgs=618 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6790/0 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3826968645 0 0) 0x7f010c004630 con 0x7f012009ba30 2024-09-16T04:14:43.039 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 >> v1:172.21.15.148:6790/0 conn(0x7f0120073570 legacy=0x7f012009fd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 >> v1:172.21.15.148:6789/0 conn(0x7f012009dcb0 legacy=0x7f012013b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 monclient: found mon.noname-c 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012009cff0 con 0x7f012009ba30 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f011c001610 con 0x7f012009ba30 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: dump: 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f01286d9640 5 monclient: authenticate success, global_id 6571 2024-09-16T04:14:43.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f01286d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f011c001870 con 0x7f012009ba30 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 1 -- 172.21.15.171:0/1589135916 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f011c001c40 con 0x7f012009ba30 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f01286d9640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f01286d9640 4 set_mon_vals no callback set 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.026+0000 7f0125473640 10 monclient: dump: 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.042 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 10 monclient: shutdown 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 1 -- 172.21.15.171:0/1589135916 >> v1:172.21.15.171:6789/0 conn(0x7f012009ba30 legacy=0x7f012009f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 1 -- 172.21.15.171:0/1589135916 shutdown_connections 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 1 -- 172.21.15.171:0/1589135916 >> 172.21.15.171:0/1589135916 conn(0x7f01200a52a0 msgr2=0x7f01200a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.043 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 1 -- 172.21.15.171:0/1589135916 shutdown_connections 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.027+0000 7f01286d9640 1 -- 172.21.15.171:0/1589135916 wait complete. 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.028+0000 7f01286d9640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.028+0000 7f01286d9640 10 monclient: monmap: 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:43.044 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.028+0000 7f01286d9640 1 Processor -- start 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.028+0000 7f01286d9640 1 -- start start 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.028+0000 7f01286d9640 10 monclient: init 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient(hunting): picked mon.noname-b con 0x7f0120073570 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient(hunting): picked mon.noname-c con 0x7f012009ba30 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient(hunting): picked mon.noname-a con 0x7f012009dcb0 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01201c18e0 con 0x7f012009dcb0 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01201a6070 con 0x7f0120073570 2024-09-16T04:14:43.045 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01201a8b30 con 0x7f012009ba30 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient(hunting): authenticate will time out at 1258.535145s 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0126475640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f0120073570 0x7f01201c11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:47090/0 (socket says 172.21.15.171:47090) 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0126475640 1 -- 172.21.15.171:0/3241026810 learned_addr learned my addr 172.21.15.171:0/3241026810 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092148752 0 0) 0x7f01201a6070 con 0x7f0120073570 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): my global_id is 6574 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0104003520 con 0x7f0120073570 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1513185025 0 0) 0x7f01201a8b30 con 0x7f012009ba30 2024-09-16T04:14:43.046 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): my global_id is 84107 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01201a6070 con 0x7f012009ba30 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876968198 0 0) 0x7f01201c18e0 con 0x7f012009dcb0 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): my global_id is 65013 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01201a8b30 con 0x7f012009dcb0 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4078026408 0 0) 0x7f0104003520 con 0x7f0120073570 2024-09-16T04:14:43.047 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01201c18e0 con 0x7f0120073570 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f011c001510 con 0x7f0120073570 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.048 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 303525046 0 0) 0x7f01201a6070 con 0x7f012009ba30 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0104003520 con 0x7f012009ba30 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f01140019e0 con 0x7f012009ba30 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.049 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2595272909 0 0) 0x7f01201a8b30 con 0x7f012009dcb0 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f01201a6070 con 0x7f012009dcb0 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0118001b70 con 0x7f012009dcb0 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.050 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.051 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3980044077 0 0) 0x7f01201c18e0 con 0x7f0120073570 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 >> v1:172.21.15.148:6790/0 conn(0x7f012009ba30 legacy=0x7f01201cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 >> v1:172.21.15.148:6789/0 conn(0x7f012009dcb0 legacy=0x7f01201cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 monclient: found mon.b 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01201a6860 con 0x7f0120073570 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f0112ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.052 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 5 monclient: authenticate success, global_id 6574 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient: _renew_subs 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.029+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f01201a8850 con 0x7f0120073570 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 10 monclient: _renew_subs 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f01201c3c40 con 0x7f0120073570 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f011c001870 con 0x7f0120073570 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f011c004d90 con 0x7f0120073570 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.053 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0124c72640 4 set_mon_vals no callback set 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 10 monclient: dump: 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:14:43.054 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f011c010020 con 0x7f0120073570 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.030+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0108005080 con 0x7f0120073570 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.031+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7f011c047810 con 0x7f0120073570 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.031+0000 7f0125c74640 10 monclient: get_auth_request con 0x7f0104039260 auth_method 0 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.036+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f011c015ad0 con 0x7f0120073570 2024-09-16T04:14:43.055 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.036+0000 7f0112ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.056 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.036+0000 7f0112ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa94c63c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa94c63c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa94c63c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa92c005370 con 0x7fa94409cc30 2024-09-16T04:14:43.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa93affd640 1 -- 172.21.15.148:0/736576637 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3446609435) 0x7fa93c01e150 con 0x7fa94409cc30 2024-09-16T04:14:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa93affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:43.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.139+0000 7fa93affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:43.142 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 >> v1:172.21.15.171:6812/7327273 conn(0x7fa928039b20 legacy=0x7fa92803bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 10 monclient: shutdown 2024-09-16T04:14:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 >> v1:172.21.15.148:6789/0 conn(0x7fa94409cc30 legacy=0x7fa9441d1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 shutdown_connections 2024-09-16T04:14:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 >> 172.21.15.148:0/736576637 conn(0x7fa9440a5640 msgr2=0x7fa9440a4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 shutdown_connections 2024-09-16T04:14:43.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.141+0000 7fa94c63c640 1 -- 172.21.15.148:0/736576637 wait complete. 2024-09-16T04:14:43.154 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:14:43.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:14:43.154 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:14:43.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.250+0000 7f99cdb98640 10 monclient: get_monmap_and_config 2024-09-16T04:14:43.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.250+0000 7f99cdb98640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.250+0000 7f99cdb98640 10 monclient: monmap: 2024-09-16T04:14:43.252 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.252 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.252 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:43.251584+0000 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:43.251584+0000 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:43.253 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 1 Processor -- start 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 1 -- start start 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient: init 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient(hunting): picked mon.noname-a con 0x7f99c809ebd0 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient(hunting): picked mon.noname-b con 0x7f99c809efb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient(hunting): picked mon.noname-c con 0x7f99c809f420 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c8137290 con 0x7f99c809ebd0 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c80a1910 con 0x7f99c809f420 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c809cbd0 con 0x7f99c809efb0 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.251+0000 7f99cdb98640 10 monclient(hunting): authenticate will time out at 1303.876964s 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f99c809efb0 0x7f99c813bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57486/0 (socket says 172.21.15.148:57486) 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c77fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f99c809ebd0 0x7f99c8136b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49590/0 (socket says 172.21.15.148:49590) 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c6ffd640 1 -- 172.21.15.148:0/2107946910 learned_addr learned my addr 172.21.15.148:0/2107946910 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824917252 0 0) 0x7f99c8137290 con 0x7f99c809ebd0 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): my global_id is 65016 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b0003520 con 0x7f99c809ebd0 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 965616089 0 0) 0x7f99c809cbd0 con 0x7f99c809efb0 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): my global_id is 84110 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c8137290 con 0x7f99c809efb0 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4143398098 0 0) 0x7f99c80a1910 con 0x7f99c809f420 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient(hunting): my global_id is 6577 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c809cbd0 con 0x7f99c809f420 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3324338051 0 0) 0x7f99c8137290 con 0x7f99c809efb0 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 >> v1:172.21.15.171:6789/0 conn(0x7f99c809f420 legacy=0x7f99c813f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 >> v1:172.21.15.148:6789/0 conn(0x7f99c809ebd0 legacy=0x7f99c8136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 monclient: found mon.noname-b 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c809ceb0 con 0x7f99c809efb0 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99bc0015f0 con 0x7f99c809efb0 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: dump: 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.267 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99cdb98640 5 monclient: authenticate success, global_id 84110 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99cdb98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f99bc001860 con 0x7f99c809efb0 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 1 -- 172.21.15.148:0/2107946910 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99bc001c70 con 0x7f99c809efb0 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99cdb98640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99cdb98640 4 set_mon_vals no callback set 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.252+0000 7f99c67fc640 10 monclient: dump: 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 10 monclient: shutdown 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 1 -- 172.21.15.148:0/2107946910 >> v1:172.21.15.148:6790/0 conn(0x7f99c809efb0 legacy=0x7f99c813bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 1 -- 172.21.15.148:0/2107946910 shutdown_connections 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 1 -- 172.21.15.148:0/2107946910 >> 172.21.15.148:0/2107946910 conn(0x7f99c80a5860 msgr2=0x7f99c80a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 1 -- 172.21.15.148:0/2107946910 shutdown_connections 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 1 -- 172.21.15.148:0/2107946910 wait complete. 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.253+0000 7f99cdb98640 10 monclient: monmap: 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.270 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 1 Processor -- start 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 1 -- start start 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient: init 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient(hunting): picked mon.noname-b con 0x7f99c809ebd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient(hunting): picked mon.noname-c con 0x7f99c809efb0 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient(hunting): picked mon.noname-a con 0x7f99c809f420 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c812aa30 con 0x7f99c809f420 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c813e940 con 0x7f99c809ebd0 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c813e280 con 0x7f99c809efb0 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99cdb98640 10 monclient(hunting): authenticate will time out at 1303.879775s 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99c6ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f99c809efb0 0x7f99c81d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57488/0 (socket says 172.21.15.148:57488) 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.254+0000 7f99c6ffd640 1 -- 172.21.15.148:0/1121201652 learned_addr learned my addr 172.21.15.148:0/1121201652 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450599265 0 0) 0x7f99c813e280 con 0x7f99c809efb0 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): my global_id is 84113 2024-09-16T04:14:43.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b4003520 con 0x7f99c809efb0 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051717255 0 0) 0x7f99c812aa30 con 0x7f99c809f420 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): my global_id is 65019 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c813e280 con 0x7f99c809f420 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523000010 0 0) 0x7f99c813e940 con 0x7f99c809ebd0 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): my global_id is 6580 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c812aa30 con 0x7f99c809ebd0 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753087148 0 0) 0x7f99b4003520 con 0x7f99c809efb0 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99c813e940 con 0x7f99c809efb0 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99bc001530 con 0x7f99c809efb0 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): dump: 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.274 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4550438 0 0) 0x7f99c813e280 con 0x7f99c809f420 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99b4003520 con 0x7f99c809f420 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 165581182 0 0) 0x7f99c812aa30 con 0x7f99c809ebd0 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99c813e280 con 0x7f99c809ebd0 2024-09-16T04:14:43.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99c0001bb0 con 0x7f99c809f420 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): dump: 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.276 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99b8001190 con 0x7f99c809ebd0 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient(hunting): dump: 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.277 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1001474390 0 0) 0x7f99b4003520 con 0x7f99c809f420 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.148:6790/0 conn(0x7f99c809efb0 legacy=0x7f99c81d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.171:6789/0 conn(0x7f99c809ebd0 legacy=0x7f99c81cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 monclient: found mon.a 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c812ad10 con 0x7f99c809f420 2024-09-16T04:14:43.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 5 monclient: authenticate success, global_id 65019 2024-09-16T04:14:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 10 monclient: _renew_subs 2024-09-16T04:14:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99c809b6b0 con 0x7f99c809f420 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 10 monclient: _renew_subs 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99c813d1f0 con 0x7f99c809f420 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f99c0003970 con 0x7f99c809f420 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99c0003cc0 con 0x7f99c809f420 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99c5ffb640 4 set_mon_vals no callback set 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.255+0000 7f99affff640 10 monclient: dump: 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.280 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f99c000ef50 con 0x7f99c809f420 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99cdb98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f99a80065c0 con 0x7f99a8002860 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7f99c00467b0 con 0x7f99c809f420 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99c6ffd640 10 monclient: get_auth_request con 0x7f99b4039160 auth_method 0 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022940319 0 0) 0x7f99a80065c0 con 0x7f99a8002860 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient(hunting): my global_id is 65022 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b4003520 con 0x7f99a8002860 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4286781388 0 0) 0x7f99b4003520 con 0x7f99a8002860 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99a80065c0 con 0x7f99a8002860 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99b80047e0 con 0x7f99a8002860 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.256+0000 7f99affff640 10 monclient: dump: 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1111915280 0 0) 0x7f99a80065c0 con 0x7f99a8002860 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f99a800a150 con 0x7f99a8002860 2024-09-16T04:14:43.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f99b8006b10 con 0x7f99a8002860 2024-09-16T04:14:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.148:6789/0 conn(0x7f99a8002860 legacy=0x7f99a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.257+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.148:6789/0 conn(0x7f99a8002860 legacy=0x7f99a8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.263+0000 7f01286d9640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.263+0000 7f01286d9640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.263+0000 7f01286d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.263+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) -- 0x7f0108005370 con 0x7f0120073570 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.267+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v14) ==== 82+0+0 (unknown 2871494026 0 0) 0x7f011c0153d0 con 0x7f0120073570 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.267+0000 7f0112ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T04:14:43.285 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.267+0000 7f0112ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99cdb98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f99a8009e40 con 0x7f99a8009100 2024-09-16T04:14:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1105486728 0 0) 0x7f99a8009e40 con 0x7f99a8009100 2024-09-16T04:14:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient(hunting): my global_id is 65025 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99a80065c0 con 0x7f99a8009100 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1450775522 0 0) 0x7f99a80065c0 con 0x7f99a8009100 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99a8009e40 con 0x7f99a8009100 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99b8004de0 con 0x7f99a8009100 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.263+0000 7f99affff640 10 monclient: dump: 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.288 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 355293914 0 0) 0x7f99a8009e40 con 0x7f99a8009100 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f99a800c3d0 con 0x7f99a8009100 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4146 (unknown 0 0 3827722130) 0x7f99b8006180 con 0x7f99a8009100 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:43.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.148:6789/0 conn(0x7f99a8009100 legacy=0x7f99a8009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99affff640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.148:6789/0 conn(0x7f99a8009100 legacy=0x7f99a8009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 38, 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 262462, 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:14:43.290 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.291 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.292 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.293 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.293 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:43.293 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:14:43.293 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.293 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.294 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:14:43.295 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.296 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.297 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:14:43.298 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:14:43.299 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.300 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.301 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.302 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:14:43.303 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.171:6812/7327273 conn(0x7f99b4039160 legacy=0x7f99b403b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 10 monclient: shutdown 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 >> v1:172.21.15.148:6789/0 conn(0x7f99c809f420 legacy=0x7f99c81d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99c7fff640 1 -- 172.21.15.148:0/1121201652 reap_dead start 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99c7fff640 1 -- 172.21.15.148:0/1121201652 reap_dead start 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 shutdown_connections 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 >> 172.21.15.148:0/1121201652 conn(0x7f99c80a5860 msgr2=0x7f99c809bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 shutdown_connections 2024-09-16T04:14:43.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.264+0000 7f99cdb98640 1 -- 172.21.15.148:0/1121201652 wait complete. 2024-09-16T04:14:43.306 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:43.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:14:43.306 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:14:43.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.404+0000 7fcabbf86640 10 monclient: get_monmap_and_config 2024-09-16T04:14:43.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.404+0000 7fcabbf86640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.404+0000 7fcabbf86640 10 monclient: monmap: 2024-09-16T04:14:43.406 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:43.405747+0000 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:43.405747+0000 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:43.407 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.405+0000 7fcabbf86640 1 Processor -- start 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.405+0000 7fcabbf86640 1 -- start start 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.405+0000 7fcabbf86640 10 monclient: init 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient(hunting): picked mon.noname-a con 0x7fcab4133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient(hunting): picked mon.noname-c con 0x7fcab406c430 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient(hunting): picked mon.noname-b con 0x7fcab413d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab409b0d0 con 0x7fcab4133910 2024-09-16T04:14:43.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab409b340 con 0x7fcab406c430 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab409b620 con 0x7fcab413d0e0 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient(hunting): authenticate will time out at 1304.031119s 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab9d22640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcab4133910 0x7fcab4133cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49640/0 (socket says 172.21.15.148:49640) 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab9d22640 1 -- 172.21.15.148:0/4212502251 learned_addr learned my addr 172.21.15.148:0/4212502251 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcaba523640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fcab413d0e0 0x7fcab413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57496/0 (socket says 172.21.15.148:57496) 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcaba523640 1 -- 172.21.15.148:0/4212502251 >> v1:172.21.15.148:6790/0 conn(0x7fcab413d0e0 legacy=0x7fcab413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcaba523640 1 -- 172.21.15.148:0/4212502251 >> v1:172.21.15.148:6790/0 conn(0x7fcab413d0e0 legacy=0x7fcab413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcaba523640 1 --1- 172.21.15.148:0/4212502251 >> v1:172.21.15.148:6790/0 conn(0x7fcab413d0e0 0x7fcab413f4d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503198274 0 0) 0x7fcab409b0d0 con 0x7fcab4133910 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): my global_id is 65028 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaa0003520 con 0x7fcab4133910 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 472466916 0 0) 0x7fcab409b340 con 0x7fcab406c430 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient(hunting): my global_id is 6583 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcab409b0d0 con 0x7fcab406c430 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 888939397 0 0) 0x7fcaa0003520 con 0x7fcab4133910 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 >> v1:172.21.15.148:6790/0 conn(0x7fcab413d0e0 legacy=0x7fcab413f4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 >> v1:172.21.15.171:6789/0 conn(0x7fcab406c430 legacy=0x7fcab413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 monclient: found mon.noname-a 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab40633c0 con 0x7fcab4133910 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcab0001610 con 0x7fcab4133910 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 5 monclient: authenticate success, global_id 65028 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:43.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: dump: 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcab0001930 con 0x7fcab4133910 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 1 -- 172.21.15.148:0/4212502251 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcab0001d00 con 0x7fcab4133910 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcabbf86640 4 set_mon_vals no callback set 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.406+0000 7fcab8d20640 10 monclient: dump: 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.422 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 10 monclient: shutdown 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 1 -- 172.21.15.148:0/4212502251 >> v1:172.21.15.148:6789/0 conn(0x7fcab4133910 legacy=0x7fcab4133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 1 -- 172.21.15.148:0/4212502251 shutdown_connections 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 1 -- 172.21.15.148:0/4212502251 >> 172.21.15.148:0/4212502251 conn(0x7fcab4062630 msgr2=0x7fcab4062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 1 -- 172.21.15.148:0/4212502251 shutdown_connections 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 1 -- 172.21.15.148:0/4212502251 wait complete. 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.407+0000 7fcabbf86640 10 monclient: monmap: 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:43.424 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 1 Processor -- start 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 1 -- start start 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient: init 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient(hunting): picked mon.noname-a con 0x7fcab406c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient(hunting): picked mon.noname-b con 0x7fcab4133910 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient(hunting): picked mon.noname-c con 0x7fcab413d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab4133250 con 0x7fcab406c430 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab41319c0 con 0x7fcab4133910 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab40a4e30 con 0x7fcab413d0e0 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcabbf86640 10 monclient(hunting): authenticate will time out at 1304.033813s 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcab9d22640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcab406c430 0x7fcab41cb230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49642/0 (socket says 172.21.15.148:49642) 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.408+0000 7fcab9d22640 1 -- 172.21.15.148:0/2748946366 learned_addr learned my addr 172.21.15.148:0/2748946366 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578609556 0 0) 0x7fcab40a4e30 con 0x7fcab413d0e0 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): my global_id is 84116 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca98003520 con 0x7fcab413d0e0 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284833549 0 0) 0x7fcab4133250 con 0x7fcab406c430 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): my global_id is 65031 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcab40a4e30 con 0x7fcab406c430 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593077736 0 0) 0x7fcab41319c0 con 0x7fcab4133910 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): my global_id is 6586 2024-09-16T04:14:43.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcab4133250 con 0x7fcab4133910 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3913469911 0 0) 0x7fcab40a4e30 con 0x7fcab406c430 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcab41319c0 con 0x7fcab406c430 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcab00015d0 con 0x7fcab406c430 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): dump: 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.428 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1004182884 0 0) 0x7fca98003520 con 0x7fcab413d0e0 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcab40a4e30 con 0x7fcab413d0e0 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcaac001850 con 0x7fcab413d0e0 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): dump: 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.429 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2704805836 0 0) 0x7fcab4133250 con 0x7fcab4133910 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca98003520 con 0x7fcab4133910 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcaa40018d0 con 0x7fcab4133910 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient(hunting): dump: 2024-09-16T04:14:43.430 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1988050038 0 0) 0x7fcab41319c0 con 0x7fcab406c430 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.148:6790/0 conn(0x7fcab413d0e0 legacy=0x7fcab41d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.171:6789/0 conn(0x7fcab4133910 legacy=0x7fcab413bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 monclient: found mon.a 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab4132f70 con 0x7fcab406c430 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 5 monclient: authenticate success, global_id 65031 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 10 monclient: _renew_subs 2024-09-16T04:14:43.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcab4131e00 con 0x7fcab406c430 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 10 monclient: _renew_subs 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcab413c460 con 0x7fcab406c430 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcab0004cc0 con 0x7fcab406c430 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcab0004e70 con 0x7fcab406c430 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaabfff640 4 set_mon_vals no callback set 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.409+0000 7fcaaa7fc640 10 monclient: dump: 2024-09-16T04:14:43.433 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcabbf86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fca9c0065c0 con 0x7fca9c002860 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcab00102a0 con 0x7fcab406c430 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fcab0047a00 con 0x7fcab406c430 2024-09-16T04:14:43.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360907355 0 0) 0x7fca9c0065c0 con 0x7fca9c002860 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 10 monclient(hunting): my global_id is 84119 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcab41319c0 con 0x7fca9c002860 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaba523640 10 monclient: get_auth_request con 0x7fca98039160 auth_method 0 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.410+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 432825557 0 0) 0x7fcab41319c0 con 0x7fca9c002860 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca9c0065c0 con 0x7fca9c002860 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcaa4003790 con 0x7fca9c002860 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 10 monclient: dump: 2024-09-16T04:14:43.435 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1592922424 0 0) 0x7fca9c0065c0 con 0x7fca9c002860 2024-09-16T04:14:43.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca9c00a150 con 0x7fca9c002860 2024-09-16T04:14:43.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcaa40059d0 con 0x7fca9c002860 2024-09-16T04:14:43.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:43.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.148:6790/0 conn(0x7fca9c002860 legacy=0x7fca9c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.411+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.148:6790/0 conn(0x7fca9c002860 legacy=0x7fca9c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.439 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.429+0000 7f01286d9640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.429+0000 7f01286d9640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.429+0000 7f01286d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.429+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) -- 0x7f0108002210 con 0x7f0120073570 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.430+0000 7f0112ffd640 1 -- 172.21.15.171:0/3241026810 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v14) ==== 82+0+0 (unknown 2871494026 0 0) 0x7f011c01df20 con 0x7f0120073570 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.430+0000 7f0112ffd640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.430+0000 7f0112ffd640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 >> v1:172.21.15.171:6812/7327273 conn(0x7f0104039260 legacy=0x7f010403b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 10 monclient: shutdown 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 >> v1:172.21.15.171:6789/0 conn(0x7f0120073570 legacy=0x7f01201c11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.440 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 shutdown_connections 2024-09-16T04:14:43.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 >> 172.21.15.171:0/3241026810 conn(0x7f01200a52a0 msgr2=0x7f0120071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 shutdown_connections 2024-09-16T04:14:43.441 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:14:43.432+0000 7f01286d9640 1 -- 172.21.15.171:0/3241026810 wait complete. 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcabbf86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fca9c009e40 con 0x7fca9c009100 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3930837227 0 0) 0x7fca9c009e40 con 0x7fca9c009100 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcaaa7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcaaa7fc640 10 monclient(hunting): my global_id is 84122 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.417+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca9c0065c0 con 0x7fca9c009100 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2950236472 0 0) 0x7fca9c0065c0 con 0x7fca9c009100 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca9c009e40 con 0x7fca9c009100 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcaa4005060 con 0x7fca9c009100 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 10 monclient: dump: 2024-09-16T04:14:43.442 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 936336425 0 0) 0x7fca9c009e40 con 0x7fca9c009100 2024-09-16T04:14:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca9c00c3d0 con 0x7fca9c009100 2024-09-16T04:14:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 3619257913) 0x7fcaa4004f20 con 0x7fca9c009100 2024-09-16T04:14:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:43.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.148:6790/0 conn(0x7fca9c009100 legacy=0x7fca9c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 38, 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 20453, 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:14:43.445 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.446 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.447 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.448 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:14:43.449 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.450 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:14:43.451 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.452 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.453 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.454 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:14:43.455 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.456 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.457 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.458 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:43.458 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.458 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.458 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:14:43.458 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:14:43.459 INFO:tasks.workunit.client.0.smithi171.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-16T04:14:43.459 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-16T04:14:43.460 INFO:tasks.workunit.client.0.smithi171.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T04:14:43.460 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-16T04:14:43.461 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=0 2024-09-16T04:14:43.461 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T04:14:43.461 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-16T04:14:43.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaaa7fc640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.148:6790/0 conn(0x7fca9c009100 legacy=0x7fca9c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.171:6812/7327273 conn(0x7fca98039160 legacy=0x7fca9803b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 10 monclient: shutdown 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 >> v1:172.21.15.148:6789/0 conn(0x7fcab406c430 legacy=0x7fcab41cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaba523640 1 -- 172.21.15.148:0/2748946366 reap_dead start 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcaba523640 1 -- 172.21.15.148:0/2748946366 reap_dead start 2024-09-16T04:14:43.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 shutdown_connections 2024-09-16T04:14:43.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 >> 172.21.15.148:0/2748946366 conn(0x7fcab4062630 msgr2=0x7fcab40a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 shutdown_connections 2024-09-16T04:14:43.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.418+0000 7fcabbf86640 1 -- 172.21.15.148:0/2748946366 wait complete. 2024-09-16T04:14:43.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:43.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:14:43.463 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:14:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.560+0000 7fec7403c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.560+0000 7fec7403c640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.560+0000 7fec7403c640 10 monclient: monmap: 2024-09-16T04:14:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.562 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:43.561504+0000 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:43.561504+0000 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:43.563 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 1 Processor -- start 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 1 -- start start 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient: init 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient(hunting): picked mon.noname-b con 0x7fec6c09aa60 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient(hunting): picked mon.noname-c con 0x7fec6c09b550 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient(hunting): picked mon.noname-a con 0x7fec6c09fa50 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c0a0d20 con 0x7fec6c09fa50 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c09c030 con 0x7fec6c09b550 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c09fe60 con 0x7fec6c09aa60 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec7403c640 10 monclient(hunting): authenticate will time out at 1304.186812s 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec71dd8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fec6c09aa60 0x7fec6c09ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57516/0 (socket says 172.21.15.148:57516) 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec725d9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fec6c09fa50 0x7fec6c13f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49654/0 (socket says 172.21.15.148:49654) 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.561+0000 7fec71dd8640 1 -- 172.21.15.148:0/3319388324 learned_addr learned my addr 172.21.15.148:0/3319388324 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 745323406 0 0) 0x7fec64001490 con 0x7fec6c09fa50 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.573 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): my global_id is 65040 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec58003520 con 0x7fec6c09fa50 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420897747 0 0) 0x7fec6c09fe60 con 0x7fec6c09aa60 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): my global_id is 84131 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec64001490 con 0x7fec6c09aa60 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883211118 0 0) 0x7fec6c09c030 con 0x7fec6c09b550 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient(hunting): my global_id is 6595 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec6c09fe60 con 0x7fec6c09b550 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 477927307 0 0) 0x7fec64001490 con 0x7fec6c09aa60 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 >> v1:172.21.15.171:6789/0 conn(0x7fec6c09b550 legacy=0x7fec6c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 >> v1:172.21.15.148:6789/0 conn(0x7fec6c09fa50 legacy=0x7fec6c13f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 monclient: found mon.noname-b 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c073210 con 0x7fec6c09aa60 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec680015f0 con 0x7fec6c09aa60 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec7403c640 5 monclient: authenticate success, global_id 84131 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec7403c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: dump: 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.576 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec7403c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec680018d0 con 0x7fec6c09aa60 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 1 -- 172.21.15.148:0/3319388324 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec68001ce0 con 0x7fec6c09aa60 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec70dd6640 10 monclient: dump: 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.577 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec7403c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.562+0000 7fec7403c640 4 set_mon_vals no callback set 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 10 monclient: shutdown 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 1 -- 172.21.15.148:0/3319388324 >> v1:172.21.15.148:6790/0 conn(0x7fec6c09aa60 legacy=0x7fec6c09ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 1 -- 172.21.15.148:0/3319388324 shutdown_connections 2024-09-16T04:14:43.578 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 1 -- 172.21.15.148:0/3319388324 >> 172.21.15.148:0/3319388324 conn(0x7fec6c0a5050 msgr2=0x7fec6c0a5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 1 -- 172.21.15.148:0/3319388324 shutdown_connections 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 1 -- 172.21.15.148:0/3319388324 wait complete. 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.563+0000 7fec7403c640 10 monclient: monmap: 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.579 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 1 Processor -- start 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 1 -- start start 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient: init 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient(hunting): picked mon.noname-b con 0x7fec6c09aa60 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient(hunting): picked mon.noname-a con 0x7fec6c09b550 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient(hunting): picked mon.noname-c con 0x7fec6c09fa50 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c137e90 con 0x7fec6c09b550 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c13a180 con 0x7fec6c09aa60 2024-09-16T04:14:43.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c13f8b0 con 0x7fec6c09fa50 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec7403c640 10 monclient(hunting): authenticate will time out at 1304.189691s 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec715d7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fec6c09b550 0x7fec6c136770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49664/0 (socket says 172.21.15.148:49664) 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec715d7640 1 -- 172.21.15.148:0/1053121964 learned_addr learned my addr 172.21.15.148:0/1053121964 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956727488 0 0) 0x7fec6c13f8b0 con 0x7fec6c09fa50 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): my global_id is 84134 2024-09-16T04:14:43.581 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec50003520 con 0x7fec6c09fa50 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956451093 0 0) 0x7fec6c137e90 con 0x7fec6c09b550 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 10 monclient(hunting): my global_id is 65043 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.564+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec6c13f8b0 con 0x7fec6c09b550 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784507142 0 0) 0x7fec6c13a180 con 0x7fec6c09aa60 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): my global_id is 6598 2024-09-16T04:14:43.582 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec6c137e90 con 0x7fec6c09aa60 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3527320808 0 0) 0x7fec50003520 con 0x7fec6c09fa50 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec6c13a180 con 0x7fec6c09fa50 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1910359056 0 0) 0x7fec6c13f8b0 con 0x7fec6c09b550 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec50003520 con 0x7fec6c09b550 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec64001690 con 0x7fec6c09fa50 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): dump: 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.583 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec5c0019d0 con 0x7fec6c09b550 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): dump: 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.584 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4167579095 0 0) 0x7fec6c137e90 con 0x7fec6c09aa60 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec6c13f8b0 con 0x7fec6c09aa60 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec680014f0 con 0x7fec6c09aa60 2024-09-16T04:14:43.585 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient(hunting): dump: 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3747609470 0 0) 0x7fec6c13a180 con 0x7fec6c09fa50 2024-09-16T04:14:43.586 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.171:6789/0 conn(0x7fec6c09aa60 legacy=0x7fec6c136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.148:6789/0 conn(0x7fec6c09b550 legacy=0x7fec6c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 monclient: found mon.c 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c13fa80 con 0x7fec6c09fa50 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 5 monclient: authenticate success, global_id 84134 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 10 monclient: _renew_subs 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.587 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec6c1d4cc0 con 0x7fec6c09fa50 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 10 monclient: _renew_subs 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fec6c1d5070 con 0x7fec6c09fa50 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec64004b60 con 0x7fec6c09fa50 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec64004eb0 con 0x7fec6c09fa50 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec63fff640 4 set_mon_vals no callback set 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.565+0000 7fec627fc640 10 monclient: dump: 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.588 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec7403c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fec64010140 con 0x7fec6c09fa50 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec540065c0 con 0x7fec54002860 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fec64047af0 con 0x7fec6c09fa50 2024-09-16T04:14:43.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375178052 0 0) 0x7fec540065c0 con 0x7fec54002860 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient(hunting): my global_id is 6601 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec6c13a180 con 0x7fec54002860 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec715d7640 10 monclient: get_auth_request con 0x7fec50039290 auth_method 0 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2788848605 0 0) 0x7fec6c13a180 con 0x7fec54002860 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec540065c0 con 0x7fec54002860 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec68004980 con 0x7fec54002860 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.566+0000 7fec627fc640 10 monclient: dump: 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.590 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4213288913 0 0) 0x7fec540065c0 con 0x7fec54002860 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec5400a150 con 0x7fec54002860 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fec68005ea0 con 0x7fec54002860 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:43.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.171:6789/0 conn(0x7fec54002860 legacy=0x7fec54004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.567+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.171:6789/0 conn(0x7fec54002860 legacy=0x7fec54004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec7403c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:43.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fec54009e40 con 0x7fec54009100 2024-09-16T04:14:43.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046603272 0 0) 0x7fec54009e40 con 0x7fec54009100 2024-09-16T04:14:43.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 10 monclient(hunting): my global_id is 6604 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec540065c0 con 0x7fec54009100 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.573+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 106791535 0 0) 0x7fec540065c0 con 0x7fec54009100 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec54009e40 con 0x7fec54009100 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fec68006420 con 0x7fec54009100 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 10 monclient: dump: 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.594 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2694924375 0 0) 0x7fec54009e40 con 0x7fec54009100 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec5400c3d0 con 0x7fec54009100 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 1049108563) 0x7fec68006240 con 0x7fec54009100 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.171:6789/0 conn(0x7fec54009100 legacy=0x7fec54009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec627fc640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.171:6789/0 conn(0x7fec54009100 legacy=0x7fec54009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 38, 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 507612, 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:14:43.596 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.597 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.598 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:14:43.599 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:14:43.600 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.602 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.604 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:43.605 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.606 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:14:43.607 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:43.608 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:43.609 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:14:43.609 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:14:43.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.171:6812/7327273 conn(0x7fec50039290 legacy=0x7fec5003b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 10 monclient: shutdown 2024-09-16T04:14:43.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 >> v1:172.21.15.148:6790/0 conn(0x7fec6c09fa50 legacy=0x7fec6c136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec725d9640 1 -- 172.21.15.148:0/1053121964 reap_dead start 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec725d9640 1 -- 172.21.15.148:0/1053121964 reap_dead start 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 shutdown_connections 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 >> 172.21.15.148:0/1053121964 conn(0x7fec6c0a5050 msgr2=0x7fec6c0a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 shutdown_connections 2024-09-16T04:14:43.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.574+0000 7fec7403c640 1 -- 172.21.15.148:0/1053121964 wait complete. 2024-09-16T04:14:43.611 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:43.611 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:14:43.612 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:14:43.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.707+0000 7fddfb21e640 10 monclient: get_monmap_and_config 2024-09-16T04:14:43.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.707+0000 7fddfb21e640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.707+0000 7fddfb21e640 10 monclient: monmap: 2024-09-16T04:14:43.709 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:43.708687+0000 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:43.708687+0000 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:43.710 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 1 Processor -- start 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 1 -- start start 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient: init 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient(hunting): picked mon.noname-c con 0x7fddf40a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient(hunting): picked mon.noname-a con 0x7fddf40a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient(hunting): picked mon.noname-b con 0x7fddf413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf40a0e00 con 0x7fddf40a1b10 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf40a10e0 con 0x7fddf40a2410 2024-09-16T04:14:43.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf40a4970 con 0x7fddf413d060 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.708+0000 7fddfb21e640 10 monclient(hunting): authenticate will time out at 1304.333978s 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf97bb640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fddf413d060 0x7fddf413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57532/0 (socket says 172.21.15.148:57532) 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf3fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fddf40a1b10 0x7fddf413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49672/0 (socket says 172.21.15.148:49672) 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf97bb640 1 -- 172.21.15.148:0/2013862060 learned_addr learned my addr 172.21.15.148:0/2013862060 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459764602 0 0) 0x7fddf40a0e00 con 0x7fddf40a1b10 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): my global_id is 65046 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdde4003520 con 0x7fddf40a1b10 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2184433202 0 0) 0x7fddf40a4970 con 0x7fddf413d060 2024-09-16T04:14:43.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): my global_id is 84137 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf40a0e00 con 0x7fddf413d060 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110352722 0 0) 0x7fddf40a10e0 con 0x7fddf40a2410 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient(hunting): my global_id is 6607 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf40a4970 con 0x7fddf40a2410 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3186866837 0 0) 0x7fdde4003520 con 0x7fddf40a1b10 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 >> v1:172.21.15.148:6790/0 conn(0x7fddf413d060 legacy=0x7fddf413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 >> v1:172.21.15.171:6789/0 conn(0x7fddf40a2410 legacy=0x7fddf40a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf406dfd0 con 0x7fddf40a1b10 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdde0001610 con 0x7fddf40a1b10 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: dump: 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.722 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.723 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.724 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddfb21e640 5 monclient: authenticate success, global_id 65046 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddfb21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdde00018b0 con 0x7fddf40a1b10 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 1 -- 172.21.15.148:0/2013862060 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdde0001cc0 con 0x7fddf40a1b10 2024-09-16T04:14:43.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddfb21e640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddfb21e640 4 set_mon_vals no callback set 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.709+0000 7fddf37fe640 10 monclient: dump: 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.726 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 10 monclient: shutdown 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 1 -- 172.21.15.148:0/2013862060 >> v1:172.21.15.148:6789/0 conn(0x7fddf40a1b10 legacy=0x7fddf413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 1 -- 172.21.15.148:0/2013862060 shutdown_connections 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 1 -- 172.21.15.148:0/2013862060 >> 172.21.15.148:0/2013862060 conn(0x7fddf40a97b0 msgr2=0x7fddf40a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 1 -- 172.21.15.148:0/2013862060 shutdown_connections 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 1 -- 172.21.15.148:0/2013862060 wait complete. 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 10 monclient: build_initial_monmap 2024-09-16T04:14:43.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.710+0000 7fddfb21e640 10 monclient: monmap: 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 1 Processor -- start 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 1 -- start start 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient: init 2024-09-16T04:14:43.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient(hunting): picked mon.noname-a con 0x7fddf40a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient(hunting): picked mon.noname-c con 0x7fddf40a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient(hunting): picked mon.noname-b con 0x7fddf413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf409e890 con 0x7fddf40a1b10 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf41cc910 con 0x7fddf413d060 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf40a0130 con 0x7fddf40a2410 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddfb21e640 10 monclient(hunting): authenticate will time out at 1304.336856s 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddf8fba640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fddf40a1b10 0x7fddf41cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49682/0 (socket says 172.21.15.148:49682) 2024-09-16T04:14:43.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.711+0000 7fddf8fba640 1 -- 172.21.15.148:0/791505578 learned_addr learned my addr 172.21.15.148:0/791505578 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3538735326 0 0) 0x7fddf40a0130 con 0x7fddf40a2410 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): my global_id is 84140 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddd8003520 con 0x7fddf40a2410 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1211837603 0 0) 0x7fddf409e890 con 0x7fddf40a1b10 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): my global_id is 65049 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf40a0130 con 0x7fddf40a1b10 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743549457 0 0) 0x7fddf41cc910 con 0x7fddf413d060 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): my global_id is 6610 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf409e890 con 0x7fddf413d060 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3753079921 0 0) 0x7fddd8003520 con 0x7fddf40a2410 2024-09-16T04:14:43.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddf41cc910 con 0x7fddf40a2410 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdde0001cc0 con 0x7fddf40a2410 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): dump: 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.732 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3838314437 0 0) 0x7fddf40a0130 con 0x7fddf40a1b10 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddd8003520 con 0x7fddf40a1b10 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdde8001870 con 0x7fddf40a1b10 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): dump: 2024-09-16T04:14:43.733 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2714102168 0 0) 0x7fddf409e890 con 0x7fddf413d060 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fddf40a0130 con 0x7fddf413d060 2024-09-16T04:14:43.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fddec0015d0 con 0x7fddf413d060 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient(hunting): dump: 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.735 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2352990901 0 0) 0x7fddf41cc910 con 0x7fddf40a2410 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 >> v1:172.21.15.171:6789/0 conn(0x7fddf413d060 legacy=0x7fddf41d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 >> v1:172.21.15.148:6789/0 conn(0x7fddf40a1b10 legacy=0x7fddf41cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 monclient: found mon.c 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf4137aa0 con 0x7fddf40a2410 2024-09-16T04:14:43.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: _finish_auth 0 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 5 monclient: authenticate success, global_id 84140 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 10 monclient: _renew_subs 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fddf4137d80 con 0x7fddf40a2410 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 10 monclient: _renew_subs 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fddf409ddc0 con 0x7fddf40a2410 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdde00018b0 con 0x7fddf40a2410 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdde0004d20 con 0x7fddf40a2410 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:43.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf2ffd640 4 set_mon_vals no callback set 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.712+0000 7fddf17fa640 10 monclient: dump: 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.713+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdde0004d20 con 0x7fddf40a2410 2024-09-16T04:14:43.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.713+0000 7fddfb21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.713+0000 7fddfb21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.713+0000 7fddfb21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.713+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd4005080 con 0x7fddf40a2410 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.713+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fdde0047830 con 0x7fddf40a2410 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.714+0000 7fddf8fba640 10 monclient: get_auth_request con 0x7fddd80391b0 auth_method 0 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.718+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdde0015b50 con 0x7fddf40a2410 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.718+0000 7fddf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:43.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.718+0000 7fddf17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:43.942 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=0 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=0 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T04:14:43.943 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T04:14:43.944 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T04:14:43.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.952+0000 7fddfb21e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:14:43.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.952+0000 7fddfb21e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:14:43.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.952+0000 7fddfb21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:43.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.952+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fddd4005b80 con 0x7fddf40a2410 2024-09-16T04:14:43.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.953+0000 7fddf17fa640 1 -- 172.21.15.148:0/791505578 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fdde001dfa0 con 0x7fddf40a2410 2024-09-16T04:14:43.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.953+0000 7fddf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:14:43.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.953+0000 7fddf17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.953+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.953+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=1758450947,config=692619927,health=3495635461,kv=2894794911,logm=1897056294}) 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.953+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=10,logm=56} crc {auth=3878564478,config=692619927,health=3744606254,kv=2270513034,logm=2479806446}) 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.953+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.953+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=1758450947,config=692619927,health=3495635461,kv=2894794911,logm=1897056294}) 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.953+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=3014338432,config=692619927,health=3495635461,kv=2270513035,logm=2070338511}) 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.954+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.956 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.954+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499748760}) 2024-09-16T04:14:43.957 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.954+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=493981136}) 2024-09-16T04:14:43.957 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.954+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.957 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.954+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499748760}) 2024-09-16T04:14:43.957 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.954+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556876279}) 2024-09-16T04:14:43.957 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.955+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.958 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.955+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702508712}) 2024-09-16T04:14:43.958 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.955+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3439888701}) 2024-09-16T04:14:43.958 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.955+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.958 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.955+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702508712}) 2024-09-16T04:14:43.958 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.955+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048043500}) 2024-09-16T04:14:43.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.955+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 >> v1:172.21.15.171:6812/7327273 conn(0x7fddd80391b0 legacy=0x7fddd803b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 10 monclient: shutdown 2024-09-16T04:14:43.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:43.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 >> v1:172.21.15.148:6790/0 conn(0x7fddf40a2410 legacy=0x7fddf41cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:43.959 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.956+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.959 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.956+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176937965}) 2024-09-16T04:14:43.959 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.956+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2951611048}) 2024-09-16T04:14:43.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 shutdown_connections 2024-09-16T04:14:43.960 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.956+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 >> 172.21.15.148:0/791505578 conn(0x7fddf40a97b0 msgr2=0x7fddf413faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:43.961 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.956+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176937965}) 2024-09-16T04:14:43.961 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.956+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367436892}) 2024-09-16T04:14:43.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 shutdown_connections 2024-09-16T04:14:43.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:43.956+0000 7fddfb21e640 1 -- 172.21.15.148:0/791505578 wait complete. 2024-09-16T04:14:43.964 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.957+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.964 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.957+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327293256}) 2024-09-16T04:14:43.964 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.957+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=754569692}) 2024-09-16T04:14:43.964 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.957+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.957+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327293256}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.957+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3442332615}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186056624}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491483546}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186056624}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112603046}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224941963}) 2024-09-16T04:14:43.965 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253654243}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224941963}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.958+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269848588}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.959+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.959+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479889838}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.959+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654344978}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.959+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.959+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479889838}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.959+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899758382}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737181539}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422971744}) 2024-09-16T04:14:43.966 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737181539}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046616844}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281179035}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021240887}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2281179035}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.960+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2487409436}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.961+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.961+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448570605}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.961+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943432437}) 2024-09-16T04:14:43.967 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.961+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.961+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448570605}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.961+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3113492410}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231971952}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418852283}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231971952}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522775519}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144163732}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247443633}) 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.968 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144163732}) 2024-09-16T04:14:43.969 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.962+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732008034}) 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.963+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.963+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1721892484}) 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.963+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1141790290}) 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.963+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.963+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1721892484}) 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.963+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768879709}) 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.972 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797592774}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353336214}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797592774}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398256541}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499863870}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3333405774}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499863870}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.964+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285494398}) 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.965+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.973 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.965+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930284280}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.965+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845555323}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.965+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.965+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930284280}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.965+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960664621}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695207852}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551990384}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695207852}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2574438065}) 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.974 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693161784}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242794515}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693161784}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.966+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2141058292}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.967+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.967+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724693066}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.967+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490241562}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.967+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.967+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724693066}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.967+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621851121}) 2024-09-16T04:14:43.975 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.976 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568122144}) 2024-09-16T04:14:43.976 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3845653025}) 2024-09-16T04:14:43.976 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.976 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568122144}) 2024-09-16T04:14:43.976 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1919717221}) 2024-09-16T04:14:43.977 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:14:43.989 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.989 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443279344}) 2024-09-16T04:14:43.989 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060936387}) 2024-09-16T04:14:43.989 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.989 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443279344}) 2024-09-16T04:14:43.989 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.968+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703039050}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807930514}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772173822}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807930514}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=667814846}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.969+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349847128}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=898614697}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349847128}) 2024-09-16T04:14:43.990 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4073464451}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867704585}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1443397803}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867704585}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.970+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166206572}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.971+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.971+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165199851}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.971+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=881941669}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.971+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.971+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165199851}) 2024-09-16T04:14:43.991 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.971+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416959165}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744860875}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3689522070}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744860875}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3277081952}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273090323}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167562817}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273090323}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.972+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104391293}) 2024-09-16T04:14:43.992 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.973+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.973+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029725919}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.973+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924129765}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.973+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.973+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029725919}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.973+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2328490574}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687699126}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048957578}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687699126}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604674230}) 2024-09-16T04:14:43.993 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404809458}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3092386536}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404809458}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.974+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626845839}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537427157}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3767569496}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537427157}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2630042065}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.975+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916928838}) 2024-09-16T04:14:43.994 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430661273}) 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916928838}) 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186458078}) 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136364908}) 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3337969682}) 2024-09-16T04:14:43.995 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136364908}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.976+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=382379259}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152577213}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2784002445}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152577213}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1550854167}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.977+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649851795}) 2024-09-16T04:14:43.996 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170634574}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3649851795}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604599425}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900909517}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122751867}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2900909517}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.978+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753877351}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.979+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.979+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=4} crc {logm=3559798798,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=394128967,mon_config_key=2589496441,monmap=2633035914,osd_metadata=2495496720}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.979+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=4015981641,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3272792657,mon_config_key=554932725,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=3823785925}) 2024-09-16T04:14:43.997 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.979+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.979+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=4} crc {logm=3559798798,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=394128967,mon_config_key=2589496441,monmap=2633035914,osd_metadata=2495496720}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.979+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=55,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=2819913972,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2842877263,mon_config_key=3438290323,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.980+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.980+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=2,osdmap=95} crc {osd_metadata=1895033306,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=2154370986}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.980+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=2,osdmap=95} crc {osd_metadata=1895033306,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=432711333}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.980+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.980+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=2,osdmap=95} crc {osd_metadata=1895033306,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=2154370986}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.980+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=1,osdmap=96} crc {osd_metadata=1895033306,osd_pg_creating=3462100867,osd_snap=3823785925,osdmap=2902010884}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.981+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.981+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2525131773}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.981+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=405021798}) 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.981+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.998 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.981+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2525131773}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.981+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=689742144}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.982+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.982+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2191298351}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.982+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3123230746}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.982+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.982+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2191298351}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.982+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1556991958}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.984+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.984+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=68103454}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.984+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=90} crc {osdmap=557221699}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.984+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.984+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=68103454}) 2024-09-16T04:14:43.999 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:14:43.984+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=88} crc {osdmap=3349876108}) 2024-09-16T04:14:44.433 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=0 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=0 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T04:14:44.434 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T04:14:44.435 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-16T04:14:44.921 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=0 2024-09-16T04:14:44.921 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T04:14:44.922 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:44.922 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-16T04:14:44.922 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-16T04:14:44.922 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:14:44.922 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T04:14:44.922 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T04:14:44.923 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-16T04:14:44.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:14:44.978 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.080+0000 7f551915f640 10 monclient: get_monmap_and_config 2024-09-16T04:14:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.080+0000 7f551915f640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.080+0000 7f551915f640 10 monclient: monmap: 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:45.081847+0000 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:45.081847+0000 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:45.083 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.081+0000 7f551915f640 1 Processor -- start 2024-09-16T04:14:45.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.081+0000 7f551915f640 1 -- start start 2024-09-16T04:14:45.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.081+0000 7f551915f640 10 monclient: init 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.081+0000 7f551915f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.081+0000 7f551915f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 10 monclient(hunting): picked mon.noname-a con 0x7f551409bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 10 monclient(hunting): picked mon.noname-b con 0x7f551409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 10 monclient(hunting): picked mon.noname-c con 0x7f551409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55140a1010 con 0x7f551409bb70 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551409f030 con 0x7f551409cc30 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551409dc60 con 0x7f551409c660 2024-09-16T04:14:45.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 10 monclient(hunting): authenticate will time out at 1305.707077s 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f5512d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f551409bb70 0x7f551409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49702/0 (socket says 172.21.15.148:49702) 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f5512d9d640 1 -- 172.21.15.148:0/35930919 learned_addr learned my addr 172.21.15.148:0/35930919 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2540118294 0 0) 0x7f551409dc60 con 0x7f551409c660 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): my global_id is 84161 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5500003520 con 0x7f551409c660 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089371193 0 0) 0x7f55140a1010 con 0x7f551409bb70 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): my global_id is 65070 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f551409dc60 con 0x7f551409bb70 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909807774 0 0) 0x7f551409f030 con 0x7f551409cc30 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient(hunting): my global_id is 6631 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55140a1010 con 0x7f551409cc30 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1983121772 0 0) 0x7f5500003520 con 0x7f551409c660 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 >> v1:172.21.15.171:6789/0 conn(0x7f551409cc30 legacy=0x7f551413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 >> v1:172.21.15.148:6789/0 conn(0x7f551409bb70 legacy=0x7f551409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 monclient: found mon.noname-b 2024-09-16T04:14:45.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551409df40 con 0x7f551409c660 2024-09-16T04:14:45.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f54fc001610 con 0x7f551409c660 2024-09-16T04:14:45.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 5 monclient: authenticate success, global_id 84161 2024-09-16T04:14:45.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: dump: 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.100 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f54fc0018c0 con 0x7f551409c660 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 1 -- 172.21.15.148:0/35930919 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f54fc001cd0 con 0x7f551409c660 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:45.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551915f640 4 set_mon_vals no callback set 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.082+0000 7f551259c640 10 monclient: dump: 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 10 monclient: shutdown 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 1 -- 172.21.15.148:0/35930919 >> v1:172.21.15.148:6790/0 conn(0x7f551409c660 legacy=0x7f5514138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 1 -- 172.21.15.148:0/35930919 shutdown_connections 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 1 -- 172.21.15.148:0/35930919 >> 172.21.15.148:0/35930919 conn(0x7f55140a5640 msgr2=0x7f55140a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 1 -- 172.21.15.148:0/35930919 shutdown_connections 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.083+0000 7f551915f640 1 -- 172.21.15.148:0/35930919 wait complete. 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.084+0000 7f551915f640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.084+0000 7f551915f640 10 monclient: monmap: 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:45.103 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.084+0000 7f551915f640 1 Processor -- start 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.084+0000 7f551915f640 1 -- start start 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.084+0000 7f551915f640 10 monclient: init 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient(hunting): picked mon.noname-b con 0x7f551409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient(hunting): picked mon.noname-c con 0x7f551409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient(hunting): picked mon.noname-a con 0x7f551409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551413bea0 con 0x7f551409cc30 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55140a3d70 con 0x7f551409bb70 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55140740a0 con 0x7f551409c660 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient(hunting): authenticate will time out at 1305.710149s 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550a59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f551409c660 0x7f55141ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57562/0 (socket says 172.21.15.148:57562) 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550a59c640 1 -- 172.21.15.148:0/4020418640 learned_addr learned my addr 172.21.15.148:0/4020418640 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2815865315 0 0) 0x7f55140740a0 con 0x7f551409c660 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): my global_id is 84164 2024-09-16T04:14:45.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54f4003520 con 0x7f551409c660 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 6129305 0 0) 0x7f551413bea0 con 0x7f551409cc30 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): my global_id is 65073 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55140740a0 con 0x7f551409cc30 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301067285 0 0) 0x7f55140a3d70 con 0x7f551409bb70 2024-09-16T04:14:45.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): my global_id is 6634 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f551413bea0 con 0x7f551409bb70 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2431837869 0 0) 0x7f54f4003520 con 0x7f551409c660 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55140a3d70 con 0x7f551409c660 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f54fc001c10 con 0x7f551409c660 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): dump: 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.107 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2758293718 0 0) 0x7f55140740a0 con 0x7f551409cc30 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54f4003520 con 0x7f551409cc30 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3725593721 0 0) 0x7f551413bea0 con 0x7f551409bb70 2024-09-16T04:14:45.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55140740a0 con 0x7f551409bb70 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f550c001bb0 con 0x7f551409cc30 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): dump: 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.109 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f55040010b0 con 0x7f551409bb70 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient(hunting): dump: 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.110 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3057673406 0 0) 0x7f55140a3d70 con 0x7f551409c660 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 >> v1:172.21.15.171:6789/0 conn(0x7f551409bb70 legacy=0x7f55141c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 >> v1:172.21.15.148:6789/0 conn(0x7f551409cc30 legacy=0x7f55141d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 monclient: found mon.c 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551411ca10 con 0x7f551409c660 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f550bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 5 monclient: authenticate success, global_id 84164 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient: _renew_subs 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.085+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f551411c170 con 0x7f551409c660 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 10 monclient: _renew_subs 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f551411b4a0 con 0x7f551409c660 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f54fc0018c0 con 0x7f551409c660 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f54fc003bd0 con 0x7f551409c660 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 10 monclient: dump: 2024-09-16T04:14:45.112 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f5511d9b640 4 set_mon_vals no callback set 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f54fc003bd0 con 0x7f551409c660 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7f54fc047a10 con 0x7f551409c660 2024-09-16T04:14:45.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.086+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f8005080 con 0x7f551409c660 2024-09-16T04:14:45.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.087+0000 7f5512d9d640 10 monclient: get_auth_request con 0x7f54f4039070 auth_method 0 2024-09-16T04:14:45.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.091+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f54fc015c70 con 0x7f551409c660 2024-09-16T04:14:45.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.091+0000 7f550bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.091+0000 7f550bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:45.239 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:14:45.239 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:14:45.239 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:45.239 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-16T04:14:45.239 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-16T04:14:45.240 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:14:45.240 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T04:14:45.240 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T04:14:45.240 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f551915f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f551915f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f551915f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f54f8005370 con 0x7f551409c660 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f550bfff640 1 -- 172.21.15.148:0/4020418640 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3667569546) 0x7f54fc01e120 con 0x7f551409c660 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f550bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.320+0000 7f550bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:45.322 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 >> v1:172.21.15.171:6812/7327273 conn(0x7f54f4039070 legacy=0x7f54f403b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 10 monclient: shutdown 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 >> v1:172.21.15.148:6790/0 conn(0x7f551409c660 legacy=0x7f55141ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 shutdown_connections 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 >> 172.21.15.148:0/4020418640 conn(0x7f55140a5640 msgr2=0x7f551413ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 shutdown_connections 2024-09-16T04:14:45.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.322+0000 7f551915f640 1 -- 172.21.15.148:0/4020418640 wait complete. 2024-09-16T04:14:45.334 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:14:45.334 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:14:45.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:14:45.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:14:45.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.431+0000 7f42eceba640 10 monclient: get_monmap_and_config 2024-09-16T04:14:45.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.431+0000 7f42eceba640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.431+0000 7f42eceba640 10 monclient: monmap: 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:45.432500+0000 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:45.432500+0000 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 1 Processor -- start 2024-09-16T04:14:45.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 1 -- start start 2024-09-16T04:14:45.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient: init 2024-09-16T04:14:45.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:45.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient(hunting): picked mon.noname-a con 0x7f42e80a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient(hunting): picked mon.noname-b con 0x7f42e809b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient(hunting): picked mon.noname-c con 0x7f42e809b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e809e040 con 0x7f42e80a0e90 2024-09-16T04:14:45.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e809d660 con 0x7f42e809b5c0 2024-09-16T04:14:45.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e809c0d0 con 0x7f42e809b170 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42eceba640 10 monclient(hunting): authenticate will time out at 1306.057841s 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42e659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f42e80a0e90 0x7f42e809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49718/0 (socket says 172.21.15.148:49718) 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42e5d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f42e809b170 0x7f42e81383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57570/0 (socket says 172.21.15.148:57570) 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.432+0000 7f42e659c640 1 -- 172.21.15.148:0/1693960575 learned_addr learned my addr 172.21.15.148:0/1693960575 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3623055879 0 0) 0x7f42d8001110 con 0x7f42e80a0e90 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): my global_id is 65082 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42d0004630 con 0x7f42e80a0e90 2024-09-16T04:14:45.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236617947 0 0) 0x7f42e809c0d0 con 0x7f42e809b170 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): my global_id is 84173 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42d8001110 con 0x7f42e809b170 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848071897 0 0) 0x7f42e0001490 con 0x7f42e809b5c0 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient(hunting): my global_id is 6643 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e809c0d0 con 0x7f42e809b5c0 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1438142869 0 0) 0x7f42d0004630 con 0x7f42e80a0e90 2024-09-16T04:14:45.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 >> v1:172.21.15.148:6790/0 conn(0x7f42e809b170 legacy=0x7f42e81383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 >> v1:172.21.15.171:6789/0 conn(0x7f42e809b5c0 legacy=0x7f42e813bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 monclient: found mon.noname-a 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e809d380 con 0x7f42e80a0e90 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42d80017e0 con 0x7f42e80a0e90 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42eceba640 5 monclient: authenticate success, global_id 65082 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42eceba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: dump: 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42eceba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42d8001a90 con 0x7f42e80a0e90 2024-09-16T04:14:45.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 1 -- 172.21.15.148:0/1693960575 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42d80048a0 con 0x7f42e80a0e90 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42eceba640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42eceba640 4 set_mon_vals no callback set 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.433+0000 7f42e559a640 10 monclient: dump: 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.449 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 10 monclient: shutdown 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 1 -- 172.21.15.148:0/1693960575 >> v1:172.21.15.148:6789/0 conn(0x7f42e80a0e90 legacy=0x7f42e809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 1 -- 172.21.15.148:0/1693960575 shutdown_connections 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 1 -- 172.21.15.148:0/1693960575 >> 172.21.15.148:0/1693960575 conn(0x7f42e80a5640 msgr2=0x7f42e80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 1 -- 172.21.15.148:0/1693960575 shutdown_connections 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 1 -- 172.21.15.148:0/1693960575 wait complete. 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.434+0000 7f42eceba640 10 monclient: monmap: 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.450 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 1 Processor -- start 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 1 -- start start 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient: init 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient(hunting): picked mon.noname-a con 0x7f42e809b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient(hunting): picked mon.noname-c con 0x7f42e809b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient(hunting): picked mon.noname-b con 0x7f42e80a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e81c1e20 con 0x7f42e809b170 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e81a65b0 con 0x7f42e80a0e90 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e81a9070 con 0x7f42e809b5c0 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42e659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f42e809b170 0x7f42e81c1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49732/0 (socket says 172.21.15.148:49732) 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42e659c640 1 -- 172.21.15.148:0/1603955979 learned_addr learned my addr 172.21.15.148:0/1603955979 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.435+0000 7f42eceba640 10 monclient(hunting): authenticate will time out at 1306.060917s 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 739126370 0 0) 0x7f42e81a9070 con 0x7f42e809b5c0 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): my global_id is 84176 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42c8003520 con 0x7f42e809b5c0 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747618516 0 0) 0x7f42e81c1e20 con 0x7f42e809b170 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): my global_id is 65085 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e81a9070 con 0x7f42e809b170 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2738277189 0 0) 0x7f42e81a65b0 con 0x7f42e80a0e90 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): my global_id is 6646 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e81c1e20 con 0x7f42e80a0e90 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3969077603 0 0) 0x7f42c8003520 con 0x7f42e809b5c0 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42e81a65b0 con 0x7f42e809b5c0 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42dc001290 con 0x7f42e809b5c0 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.454 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2722079692 0 0) 0x7f42e81a9070 con 0x7f42e809b170 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42c8003520 con 0x7f42e809b170 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3860920226 0 0) 0x7f42e81c1e20 con 0x7f42e80a0e90 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42e81a9070 con 0x7f42e80a0e90 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42d8001600 con 0x7f42e809b170 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.455 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42e0001d80 con 0x7f42e80a0e90 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.456 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 467844467 0 0) 0x7f42e81a65b0 con 0x7f42e809b5c0 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.171:6789/0 conn(0x7f42e80a0e90 legacy=0x7f42e81d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.148:6789/0 conn(0x7f42e809b170 legacy=0x7f42e81c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 monclient: found mon.c 2024-09-16T04:14:45.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e81a6da0 con 0x7f42e809b5c0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 5 monclient: authenticate success, global_id 84176 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 10 monclient: _renew_subs 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42e81a8d90 con 0x7f42e809b5c0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 10 monclient: _renew_subs 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f42e81c4180 con 0x7f42e809b5c0 2024-09-16T04:14:45.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42dc0047e0 con 0x7f42e809b5c0 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42dc004b30 con 0x7f42e809b5c0 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42e4d99640 4 set_mon_vals no callback set 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.436+0000 7f42d6ffd640 10 monclient: dump: 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.459 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f42dc00fdc0 con 0x7f42e809b5c0 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42eceba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7f42dc047810 con 0x7f42e809b5c0 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f42cc0065c0 con 0x7f42cc002860 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42e659c640 10 monclient: get_auth_request con 0x7f42c803d5d0 auth_method 0 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4170688932 0 0) 0x7f42cc0065c0 con 0x7f42cc002860 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 10 monclient(hunting): my global_id is 65088 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.437+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e81a65b0 con 0x7f42cc002860 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1925360021 0 0) 0x7f42e81a65b0 con 0x7f42cc002860 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42cc0065c0 con 0x7f42cc002860 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42e0003970 con 0x7f42cc002860 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 10 monclient: dump: 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.461 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 902478755 0 0) 0x7f42cc0065c0 con 0x7f42cc002860 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42cc00a150 con 0x7f42cc002860 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f42e0004da0 con 0x7f42cc002860 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.148:6789/0 conn(0x7f42cc002860 legacy=0x7f42cc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.438+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.148:6789/0 conn(0x7f42cc002860 legacy=0x7f42cc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42eceba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f42cc009e20 con 0x7f42cc0090e0 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063266217 0 0) 0x7f42cc009e20 con 0x7f42cc0090e0 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42d6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42d6ffd640 10 monclient(hunting): my global_id is 65091 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.444+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42cc0065c0 con 0x7f42cc0090e0 2024-09-16T04:14:45.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 539383327 0 0) 0x7f42cc0065c0 con 0x7f42cc0090e0 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42cc009e20 con 0x7f42cc0090e0 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f42e0005400 con 0x7f42cc0090e0 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 10 monclient: dump: 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.465 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2014703853 0 0) 0x7f42cc009e20 con 0x7f42cc0090e0 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42cc00c390 con 0x7f42cc0090e0 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4146 (unknown 0 0 3269335142) 0x7f42e00052d0 con 0x7f42cc0090e0 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.148:6789/0 conn(0x7f42cc0090e0 legacy=0x7f42cc009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42d6ffd640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.148:6789/0 conn(0x7f42cc0090e0 legacy=0x7f42cc009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.466 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 38, 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 264643, 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:45.467 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.468 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.469 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:14:45.470 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:14:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:14:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:14:45.471 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:14:45.472 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.473 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.474 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.475 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:14:45.476 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.477 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.478 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.479 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:45.479 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.479 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.479 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.479 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:14:45.479 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:14:45.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.171:6812/7327273 conn(0x7f42c803d5d0 legacy=0x7f42c803fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 10 monclient: shutdown 2024-09-16T04:14:45.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 >> v1:172.21.15.148:6790/0 conn(0x7f42e809b5c0 legacy=0x7f42e81cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.480 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42e6d9d640 1 -- 172.21.15.148:0/1603955979 reap_dead start 2024-09-16T04:14:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42e6d9d640 1 -- 172.21.15.148:0/1603955979 reap_dead start 2024-09-16T04:14:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 shutdown_connections 2024-09-16T04:14:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 >> 172.21.15.148:0/1603955979 conn(0x7f42e80a5640 msgr2=0x7f42e80712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 shutdown_connections 2024-09-16T04:14:45.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.445+0000 7f42eceba640 1 -- 172.21.15.148:0/1603955979 wait complete. 2024-09-16T04:14:45.481 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:45.481 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:14:45.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T04:14:45.561 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T04:14:45.562 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-16T04:14:45.579 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.578+0000 7fad08874640 10 monclient: get_monmap_and_config 2024-09-16T04:14:45.579 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.578+0000 7fad08874640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.578+0000 7fad08874640 10 monclient: monmap: 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:45.579072+0000 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:45.579072+0000 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:45.580 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.578+0000 7fad08874640 1 Processor -- start 2024-09-16T04:14:45.589 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.578+0000 7fad08874640 1 -- start start 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.578+0000 7fad08874640 10 monclient: init 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient(hunting): picked mon.noname-c con 0x7fad040a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient(hunting): picked mon.noname-a con 0x7fad040a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient(hunting): picked mon.noname-b con 0x7fad0413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad040a0e00 con 0x7fad040a1b10 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad040a10e0 con 0x7fad040a2410 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad040a4970 con 0x7fad0413d060 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad08874640 10 monclient(hunting): authenticate will time out at 1306.204584s 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad01d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fad040a1b10 0x7fad0413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49762/0 (socket says 172.21.15.148:49762) 2024-09-16T04:14:45.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad01d9b640 1 -- 172.21.15.148:0/2100680654 learned_addr learned my addr 172.21.15.148:0/2100680654 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788691474 0 0) 0x7fad040a4970 con 0x7fad0413d060 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): my global_id is 84179 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facf0003520 con 0x7fad0413d060 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1356425068 0 0) 0x7fad040a0e00 con 0x7fad040a1b10 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 10 monclient(hunting): my global_id is 65094 2024-09-16T04:14:45.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.579+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad040a4970 con 0x7fad040a1b10 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871121749 0 0) 0x7fad040a10e0 con 0x7fad040a2410 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient(hunting): my global_id is 6649 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad040a0e00 con 0x7fad040a2410 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3195516940 0 0) 0x7facf0003520 con 0x7fad0413d060 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 >> v1:172.21.15.171:6789/0 conn(0x7fad040a2410 legacy=0x7fad040a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 >> v1:172.21.15.148:6789/0 conn(0x7fad040a1b10 legacy=0x7fad0413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 monclient: found mon.noname-b 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0406dfd0 con 0x7fad0413d060 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 5 monclient: authenticate success, global_id 84179 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facfc001540 con 0x7fad0413d060 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: dump: 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.593 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7facfc0017e0 con 0x7fad0413d060 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 1 -- 172.21.15.148:0/2100680654 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facfc001bf0 con 0x7fad0413d060 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 4 set_mon_vals no callback set 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad0159a640 10 monclient: dump: 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 10 monclient: shutdown 2024-09-16T04:14:45.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 1 -- 172.21.15.148:0/2100680654 >> v1:172.21.15.148:6790/0 conn(0x7fad0413d060 legacy=0x7fad0413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 1 -- 172.21.15.148:0/2100680654 shutdown_connections 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 1 -- 172.21.15.148:0/2100680654 >> 172.21.15.148:0/2100680654 conn(0x7fad040a97b0 msgr2=0x7fad040a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 1 -- 172.21.15.148:0/2100680654 shutdown_connections 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.580+0000 7fad08874640 1 -- 172.21.15.148:0/2100680654 wait complete. 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.581+0000 7fad08874640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.581+0000 7fad08874640 10 monclient: monmap: 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.596 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.581+0000 7fad08874640 1 Processor -- start 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 1 -- start start 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient: init 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient(hunting): picked mon.noname-b con 0x7fad040a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient(hunting): picked mon.noname-a con 0x7fad040a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient(hunting): picked mon.noname-c con 0x7fad0413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0409e880 con 0x7fad040a2410 2024-09-16T04:14:45.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad041cc910 con 0x7fad040a1b10 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0413cc00 con 0x7fad0413d060 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad08874640 10 monclient(hunting): authenticate will time out at 1306.207526s 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad01d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fad040a2410 0x7fad041cc200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49764/0 (socket says 172.21.15.148:49764) 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7fad01d9b640 1 -- 172.21.15.148:0/177993270 learned_addr learned my addr 172.21.15.148:0/177993270 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934146231 0 0) 0x7fad0409e880 con 0x7fad040a2410 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): my global_id is 65097 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7face4003550 con 0x7fad040a2410 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972519580 0 0) 0x7fad0413cc00 con 0x7fad0413d060 2024-09-16T04:14:45.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): my global_id is 84182 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad0409e880 con 0x7fad0413d060 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032195788 0 0) 0x7fad041cc910 con 0x7fad040a1b10 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 10 monclient(hunting): my global_id is 6652 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad0413cc00 con 0x7fad040a1b10 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.582+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3892666602 0 0) 0x7fad0409e880 con 0x7fad0413d060 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad041cc910 con 0x7fad0413d060 2024-09-16T04:14:45.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facfc001b30 con 0x7fad0413d060 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.600 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3925727996 0 0) 0x7face4003550 con 0x7fad040a2410 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fad0409e880 con 0x7fad040a2410 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2058355925 0 0) 0x7fad0413cc00 con 0x7fad040a1b10 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7face4003550 con 0x7fad040a1b10 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facec0012b0 con 0x7fad040a2410 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facf4001bb0 con 0x7fad040a1b10 2024-09-16T04:14:45.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3384579037 0 0) 0x7fad041cc910 con 0x7fad0413d060 2024-09-16T04:14:45.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.171:6789/0 conn(0x7fad040a1b10 legacy=0x7fad041cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.148:6789/0 conn(0x7fad040a2410 legacy=0x7fad041cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 monclient: found mon.c 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad040a0130 con 0x7fad0413d060 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 5 monclient: authenticate success, global_id 84182 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 10 monclient: _renew_subs 2024-09-16T04:14:45.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fad04138260 con 0x7fad0413d060 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 10 monclient: _renew_subs 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fad0409f600 con 0x7fad0413d060 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7facfc003960 con 0x7fad0413d060 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facfc003d30 con 0x7fad0413d060 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7fad00d99640 4 set_mon_vals no callback set 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.583+0000 7facfaffd640 10 monclient: dump: 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7facfc00f260 con 0x7fad0413d060 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7fad08874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7face80065c0 con 0x7face8002860 2024-09-16T04:14:45.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7facfc0466d0 con 0x7fad0413d060 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886750953 0 0) 0x7face80065c0 con 0x7face8002860 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient(hunting): my global_id is 84185 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad041cc910 con 0x7face8002860 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7fad01d9b640 10 monclient: get_auth_request con 0x7face40391c0 auth_method 0 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2351380218 0 0) 0x7fad041cc910 con 0x7face8002860 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7face80065c0 con 0x7face8002860 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facf4003b10 con 0x7face8002860 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 10 monclient: dump: 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 315233798 0 0) 0x7face80065c0 con 0x7face8002860 2024-09-16T04:14:45.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.584+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7face800a150 con 0x7face8002860 2024-09-16T04:14:45.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.585+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7facf4003b30 con 0x7face8002860 2024-09-16T04:14:45.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.585+0000 7facfaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.585+0000 7facfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:45.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.585+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.148:6790/0 conn(0x7face8002860 legacy=0x7face8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.585+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.148:6790/0 conn(0x7face8002860 legacy=0x7face8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7fad08874640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7face8009e40 con 0x7face8009100 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854659177 0 0) 0x7face8009e40 con 0x7face8009100 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient(hunting): my global_id is 84188 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7face80065c0 con 0x7face8009100 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4148731806 0 0) 0x7face80065c0 con 0x7face8009100 2024-09-16T04:14:45.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7face8009e40 con 0x7face8009100 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7facf40040b0 con 0x7face8009100 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient: dump: 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.612 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 953391613 0 0) 0x7face8009e40 con 0x7face8009100 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7face800c3d0 con 0x7face8009100 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 1766721529) 0x7facf40042e0 con 0x7face8009100 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.148:6790/0 conn(0x7face8009100 legacy=0x7face8009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.591+0000 7facfaffd640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.148:6790/0 conn(0x7face8009100 legacy=0x7face8009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:14:45.613 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 38, 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 22626, 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:14:45.614 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:14:45.615 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.616 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:14:45.617 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.618 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:14:45.619 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:45.620 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.621 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.622 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.623 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:14:45.624 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.171:6812/7327273 conn(0x7face40391c0 legacy=0x7face403b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 10 monclient: shutdown 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 >> v1:172.21.15.148:6790/0 conn(0x7fad0413d060 legacy=0x7fad041d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad02d9d640 1 -- 172.21.15.148:0/177993270 reap_dead start 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad02d9d640 1 -- 172.21.15.148:0/177993270 reap_dead start 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 shutdown_connections 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 >> 172.21.15.148:0/177993270 conn(0x7fad040a97b0 msgr2=0x7fad040a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 shutdown_connections 2024-09-16T04:14:45.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.592+0000 7fad08874640 1 -- 172.21.15.148:0/177993270 wait complete. 2024-09-16T04:14:45.627 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:45.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:14:45.627 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:14:45.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.725+0000 7fc9eadfd640 10 monclient: get_monmap_and_config 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.725+0000 7fc9eadfd640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.725+0000 7fc9eadfd640 10 monclient: monmap: 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:45.726101+0000 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:45.726101+0000 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:45.727 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:45.728 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.725+0000 7fc9eadfd640 1 Processor -- start 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 1 -- start start 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient: init 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e40a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e40a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40a0e00 con 0x7fc9e40a1b10 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40a10e0 con 0x7fc9e40a2410 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e40a4970 con 0x7fc9e413d060 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9eadfd640 10 monclient(hunting): authenticate will time out at 1306.351575s 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e3fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc9e40a1b10 0x7fc9e413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49780/0 (socket says 172.21.15.148:49780) 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e3fff640 1 -- 172.21.15.148:0/3381207313 learned_addr learned my addr 172.21.15.148:0/3381207313 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e939a640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc9e413d060 0x7fc9e413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57628/0 (socket says 172.21.15.148:57628) 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473402580 0 0) 0x7fc9e40a4970 con 0x7fc9e413d060 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): my global_id is 84197 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9d4003520 con 0x7fc9e413d060 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484596063 0 0) 0x7fc9e40a0e00 con 0x7fc9e40a1b10 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 10 monclient(hunting): my global_id is 65106 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.726+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e40a4970 con 0x7fc9e40a1b10 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507894497 0 0) 0x7fc9e40a10e0 con 0x7fc9e40a2410 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient(hunting): my global_id is 6661 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e40a0e00 con 0x7fc9e40a2410 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2232866086 0 0) 0x7fc9e40a4970 con 0x7fc9e40a1b10 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 >> v1:172.21.15.148:6790/0 conn(0x7fc9e413d060 legacy=0x7fc9e413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 >> v1:172.21.15.171:6789/0 conn(0x7fc9e40a2410 legacy=0x7fc9e40a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e406dfd0 con 0x7fc9e40a1b10 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d0001520 con 0x7fc9e40a1b10 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 5 monclient: authenticate success, global_id 65106 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: dump: 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.742 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9d0001880 con 0x7fc9e40a1b10 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 1 -- 172.21.15.148:0/3381207313 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d0001c90 con 0x7fc9e40a1b10 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 4 set_mon_vals no callback set 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9e37fe640 10 monclient: dump: 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.743 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 10 monclient: shutdown 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 1 -- 172.21.15.148:0/3381207313 >> v1:172.21.15.148:6789/0 conn(0x7fc9e40a1b10 legacy=0x7fc9e413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 1 -- 172.21.15.148:0/3381207313 shutdown_connections 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 1 -- 172.21.15.148:0/3381207313 >> 172.21.15.148:0/3381207313 conn(0x7fc9e40a97b0 msgr2=0x7fc9e40a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 1 -- 172.21.15.148:0/3381207313 shutdown_connections 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.727+0000 7fc9eadfd640 1 -- 172.21.15.148:0/3381207313 wait complete. 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.728+0000 7fc9eadfd640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.728+0000 7fc9eadfd640 10 monclient: monmap: 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:45.745 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.728+0000 7fc9eadfd640 1 Processor -- start 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.728+0000 7fc9eadfd640 1 -- start start 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.728+0000 7fc9eadfd640 10 monclient: init 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e40a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient(hunting): picked mon.noname-c con 0x7fc9e40a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient(hunting): picked mon.noname-a con 0x7fc9e413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e409d6f0 con 0x7fc9e413d060 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e413b660 con 0x7fc9e40a1b10 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e409f0d0 con 0x7fc9e40a2410 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9eadfd640 10 monclient(hunting): authenticate will time out at 1306.354563s 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e3fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc9e40a2410 0x7fc9e409cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:57636/0 (socket says 172.21.15.148:57636) 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e3fff640 1 -- 172.21.15.148:0/2416233220 learned_addr learned my addr 172.21.15.148:0/2416233220 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399162483 0 0) 0x7fc9e413b660 con 0x7fc9e40a1b10 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): my global_id is 6664 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c8003520 con 0x7fc9e40a1b10 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944792132 0 0) 0x7fc9e409f0d0 con 0x7fc9e40a2410 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): my global_id is 84200 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e413b660 con 0x7fc9e40a2410 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 80914226 0 0) 0x7fc9e409d6f0 con 0x7fc9e413d060 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 10 monclient(hunting): my global_id is 65109 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.729+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e409f0d0 con 0x7fc9e413d060 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3555155558 0 0) 0x7fc9c8003520 con 0x7fc9e40a1b10 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e409d6f0 con 0x7fc9e40a1b10 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d80018b0 con 0x7fc9e40a1b10 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): dump: 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.749 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 351866139 0 0) 0x7fc9e413b660 con 0x7fc9e40a2410 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c8003520 con 0x7fc9e40a2410 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 166854085 0 0) 0x7fc9e409f0d0 con 0x7fc9e413d060 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e413b660 con 0x7fc9e413d060 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d0001bd0 con 0x7fc9e40a2410 2024-09-16T04:14:45.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): dump: 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9dc001490 con 0x7fc9e413d060 2024-09-16T04:14:45.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): dump: 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.752 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e939a640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.148:6789/0 conn(0x7fc9e413d060 legacy=0x7fc9e41d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e939a640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.148:6789/0 conn(0x7fc9e413d060 legacy=0x7fc9e41d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e939a640 1 --1- 172.21.15.148:0/2416233220 >> v1:172.21.15.148:6789/0 conn(0x7fc9e413d060 0x7fc9e41d3b40 :-1 s=OPENED pgs=434 cs=1 l=1).handle_message read tag failed 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e939a640 1 --1- 172.21.15.148:0/2416233220 >> v1:172.21.15.148:6789/0 conn(0x7fc9e413d060 0x7fc9e41d3b40 :-1 s=OPENED pgs=434 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2821473972 0 0) 0x7fc9e409d6f0 con 0x7fc9e40a1b10 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.148:6790/0 conn(0x7fc9e40a2410 legacy=0x7fc9e409cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.148:6789/0 conn(0x7fc9e413d060 legacy=0x7fc9e41d3b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 monclient: found mon.b 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e409ee50 con 0x7fc9e40a1b10 2024-09-16T04:14:45.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 5 monclient: authenticate success, global_id 6664 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 10 monclient: _renew_subs 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e413cbb0 con 0x7fc9e40a1b10 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 10 monclient: _renew_subs 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9e4137a30 con 0x7fc9e40a1b10 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9d8001c60 con 0x7fc9e40a1b10 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d8003a60 con 0x7fc9e40a1b10 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e2ffd640 4 set_mon_vals no callback set 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.730+0000 7fc9e17fa640 10 monclient: dump: 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.755 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9eadfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc9c40065c0 con 0x7fc9c4002860 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc9d800ecf0 con 0x7fc9e40a1b10 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fc9d8046750 con 0x7fc9e40a1b10 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e939a640 10 monclient: get_auth_request con 0x7fc9c8039130 auth_method 0 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3725604295 0 0) 0x7fc9c40065c0 con 0x7fc9c4002860 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient(hunting): my global_id is 6667 2024-09-16T04:14:45.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e409d6f0 con 0x7fc9c4002860 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4169888067 0 0) 0x7fc9e409d6f0 con 0x7fc9c4002860 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c40065c0 con 0x7fc9c4002860 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d00010b0 con 0x7fc9c4002860 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 10 monclient: dump: 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.757 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.731+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 645867064 0 0) 0x7fc9c40065c0 con 0x7fc9c4002860 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.732+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9c400a150 con 0x7fc9c4002860 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.732+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc9d0007010 con 0x7fc9c4002860 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.732+0000 7fc9e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.732+0000 7fc9e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.732+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.171:6789/0 conn(0x7fc9c4002860 legacy=0x7fc9c4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.732+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.171:6789/0 conn(0x7fc9c4002860 legacy=0x7fc9c4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9eadfd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc9c4009e40 con 0x7fc9c4009100 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546726649 0 0) 0x7fc9c4009e40 con 0x7fc9c4009100 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9e17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9e17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9e17fa640 10 monclient(hunting): my global_id is 6670 2024-09-16T04:14:45.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.738+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c40065c0 con 0x7fc9c4009100 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3895104094 0 0) 0x7fc9c40065c0 con 0x7fc9c4009100 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9c4009e40 con 0x7fc9c4009100 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9d0005510 con 0x7fc9c4009100 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 10 monclient: dump: 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.761 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2738960328 0 0) 0x7fc9c4009e40 con 0x7fc9c4009100 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc9c400c3d0 con 0x7fc9c4009100 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 3199422912) 0x7fc9d00068c0 con 0x7fc9c4009100 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.171:6789/0 conn(0x7fc9c4009100 legacy=0x7fc9c4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9e17fa640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.171:6789/0 conn(0x7fc9c4009100 legacy=0x7fc9c4009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:45.762 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 38, 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 509777, 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:14:45.763 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.764 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.765 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:14:45.766 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:14:45.767 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:14:45.768 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:14:45.769 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:14:45.770 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:14:45.771 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:14:45.772 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:14:45.774 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 7 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:14:45.775 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:14:45.776 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.739+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.171:6812/7327273 conn(0x7fc9c8039130 legacy=0x7fc9c803b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 10 monclient: shutdown 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 >> v1:172.21.15.171:6789/0 conn(0x7fc9e40a1b10 legacy=0x7fc9e409c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9e939a640 1 -- 172.21.15.148:0/2416233220 reap_dead start 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9e939a640 1 -- 172.21.15.148:0/2416233220 reap_dead start 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 shutdown_connections 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 >> 172.21.15.148:0/2416233220 conn(0x7fc9e40a97b0 msgr2=0x7fc9e4135920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 shutdown_connections 2024-09-16T04:14:45.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.740+0000 7fc9eadfd640 1 -- 172.21.15.148:0/2416233220 wait complete. 2024-09-16T04:14:45.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:45.779 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:14:45.779 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:14:45.779 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:14:45.779 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:14:45.779 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:14:45.780 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:45.789 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:14:45.789 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:14:45.790 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-16T04:14:45.819 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=110 2024-09-16T04:14:45.820 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T04:14:45.820 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T04:14:45.820 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-16T04:14:45.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.925+0000 7fc77ce23640 10 monclient: get_monmap_and_config 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.925+0000 7fc77ce23640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.925+0000 7fc77ce23640 10 monclient: monmap: 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:45.926122+0000 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:45.926122+0000 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:45.927 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:45.928 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.925+0000 7fc77ce23640 1 Processor -- start 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.925+0000 7fc77ce23640 1 -- start start 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.925+0000 7fc77ce23640 10 monclient: init 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient(hunting): picked mon.noname-a con 0x7fc7780a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient(hunting): picked mon.noname-c con 0x7fc7780a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient(hunting): picked mon.noname-b con 0x7fc77813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7780a0e00 con 0x7fc7780a2410 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7780a10e0 con 0x7fc7780a1b10 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7780a4970 con 0x7fc77813d060 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77ce23640 10 monclient(hunting): authenticate will time out at 1306.551578s 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc776d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc77813d060 legacy=0x7fc77813f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc7780a2410 0x7fc7780a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49796/0 (socket says 172.21.15.148:49796) 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77659c640 1 -- 172.21.15.148:0/2019547769 learned_addr learned my addr 172.21.15.148:0/2019547769 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3092282858 0 0) 0x7fc7780a0e00 con 0x7fc7780a2410 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77559a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77559a640 10 monclient(hunting): my global_id is 65118 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.926+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc760003520 con 0x7fc7780a2410 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1031585224 0 0) 0x7fc760003520 con 0x7fc7780a2410 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 >> v1:172.21.15.148:6790/0 conn(0x7fc77813d060 legacy=0x7fc77813f450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 >> v1:172.21.15.171:6789/0 conn(0x7fc7780a1b10 legacy=0x7fc77813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 monclient: found mon.noname-a 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77806dfd0 con 0x7fc7780a2410 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc76c001990 con 0x7fc7780a2410 2024-09-16T04:14:45.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 5 monclient: authenticate success, global_id 65118 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: dump: 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.937 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc76c001cf0 con 0x7fc7780a2410 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 1 -- 172.21.15.148:0/2019547769 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc76c004c20 con 0x7fc7780a2410 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 4 set_mon_vals no callback set 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77559a640 10 monclient: dump: 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 10 monclient: shutdown 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 1 -- 172.21.15.148:0/2019547769 >> v1:172.21.15.148:6789/0 conn(0x7fc7780a2410 legacy=0x7fc7780a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 1 -- 172.21.15.148:0/2019547769 shutdown_connections 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 1 -- 172.21.15.148:0/2019547769 >> 172.21.15.148:0/2019547769 conn(0x7fc7780a97b0 msgr2=0x7fc7780a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 1 -- 172.21.15.148:0/2019547769 shutdown_connections 2024-09-16T04:14:45.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.927+0000 7fc77ce23640 1 -- 172.21.15.148:0/2019547769 wait complete. 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.928+0000 7fc77ce23640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.928+0000 7fc77ce23640 10 monclient: monmap: 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.928+0000 7fc77ce23640 1 Processor -- start 2024-09-16T04:14:45.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.928+0000 7fc77ce23640 1 -- start start 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.928+0000 7fc77ce23640 10 monclient: init 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient(hunting): picked mon.noname-c con 0x7fc7780a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient(hunting): picked mon.noname-b con 0x7fc7780a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient(hunting): picked mon.noname-a con 0x7fc77813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc778137860 con 0x7fc77813d060 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc77809d870 con 0x7fc7780a2410 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7780a0130 con 0x7fc7780a1b10 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77ce23640 10 monclient(hunting): authenticate will time out at 1306.554370s 2024-09-16T04:14:45.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc77659c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc7780a1b10 legacy=0x7fc77809ca50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc776d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc77813d060 0x7fc7781d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49806/0 (socket says 172.21.15.148:49806) 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc776d9d640 1 -- 172.21.15.148:0/750385987 learned_addr learned my addr 172.21.15.148:0/750385987 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147329607 0 0) 0x7fc778137860 con 0x7fc77813d060 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): my global_id is 65121 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc758003520 con 0x7fc77813d060 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1597090072 0 0) 0x7fc758003520 con 0x7fc77813d060 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc778137860 con 0x7fc77813d060 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc770004030 con 0x7fc77813d060 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:45.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.943 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.929+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2605950375 0 0) 0x7fc778137860 con 0x7fc77813d060 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 >> v1:172.21.15.148:6790/0 conn(0x7fc7780a1b10 legacy=0x7fc77809ca50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 >> v1:172.21.15.171:6789/0 conn(0x7fc7780a2410 legacy=0x7fc77809d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 monclient: found mon.a 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc778137580 con 0x7fc77813d060 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 5 monclient: authenticate success, global_id 65121 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 10 monclient: _renew_subs 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc778136cf0 con 0x7fc77813d060 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 10 monclient: _renew_subs 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc77813c380 con 0x7fc77813d060 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc770004320 con 0x7fc77813d060 2024-09-16T04:14:45.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc7700046f0 con 0x7fc77813d060 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 10 monclient: dump: 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:45.946 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc774d99640 4 set_mon_vals no callback set 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc770004950 con 0x7fc77813d060 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.930+0000 7fc77ce23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.931+0000 7fc77ce23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.931+0000 7fc77ce23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.931+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7780008d0 con 0x7fc77813d060 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.931+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fc770047a20 con 0x7fc77813d060 2024-09-16T04:14:45.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.931+0000 7fc77659c640 10 monclient: get_auth_request con 0x7fc758038ee0 auth_method 0 2024-09-16T04:14:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.935+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc770015bc0 con 0x7fc77813d060 2024-09-16T04:14:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.935+0000 7fc766ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:45.935+0000 7fc766ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc77ce23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc77ce23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc77ce23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc77813c670 con 0x7fc77813d060 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc766ffd640 1 -- 172.21.15.148:0/750385987 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 1402359354) 0x7fc770019870 con 0x7fc77813d060 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc766ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:46.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.165+0000 7fc766ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:46.168 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:46.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 >> v1:172.21.15.171:6812/7327273 conn(0x7fc758038ee0 legacy=0x7fc75803b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 10 monclient: shutdown 2024-09-16T04:14:46.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:46.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 >> v1:172.21.15.148:6789/0 conn(0x7fc77813d060 legacy=0x7fc7781d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 shutdown_connections 2024-09-16T04:14:46.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 >> 172.21.15.148:0/750385987 conn(0x7fc7780a97b0 msgr2=0x7fc7780a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:46.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 shutdown_connections 2024-09-16T04:14:46.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:46.167+0000 7fc77ce23640 1 -- 172.21.15.148:0/750385987 wait complete. 2024-09-16T04:14:49.181 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:49.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.287+0000 7f0d008ba640 10 monclient: get_monmap_and_config 2024-09-16T04:14:49.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.287+0000 7f0d008ba640 10 monclient: build_initial_monmap 2024-09-16T04:14:49.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.287+0000 7f0d008ba640 10 monclient: monmap: 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:49.288987+0000 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:49.288987+0000 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:49.290 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.287+0000 7f0d008ba640 1 Processor -- start 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.287+0000 7f0d008ba640 1 -- start start 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.287+0000 7f0d008ba640 10 monclient: init 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient(hunting): picked mon.noname-c con 0x7f0cfc0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:49.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient(hunting): picked mon.noname-b con 0x7f0cfc0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient(hunting): picked mon.noname-a con 0x7f0cfc13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc0a0e00 con 0x7f0cfc13d060 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc0a10e0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc0a4970 con 0x7f0cfc0a1b10 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0d008ba640 10 monclient(hunting): authenticate will time out at 1309.914454s 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf9d9b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0cfc0a1b10 legacy=0x7f0cfc13b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cfad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0cfc13d060 0x7f0cfc13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49812/0 (socket says 172.21.15.148:49812) 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cfad9d640 1 -- 172.21.15.148:0/3604527922 learned_addr learned my addr 172.21.15.148:0/3604527922 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 93949602 0 0) 0x7f0cfc0a0e00 con 0x7f0cfc13d060 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf959a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:49.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf959a640 10 monclient(hunting): my global_id is 65124 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.288+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ce8003520 con 0x7f0cfc13d060 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1623314507 0 0) 0x7f0ce8003520 con 0x7f0cfc13d060 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 >> v1:172.21.15.148:6790/0 conn(0x7f0cfc0a1b10 legacy=0x7f0cfc13b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 >> v1:172.21.15.171:6789/0 conn(0x7f0cfc0a2410 legacy=0x7f0cfc0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: _finish_hunting 0 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 monclient: found mon.noname-a 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc06dfd0 con 0x7f0cfc13d060 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: _finish_auth 0 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:49.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0cec001990 con 0x7f0cfc13d060 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 5 monclient: authenticate success, global_id 65124 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: dump: 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:49.300 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0cec001cf0 con 0x7f0cfc13d060 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 1 -- 172.21.15.148:0/3604527922 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0cec004c20 con 0x7f0cfc13d060 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 4 set_mon_vals no callback set 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0cf959a640 10 monclient: dump: 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:49.301 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 10 monclient: shutdown 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 1 -- 172.21.15.148:0/3604527922 >> v1:172.21.15.148:6789/0 conn(0x7f0cfc13d060 legacy=0x7f0cfc13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 1 -- 172.21.15.148:0/3604527922 shutdown_connections 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 1 -- 172.21.15.148:0/3604527922 >> 172.21.15.148:0/3604527922 conn(0x7f0cfc0a97b0 msgr2=0x7f0cfc0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 1 -- 172.21.15.148:0/3604527922 shutdown_connections 2024-09-16T04:14:49.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.289+0000 7f0d008ba640 1 -- 172.21.15.148:0/3604527922 wait complete. 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.290+0000 7f0d008ba640 10 monclient: build_initial_monmap 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.290+0000 7f0d008ba640 10 monclient: monmap: 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.290+0000 7f0d008ba640 1 Processor -- start 2024-09-16T04:14:49.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.290+0000 7f0d008ba640 1 -- start start 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient: init 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient(hunting): picked mon.noname-b con 0x7f0cfc0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient(hunting): picked mon.noname-a con 0x7f0cfc0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient(hunting): picked mon.noname-c con 0x7f0cfc13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc137430 con 0x7f0cfc0a2410 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc09d790 con 0x7f0cfc0a1b10 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cfad9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f0cfc13d060 legacy=0x7f0cfc1d3b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc136ba0 con 0x7f0cfc13d060 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0d008ba640 10 monclient(hunting): authenticate will time out at 1309.917503s 2024-09-16T04:14:49.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf9d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0cfc0a2410 0x7f0cfc09d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:49824/0 (socket says 172.21.15.148:49824) 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf9d9b640 1 -- 172.21.15.148:0/3954338407 learned_addr learned my addr 172.21.15.148:0/3954338407 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3962179360 0 0) 0x7f0cfc137430 con 0x7f0cfc0a2410 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): my global_id is 65127 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cdc003550 con 0x7f0cfc0a2410 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4215472001 0 0) 0x7f0cdc003550 con 0x7f0cfc0a2410 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0cfc137430 con 0x7f0cfc0a2410 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0ce4004150 con 0x7f0cfc0a2410 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:49.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.291+0000 7f0cf2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4154946483 0 0) 0x7f0cfc137430 con 0x7f0cfc0a2410 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 >> v1:172.21.15.148:6790/0 conn(0x7f0cfc13d060 legacy=0x7f0cfc1d3b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 >> v1:172.21.15.171:6789/0 conn(0x7f0cfc0a1b10 legacy=0x7f0cfc09c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:49.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 monclient: found mon.a 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc09f170 con 0x7f0cfc0a2410 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 5 monclient: authenticate success, global_id 65127 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 10 monclient: _renew_subs 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0cfc09eef0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 10 monclient: _renew_subs 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:49.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0cfc137710 con 0x7f0cfc0a2410 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ce40044b0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0ce4004880 con 0x7f0cfc0a2410 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf2ffd640 10 monclient: dump: 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:49.308 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.292+0000 7f0cf8d99640 4 set_mon_vals no callback set 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0ce4004ae0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0d008ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0d008ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0d008ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cfc0008d0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7f0ce4047b60 con 0x7f0cfc0a2410 2024-09-16T04:14:49.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.293+0000 7f0cfa59c640 10 monclient: get_auth_request con 0x7f0cdc038f10 auth_method 0 2024-09-16T04:14:49.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.298+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f0ce4015dc0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.298+0000 7f0cf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:49.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.298+0000 7f0cf2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.528+0000 7f0d008ba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.528+0000 7f0d008ba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.528+0000 7f0d008ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.528+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0cfc137950 con 0x7f0cfc0a2410 2024-09-16T04:14:49.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.529+0000 7f0cf2ffd640 1 -- 172.21.15.148:0/3954338407 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 4117876904) 0x7f0ce40156c0 con 0x7f0cfc0a2410 2024-09-16T04:14:49.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.529+0000 7f0cf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:49.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.529+0000 7f0cf2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:49.533 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 >> v1:172.21.15.171:6812/7327273 conn(0x7f0cdc038f10 legacy=0x7f0cdc03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 10 monclient: shutdown 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 >> v1:172.21.15.148:6789/0 conn(0x7f0cfc0a2410 legacy=0x7f0cfc09d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 shutdown_connections 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 >> 172.21.15.148:0/3954338407 conn(0x7f0cfc0a97b0 msgr2=0x7f0cfc135860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 shutdown_connections 2024-09-16T04:14:49.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:49.531+0000 7f0d008ba640 1 -- 172.21.15.148:0/3954338407 wait complete. 2024-09-16T04:14:52.548 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:52.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.654+0000 7f77b4ea6640 10 monclient: get_monmap_and_config 2024-09-16T04:14:52.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.654+0000 7f77b4ea6640 10 monclient: build_initial_monmap 2024-09-16T04:14:52.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.654+0000 7f77b4ea6640 10 monclient: monmap: 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:14:52.656811+0000 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:14:52.656811+0000 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:14:52.658 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.655+0000 7f77b4ea6640 1 Processor -- start 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.655+0000 7f77b4ea6640 1 -- start start 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.655+0000 7f77b4ea6640 10 monclient: init 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient(hunting): picked mon.noname-a con 0x7f77b0134870 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient(hunting): picked mon.noname-c con 0x7f77b009e240 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient(hunting): picked mon.noname-b con 0x7f77b013ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b0135360 con 0x7f77b0134870 2024-09-16T04:14:52.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b0134440 con 0x7f77b009e240 2024-09-16T04:14:52.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b009e650 con 0x7f77b013ae80 2024-09-16T04:14:52.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:52.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77b4ea6640 10 monclient(hunting): authenticate will time out at 1313.282268s 2024-09-16T04:14:52.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77aed9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f77b013d270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:52.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77ae59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f77b0134870 0x7f77b0134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47536/0 (socket says 172.21.15.148:47536) 2024-09-16T04:14:52.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.656+0000 7f77ae59c640 1 -- 172.21.15.148:0/1604261639 learned_addr learned my addr 172.21.15.148:0/1604261639 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:52.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:52.857+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f77b013d270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:53.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:53.258+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f77b013d270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:54.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:54.059+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f77b013d270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:55.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): tick 2024-09-16T04:14:55.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:14:55.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:14:55.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:14:55.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f77b013d270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:55.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.171:6789/0 conn(0x7f77b009e240 legacy=0x7f77b0139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6789/0 conn(0x7f77b0134870 legacy=0x7f77b0134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f77900036c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f77900072b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f77b013ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b0073230 con 0x7f77900036c0 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b009e650 con 0x7f77900072b0 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7790010910 con 0x7f77b013ae80 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77a7fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:55.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.657+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:55.858+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:56.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:56.259+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:57.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:57.060+0000 7f77aed9d640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:58.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): tick 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.171:6789/0 conn(0x7f77900072b0 legacy=0x7f7790009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6789/0 conn(0x7f77900036c0 legacy=0x7f7790005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7790010bd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f77b013ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7790012e00 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7790011b40 con 0x7f7790012e00 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7790010910 con 0x7f7790010bd0 2024-09-16T04:14:58.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7790009f10 con 0x7f77b013ae80 2024-09-16T04:14:58.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77a7fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:58.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.657+0000 7f77add9b640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:58.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:58.858+0000 7f77add9b640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:59.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.259+0000 7f77add9b640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:59.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.918+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816118188 0 0) 0x7f7790011b40 con 0x7f7790012e00 2024-09-16T04:14:59.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.918+0000 7f77ad59a640 10 monclient(hunting): _negotiate 2024-09-16T04:14:59.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.918+0000 7f77ad59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:59.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.918+0000 7f77ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:59.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.918+0000 7f77ad59a640 10 monclient(hunting): my global_id is 65130 2024-09-16T04:14:59.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.918+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f779c003520 con 0x7f7790012e00 2024-09-16T04:14:59.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1567529557 0 0) 0x7f779c003520 con 0x7f7790012e00 2024-09-16T04:14:59.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6790/0 conn(0x7f77b013ae80 legacy=0x7f779000d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:59.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.171:6789/0 conn(0x7f7790010bd0 legacy=0x7f7790010fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:59.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: _finish_hunting 0 2024-09-16T04:14:59.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 monclient: found mon.noname-a 2024-09-16T04:14:59.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7790009720 con 0x7f7790012e00 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: _finish_auth 0 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a0004270 con 0x7f7790012e00 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 10 monclient: dump: 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:59.925 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77b4ea6640 5 monclient: authenticate success, global_id 65130 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77b4ea6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.919+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77a0004510 con 0x7f7790012e00 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77ad59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77ad59a640 1 -- 172.21.15.148:0/1604261639 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a00048e0 con 0x7f7790012e00 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:59.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77ad59a640 10 monclient: dump: 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77b4ea6640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:59.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.920+0000 7f77b4ea6640 4 set_mon_vals no callback set 2024-09-16T04:14:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 10 monclient: shutdown 2024-09-16T04:14:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 1 -- 172.21.15.148:0/1604261639 >> v1:172.21.15.148:6789/0 conn(0x7f7790012e00 legacy=0x7f77900151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 1 -- 172.21.15.148:0/1604261639 shutdown_connections 2024-09-16T04:14:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 1 -- 172.21.15.148:0/1604261639 >> 172.21.15.148:0/1604261639 conn(0x7f77b00a5640 msgr2=0x7f77b00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:59.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 1 -- 172.21.15.148:0/1604261639 shutdown_connections 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.921+0000 7f77b4ea6640 1 -- 172.21.15.148:0/1604261639 wait complete. 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.922+0000 7f77b4ea6640 10 monclient: build_initial_monmap 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.922+0000 7f77b4ea6640 10 monclient: monmap: 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:59.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.922+0000 7f77b4ea6640 1 Processor -- start 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 1 -- start start 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient: init 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient(hunting): picked mon.noname-c con 0x7f7790010bd0 addr v1:172.21.15.148:6790/0 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient(hunting): picked mon.noname-b con 0x7f7790012e00 addr v1:172.21.15.171:6789/0 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient(hunting): picked mon.noname-a con 0x7f77b013ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b01cc030 con 0x7f77b013ae80 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b009dae0 con 0x7f7790012e00 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b00a4280 con 0x7f7790010bd0 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77ae59c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7790010bd0 legacy=0x7f77b0131a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:59.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77b4ea6640 10 monclient(hunting): authenticate will time out at 1320.548632s 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77aed9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f77b013ae80 0x7f77b01d4020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47566/0 (socket says 172.21.15.148:47566) 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.923+0000 7f77aed9d640 1 -- 172.21.15.148:0/3911167273 learned_addr learned my addr 172.21.15.148:0/3911167273 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 577388447 0 0) 0x7f7790009720 con 0x7f77b013ae80 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): my global_id is 65133 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7790011b40 con 0x7f77b013ae80 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1543839079 0 0) 0x7f7790011b40 con 0x7f77b013ae80 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7790009720 con 0x7f77b013ae80 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a00046c0 con 0x7f77b013ae80 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:59.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): dump: 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900540899 0 0) 0x7f77b009dae0 con 0x7f7790012e00 2024-09-16T04:14:59.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient(hunting): my global_id is 6679 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7790011b40 con 0x7f7790012e00 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2182843176 0 0) 0x7f7790009720 con 0x7f77b013ae80 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 >> v1:172.21.15.148:6790/0 conn(0x7f7790010bd0 legacy=0x7f77b0131a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 >> v1:172.21.15.171:6789/0 conn(0x7f7790012e00 legacy=0x7f77b01d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 monclient: found mon.a 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b0132140 con 0x7f77b013ae80 2024-09-16T04:14:59.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 5 monclient: authenticate success, global_id 65133 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 10 monclient: _renew_subs 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77b013ab30 con 0x7f77b013ae80 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 10 monclient: _renew_subs 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f77b009d800 con 0x7f77b013ae80 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77a0004c00 con 0x7f77b013ae80 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77a0004e30 con 0x7f77b013ae80 2024-09-16T04:14:59.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77acd99640 4 set_mon_vals no callback set 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.925+0000 7f77a6ffd640 10 monclient: dump: 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:14:59.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.926+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f77a0005050 con 0x7f77b013ae80 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.927+0000 7f77b4ea6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.927+0000 7f77b4ea6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.927+0000 7f77b4ea6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.927+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7794005080 con 0x7f77b013ae80 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.927+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7f77a0047a90 con 0x7f77b013ae80 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.927+0000 7f77ae59c640 10 monclient: get_auth_request con 0x7f779003a8d0 auth_method 0 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.932+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f77b01cc030 con 0x7f77b013ae80 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.932+0000 7f77a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:59.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:14:59.932+0000 7f77a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.166+0000 7f77b4ea6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.166+0000 7f77b4ea6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.166+0000 7f77b4ea6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.166+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7794005370 con 0x7f77b013ae80 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.167+0000 7f77a6ffd640 1 -- 172.21.15.148:0/3911167273 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 464692294) 0x7f77a0015e10 con 0x7f77b013ae80 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.167+0000 7f77a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.167+0000 7f77a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:15:00.169 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 >> v1:172.21.15.171:6812/7327273 conn(0x7f779003a8d0 legacy=0x7f779003cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 10 monclient: shutdown 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 >> v1:172.21.15.148:6789/0 conn(0x7f77b013ae80 legacy=0x7f77b01d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 shutdown_connections 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 >> 172.21.15.148:0/3911167273 conn(0x7f77b00a5640 msgr2=0x7f77b006e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 shutdown_connections 2024-09-16T04:15:00.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.169+0000 7f77b4ea6640 1 -- 172.21.15.148:0/3911167273 wait complete. 2024-09-16T04:15:00.182 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:15:00.182 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:15:00.182 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:15:00.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.278+0000 7fe94f96d640 10 monclient: get_monmap_and_config 2024-09-16T04:15:00.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.278+0000 7fe94f96d640 10 monclient: build_initial_monmap 2024-09-16T04:15:00.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.278+0000 7fe94f96d640 10 monclient: monmap: 2024-09-16T04:15:00.280 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:15:00.280 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.280 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:15:00.279244+0000 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:15:00.279244+0000 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.278+0000 7fe94f96d640 1 Processor -- start 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 1 -- start start 2024-09-16T04:15:00.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient: init 2024-09-16T04:15:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient: _reopen_session rank -1 2024-09-16T04:15:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:15:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient(hunting): picked mon.noname-b con 0x7fe94809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:15:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient(hunting): picked mon.noname-c con 0x7fe94809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:15:00.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient(hunting): picked mon.noname-a con 0x7fe94809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9480a1010 con 0x7fe94809cc30 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe94809f030 con 0x7fe94809c660 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe94809dc60 con 0x7fe94809bb70 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient(hunting): _renew_subs 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94f96d640 10 monclient(hunting): authenticate will time out at 1320.904580s 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94d709640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe94809bb70 legacy=0x7fe94809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94df0a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe94809cc30 0x7fe94813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47580/0 (socket says 172.21.15.148:47580) 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe94df0a640 1 -- 172.21.15.148:0/4103418377 learned_addr learned my addr 172.21.15.148:0/4103418377 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213595791 0 0) 0x7fe9480a1010 con 0x7fe94809cc30 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe93ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe93ffff640 10 monclient(hunting): my global_id is 65136 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.279+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe934003520 con 0x7fe94809cc30 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3353317200 0 0) 0x7fe94809f030 con 0x7fe94809c660 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient(hunting): my global_id is 6682 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9480a1010 con 0x7fe94809c660 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1513462287 0 0) 0x7fe934003520 con 0x7fe94809cc30 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 >> v1:172.21.15.148:6790/0 conn(0x7fe94809bb70 legacy=0x7fe94809bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 >> v1:172.21.15.171:6789/0 conn(0x7fe94809c660 legacy=0x7fe948138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 monclient: found mon.noname-a 2024-09-16T04:15:00.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94809df40 con 0x7fe94809cc30 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: _finish_auth 0 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 5 monclient: authenticate success, global_id 65136 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe940001990 con 0x7fe94809cc30 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: dump: 2024-09-16T04:15:00.292 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe940001bf0 con 0x7fe94809cc30 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:15:00.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 1 -- 172.21.15.148:0/4103418377 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe940001f40 con 0x7fe94809cc30 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe93ffff640 10 monclient: dump: 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 10 monclient: get_monmap_and_config success 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 4 set_mon_vals no callback set 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 10 monclient: shutdown 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 1 -- 172.21.15.148:0/4103418377 >> v1:172.21.15.148:6789/0 conn(0x7fe94809cc30 legacy=0x7fe94813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 1 -- 172.21.15.148:0/4103418377 shutdown_connections 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 1 -- 172.21.15.148:0/4103418377 >> 172.21.15.148:0/4103418377 conn(0x7fe9480a5640 msgr2=0x7fe9480a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 1 -- 172.21.15.148:0/4103418377 shutdown_connections 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.280+0000 7fe94f96d640 1 -- 172.21.15.148:0/4103418377 wait complete. 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.281+0000 7fe94f96d640 10 monclient: build_initial_monmap 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.281+0000 7fe94f96d640 10 monclient: monmap: 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:15:00.295 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.281+0000 7fe94f96d640 1 Processor -- start 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.281+0000 7fe94f96d640 1 -- start start 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.281+0000 7fe94f96d640 10 monclient: init 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient: _reopen_session rank -1 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:15:00.296 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient(hunting): picked mon.noname-c con 0x7fe94809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient(hunting): picked mon.noname-b con 0x7fe94809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient(hunting): picked mon.noname-a con 0x7fe94809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe94811c520 con 0x7fe94809cc30 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94d709640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fe94809bb70 legacy=0x7fe94813a0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe94811c800 con 0x7fe94809c660 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9480740a0 con 0x7fe94809bb70 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient(hunting): _renew_subs 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient(hunting): authenticate will time out at 1320.907255s 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94df0a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe94809cc30 0x7fe94813aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47584/0 (socket says 172.21.15.148:47584) 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94df0a640 1 -- 172.21.15.148:0/2721742809 learned_addr learned my addr 172.21.15.148:0/2721742809 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877180487 0 0) 0x7fe94811c520 con 0x7fe94809cc30 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): my global_id is 65139 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe92c003520 con 0x7fe94809cc30 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2680257841 0 0) 0x7fe94811c800 con 0x7fe94809c660 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): my global_id is 6685 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe94811c520 con 0x7fe94809c660 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 429476464 0 0) 0x7fe92c003520 con 0x7fe94809cc30 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe94811c800 con 0x7fe94809cc30 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9400018d0 con 0x7fe94809cc30 2024-09-16T04:15:00.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): dump: 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2190927575 0 0) 0x7fe94811c520 con 0x7fe94809c660 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe92c003520 con 0x7fe94809c660 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe9380018b0 con 0x7fe94809c660 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient(hunting): dump: 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.300 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2026917634 0 0) 0x7fe94811c800 con 0x7fe94809cc30 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.148:6790/0 conn(0x7fe94809bb70 legacy=0x7fe94813a0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.171:6789/0 conn(0x7fe94809c660 legacy=0x7fe94813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 monclient: found mon.a 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94813c170 con 0x7fe94809cc30 2024-09-16T04:15:00.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient: _finish_auth 0 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe93dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 5 monclient: authenticate success, global_id 65139 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient: _renew_subs 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.282+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe94811c240 con 0x7fe94809cc30 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe940004e10 con 0x7fe94809cc30 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe940005160 con 0x7fe94809cc30 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:15:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 10 monclient: dump: 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe94f96d640 10 monclient: _renew_subs 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93f7fe640 4 set_mon_vals no callback set 2024-09-16T04:15:00.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe94f96d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe9481d2a10 con 0x7fe94809cc30 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe940005300 con 0x7fe94809cc30 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe94f96d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.283+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe9480008d0 con 0x7fe94802d7c0 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fe940047cf0 con 0x7fe94809cc30 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe94cf08640 10 monclient: get_auth_request con 0x7fe92c038f10 auth_method 0 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3670588537 0 0) 0x7fe9480008d0 con 0x7fe94802d7c0 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient(hunting): my global_id is 65142 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe94811c800 con 0x7fe94802d7c0 2024-09-16T04:15:00.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 160994341 0 0) 0x7fe94811c800 con 0x7fe94802d7c0 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe9480008d0 con 0x7fe94802d7c0 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe944003f60 con 0x7fe94802d7c0 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient: dump: 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.305 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 593662296 0 0) 0x7fe9480008d0 con 0x7fe94802d7c0 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe94802dba0 con 0x7fe94802d7c0 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe944006370 con 0x7fe94802d7c0 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.148:6789/0 conn(0x7fe94802d7c0 legacy=0x7fe948068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.284+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.148:6789/0 conn(0x7fe94802d7c0 legacy=0x7fe948068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:15:00.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.290+0000 7fe94f96d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.290+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe948068890 con 0x7fe948071970 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087466096 0 0) 0x7fe948068890 con 0x7fe948071970 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient(hunting): my global_id is 65145 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9480008d0 con 0x7fe948071970 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3950908386 0 0) 0x7fe9480008d0 con 0x7fe948071970 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe948068890 con 0x7fe948071970 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe944004620 con 0x7fe948071970 2024-09-16T04:15:00.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient: dump: 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3375846592 0 0) 0x7fe948068890 con 0x7fe948071970 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe94813cb30 con 0x7fe948071970 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4135 (unknown 0 0 3606928618) 0x7fe9440058c0 con 0x7fe948071970 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.148:6789/0 conn(0x7fe948071970 legacy=0x7fe9481d6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe93dffb640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.148:6789/0 conn(0x7fe948071970 legacy=0x7fe9481d6c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:15:00.310 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 42, 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 279490, 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:15:00.311 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:15:00.312 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.313 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:15:00.314 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:15:00.315 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:15:00.316 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:15:00.317 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.318 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:15:00.319 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.320 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.321 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:15:00.322 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.171:6812/7327273 conn(0x7fe92c038f10 legacy=0x7fe92c03b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94f96d640 10 monclient: shutdown 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94f96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 >> v1:172.21.15.148:6789/0 conn(0x7fe94809cc30 legacy=0x7fe94813aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94df0a640 1 -- 172.21.15.148:0/2721742809 reap_dead start 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94df0a640 1 -- 172.21.15.148:0/2721742809 reap_dead start 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 shutdown_connections 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.291+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 >> 172.21.15.148:0/2721742809 conn(0x7fe9480a5640 msgr2=0x7fe948072ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:15:00.324 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.292+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 shutdown_connections 2024-09-16T04:15:00.325 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.292+0000 7fe94f96d640 1 -- 172.21.15.148:0/2721742809 wait complete. 2024-09-16T04:15:00.325 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:15:00.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:15:00.325 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.421+0000 7fc640477640 10 monclient: get_monmap_and_config 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.421+0000 7fc640477640 10 monclient: build_initial_monmap 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.421+0000 7fc640477640 10 monclient: monmap: 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:15:00.422246+0000 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:15:00.422246+0000 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.423 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:15:00.424 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:15:00.424 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:15:00.424 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.421+0000 7fc640477640 1 Processor -- start 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.421+0000 7fc640477640 1 -- start start 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.421+0000 7fc640477640 10 monclient: init 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient: _reopen_session rank -1 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient(hunting): picked mon.noname-a con 0x7fc63809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient(hunting): picked mon.noname-b con 0x7fc63809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient(hunting): picked mon.noname-c con 0x7fc63809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:15:00.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6380a1010 con 0x7fc63809bb70 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63809f030 con 0x7fc63809cc30 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63809dc60 con 0x7fc63809c660 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient(hunting): _renew_subs 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc640477640 10 monclient(hunting): authenticate will time out at 1321.047621s 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63da12640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fc63809c660 legacy=0x7fc638138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63e213640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc63809bb70 0x7fc63809bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47596/0 (socket says 172.21.15.148:47596) 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63e213640 1 -- 172.21.15.148:0/3500738643 learned_addr learned my addr 172.21.15.148:0/3500738643 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3587017292 0 0) 0x7fc6380a1010 con 0x7fc63809bb70 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63d211640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63d211640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63d211640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63d211640 10 monclient(hunting): my global_id is 65148 2024-09-16T04:15:00.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.422+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc624003520 con 0x7fc63809bb70 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270373148 0 0) 0x7fc63809f030 con 0x7fc63809cc30 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient(hunting): my global_id is 6688 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6380a1010 con 0x7fc63809cc30 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4209675661 0 0) 0x7fc624003520 con 0x7fc63809bb70 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 >> v1:172.21.15.148:6790/0 conn(0x7fc63809c660 legacy=0x7fc638138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 >> v1:172.21.15.171:6789/0 conn(0x7fc63809cc30 legacy=0x7fc63813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: _finish_hunting 0 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 monclient: found mon.noname-a 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:15:00.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63809df40 con 0x7fc63809bb70 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: _finish_auth 0 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: _check_auth_tickets 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc62c001610 con 0x7fc63809bb70 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 5 monclient: authenticate success, global_id 65148 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: dump: 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.435 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc62c0018c0 con 0x7fc63809bb70 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: handle_config config(0 keys) 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 1 -- 172.21.15.148:0/3500738643 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc62c001c90 con 0x7fc63809bb70 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 10 monclient: get_monmap_and_config success 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 4 set_mon_vals no callback set 2024-09-16T04:15:00.436 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc63d211640 10 monclient: dump: 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 10 monclient: shutdown 2024-09-16T04:15:00.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 1 -- 172.21.15.148:0/3500738643 >> v1:172.21.15.148:6789/0 conn(0x7fc63809bb70 legacy=0x7fc63809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 1 -- 172.21.15.148:0/3500738643 shutdown_connections 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 1 -- 172.21.15.148:0/3500738643 >> 172.21.15.148:0/3500738643 conn(0x7fc6380a5640 msgr2=0x7fc6380a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 1 -- 172.21.15.148:0/3500738643 shutdown_connections 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.423+0000 7fc640477640 1 -- 172.21.15.148:0/3500738643 wait complete. 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.424+0000 7fc640477640 10 monclient: build_initial_monmap 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.424+0000 7fc640477640 10 monclient: monmap: 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:15:00.438 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.424+0000 7fc640477640 1 Processor -- start 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.424+0000 7fc640477640 1 -- start start 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.424+0000 7fc640477640 10 monclient: init 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient: _reopen_session rank -1 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient(hunting): picked mon.noname-a con 0x7fc63809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient(hunting): picked mon.noname-c con 0x7fc63809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient(hunting): picked mon.noname-b con 0x7fc63809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:15:00.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63813c750 con 0x7fc63809bb70 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63806e370 con 0x7fc63809cc30 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63811c8f0 con 0x7fc63809c660 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient(hunting): _renew_subs 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient(hunting): authenticate will time out at 1321.050252s 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc63e213640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc63809bb70 0x7fc6381c9a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47598/0 (socket says 172.21.15.148:47598) 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc63e213640 1 -- 172.21.15.148:0/4145800016 learned_addr learned my addr 172.21.15.148:0/4145800016 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1734898194 0 0) 0x7fc63813c750 con 0x7fc63809bb70 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): my global_id is 65151 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc61c003550 con 0x7fc63809bb70 2024-09-16T04:15:00.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 441417990 0 0) 0x7fc61c003550 con 0x7fc63809bb70 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc63813c750 con 0x7fc63809bb70 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc62c001c10 con 0x7fc63809bb70 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): dump: 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.441 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496938456 0 0) 0x7fc6380a1010 con 0x7fc63809cc30 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient(hunting): my global_id is 6691 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc61c003550 con 0x7fc63809cc30 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3597975013 0 0) 0x7fc63813c750 con 0x7fc63809bb70 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.148:6790/0 conn(0x7fc63809c660 legacy=0x7fc6381cea10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.171:6789/0 conn(0x7fc63809cc30 legacy=0x7fc6381d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: _finish_hunting 0 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 monclient: found mon.a 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63811cbd0 con 0x7fc63809bb70 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: _finish_auth 0 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: _check_auth_tickets 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 5 monclient: authenticate success, global_id 65151 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient: _renew_subs 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc63811c610 con 0x7fc63809bb70 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient: _renew_subs 2024-09-16T04:15:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc63813b410 con 0x7fc63809bb70 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc62c004c70 con 0x7fc63809bb70 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc62c004fc0 con 0x7fc63809bb70 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc63ca10640 4 set_mon_vals no callback set 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.425+0000 7fc62affd640 10 monclient: dump: 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.426+0000 7fc640477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.426+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc6200065c0 con 0x7fc620002860 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.426+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc62c005160 con 0x7fc63809bb70 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.426+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 1..255) ==== 6817+0+0 (unknown 2092777191 0 0) 0x7fc62c047ca0 con 0x7fc63809bb70 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc63ea14640 10 monclient: get_auth_request con 0x7fc61c039090 auth_method 0 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986402852 0 0) 0x7fc6200065c0 con 0x7fc620002860 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient(hunting): my global_id is 6694 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc63813c750 con 0x7fc620002860 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 53042858 0 0) 0x7fc63813c750 con 0x7fc620002860 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc6200065c0 con 0x7fc620002860 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc634003e10 con 0x7fc620002860 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient: dump: 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.446 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3173189512 0 0) 0x7fc6200065c0 con 0x7fc620002860 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc62000a150 con 0x7fc620002860 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc634006260 con 0x7fc620002860 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.171:6789/0 conn(0x7fc620002860 legacy=0x7fc620004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.427+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.171:6789/0 conn(0x7fc620002860 legacy=0x7fc620004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc640477640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc620009e40 con 0x7fc620009100 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1218977396 0 0) 0x7fc620009e40 con 0x7fc620009100 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient(hunting): _negotiate 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:15:00.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient(hunting): my global_id is 6697 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6200065c0 con 0x7fc620009100 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 410743009 0 0) 0x7fc6200065c0 con 0x7fc620009100 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc620009e40 con 0x7fc620009100 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc6340045a0 con 0x7fc620009100 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 10 monclient: dump: 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:15:00.450 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3008801016 0 0) 0x7fc620009e40 con 0x7fc620009100 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.434+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc62000c3d0 con 0x7fc620009100 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4300 (unknown 0 0 104693365) 0x7fc634005880 con 0x7fc620009100 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc62affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc62affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.171:6789/0 conn(0x7fc620009100 legacy=0x7fc620009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc62affd640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.171:6789/0 conn(0x7fc620009100 legacy=0x7fc620009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 42, 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 524473, 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:15:00.452 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:15:00.453 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:15:00.454 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:15:00.455 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:15:00.456 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:15:00.457 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:15:00.458 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.459 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:15:00.460 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:15:00.461 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.462 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.463 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:15:00.464 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:15:00.465 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:15:00.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.171:6812/7327273 conn(0x7fc61c039090 legacy=0x7fc61c03b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 10 monclient: shutdown 2024-09-16T04:15:00.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:15:00.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 >> v1:172.21.15.148:6789/0 conn(0x7fc63809bb70 legacy=0x7fc6381c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:15:00.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc63ea14640 1 -- 172.21.15.148:0/4145800016 reap_dead start 2024-09-16T04:15:00.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc63ea14640 1 -- 172.21.15.148:0/4145800016 reap_dead start 2024-09-16T04:15:00.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 shutdown_connections 2024-09-16T04:15:00.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 >> 172.21.15.148:0/4145800016 conn(0x7fc6380a5640 msgr2=0x7fc638072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:15:00.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 shutdown_connections 2024-09-16T04:15:00.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:15:00.435+0000 7fc640477640 1 -- 172.21.15.148:0/4145800016 wait complete. 2024-09-16T04:15:00.467 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:15:00.467 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:16:30.468 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:16:30.469 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:16:30.469 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:16:30.470 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:16:30.473 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:16:30.474 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:16:30.474 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:16:30.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.581+0000 7fd150d98640 10 monclient: get_monmap_and_config 2024-09-16T04:16:30.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.581+0000 7fd150d98640 10 monclient: build_initial_monmap 2024-09-16T04:16:30.583 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.581+0000 7fd150d98640 10 monclient: monmap: 2024-09-16T04:16:30.583 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:30.583 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.583 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:30.582673+0000 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:30.582673+0000 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:30.584 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.582+0000 7fd150d98640 1 Processor -- start 2024-09-16T04:16:30.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.582+0000 7fd150d98640 1 -- start start 2024-09-16T04:16:30.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.582+0000 7fd150d98640 10 monclient: init 2024-09-16T04:16:30.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:30.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:16:30.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient(hunting): picked mon.noname-c con 0x7fd14c0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient(hunting): picked mon.noname-b con 0x7fd14c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient(hunting): picked mon.noname-a con 0x7fd14c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c0a0e00 con 0x7fd14c13d060 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c0a10e0 con 0x7fd14c0a2410 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c0a4970 con 0x7fd14c0a1b10 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient(hunting): authenticate will time out at 1411.208166s 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14ad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd14c13d060 0x7fd14c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54822/0 (socket says 172.21.15.148:54822) 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14ad9d640 1 -- 172.21.15.148:0/3388391529 learned_addr learned my addr 172.21.15.148:0/3388391529 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365095098 0 0) 0x7fd14c0a0e00 con 0x7fd14c13d060 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): my global_id is 65154 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd134003520 con 0x7fd14c13d060 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16686521 0 0) 0x7fd14c0a10e0 con 0x7fd14c0a2410 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient(hunting): my global_id is 6700 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd14c0a0e00 con 0x7fd14c0a2410 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1803545404 0 0) 0x7fd134003520 con 0x7fd14c13d060 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 >> v1:172.21.15.148:6790/0 conn(0x7fd14c0a1b10 legacy=0x7fd14c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.593 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 >> v1:172.21.15.171:6789/0 conn(0x7fd14c0a2410 legacy=0x7fd14c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: _finish_hunting 0 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 monclient: found mon.noname-a 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c06dfd0 con 0x7fd14c13d060 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: _finish_auth 0 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: _check_auth_tickets 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd144001990 con 0x7fd14c13d060 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 5 monclient: authenticate success, global_id 65154 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: dump: 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd144001bf0 con 0x7fd14c13d060 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 1 -- 172.21.15.148:0/3388391529 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd144001fc0 con 0x7fd14c13d060 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd14959a640 10 monclient: dump: 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.596 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.583+0000 7fd150d98640 4 set_mon_vals no callback set 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 10 monclient: shutdown 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 1 -- 172.21.15.148:0/3388391529 >> v1:172.21.15.148:6789/0 conn(0x7fd14c13d060 legacy=0x7fd14c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 1 -- 172.21.15.148:0/3388391529 shutdown_connections 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 1 -- 172.21.15.148:0/3388391529 >> 172.21.15.148:0/3388391529 conn(0x7fd14c0a97b0 msgr2=0x7fd14c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 1 -- 172.21.15.148:0/3388391529 shutdown_connections 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.584+0000 7fd150d98640 1 -- 172.21.15.148:0/3388391529 wait complete. 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.585+0000 7fd150d98640 10 monclient: build_initial_monmap 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.585+0000 7fd150d98640 10 monclient: monmap: 2024-09-16T04:16:30.597 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.585+0000 7fd150d98640 1 Processor -- start 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 1 -- start start 2024-09-16T04:16:30.598 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient: init 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient(hunting): picked mon.noname-c con 0x7fd14c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient(hunting): picked mon.noname-a con 0x7fd14c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient(hunting): picked mon.noname-b con 0x7fd14c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c11f7c0 con 0x7fd14c0a2410 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c09e620 con 0x7fd14c13d060 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd14c09cec0 con 0x7fd14c0a1b10 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd150d98640 10 monclient(hunting): authenticate will time out at 1411.211447s 2024-09-16T04:16:30.599 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd149d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd14c0a2410 0x7fd14c09c7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54836/0 (socket says 172.21.15.148:54836) 2024-09-16T04:16:30.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd149d9b640 1 -- 172.21.15.148:0/1987930791 learned_addr learned my addr 172.21.15.148:0/1987930791 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:30.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2436824254 0 0) 0x7fd14c09e620 con 0x7fd14c13d060 2024-09-16T04:16:30.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd13affd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd13affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd13affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd13affd640 10 monclient(hunting): my global_id is 6703 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.586+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd12c003520 con 0x7fd14c13d060 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2087279012 0 0) 0x7fd12c003520 con 0x7fd14c13d060 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd14c09e620 con 0x7fd14c13d060 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd1440016f0 con 0x7fd14c13d060 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): dump: 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.601 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2652423483 0 0) 0x7fd14c11f7c0 con 0x7fd14c0a2410 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient(hunting): my global_id is 65157 2024-09-16T04:16:30.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd12c003520 con 0x7fd14c0a2410 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3391396990 0 0) 0x7fd14c09e620 con 0x7fd14c13d060 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 >> v1:172.21.15.148:6790/0 conn(0x7fd14c0a1b10 legacy=0x7fd14c09c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 >> v1:172.21.15.148:6789/0 conn(0x7fd14c0a2410 legacy=0x7fd14c09c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: _finish_hunting 0 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 monclient: found mon.b 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c09e8a0 con 0x7fd14c13d060 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: _finish_auth 0 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: _check_auth_tickets 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 5 monclient: authenticate success, global_id 6703 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 10 monclient: _renew_subs 2024-09-16T04:16:30.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c11faa0 con 0x7fd14c13d060 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 10 monclient: _renew_subs 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd14c09f790 con 0x7fd14c13d060 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd144001bf0 con 0x7fd14c13d060 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd144005110 con 0x7fd14c13d060 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd148d99640 4 set_mon_vals no callback set 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.587+0000 7fd13affd640 10 monclient: dump: 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.604 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.588+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd144010540 con 0x7fd14c13d060 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.588+0000 7fd150d98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.588+0000 7fd150d98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.588+0000 7fd150d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.588+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd130005080 con 0x7fd14c13d060 2024-09-16T04:16:30.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.588+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fd144047e80 con 0x7fd14c13d060 2024-09-16T04:16:30.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.589+0000 7fd14a59c640 10 monclient: get_auth_request con 0x7fd12c039020 auth_method 0 2024-09-16T04:16:30.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.593+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd1440161a0 con 0x7fd14c13d060 2024-09-16T04:16:30.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.593+0000 7fd13affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:30.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.593+0000 7fd13affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:30.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.823+0000 7fd150d98640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:16:30.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.823+0000 7fd150d98640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:16:30.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.823+0000 7fd150d98640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:30.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.823+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd130005370 con 0x7fd14c13d060 2024-09-16T04:16:30.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.824+0000 7fd13affd640 1 -- 172.21.15.148:0/1987930791 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 994946471) 0x7fd144015aa0 con 0x7fd14c13d060 2024-09-16T04:16:30.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.824+0000 7fd13affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:16:30.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.824+0000 7fd13affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:30.826 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.826+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 >> v1:172.21.15.171:6812/7327273 conn(0x7fd12c039020 legacy=0x7fd12c03b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.826+0000 7fd150d98640 10 monclient: shutdown 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.826+0000 7fd150d98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.826+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 >> v1:172.21.15.171:6789/0 conn(0x7fd14c13d060 legacy=0x7fd14c1d8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.826+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 shutdown_connections 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.827+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 >> 172.21.15.148:0/1987930791 conn(0x7fd14c0a97b0 msgr2=0x7fd14c071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.827+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 shutdown_connections 2024-09-16T04:16:30.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.827+0000 7fd150d98640 1 -- 172.21.15.148:0/1987930791 wait complete. 2024-09-16T04:16:30.839 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:16:30.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:16:30.839 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.936+0000 7f6972482640 10 monclient: get_monmap_and_config 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.936+0000 7f6972482640 10 monclient: build_initial_monmap 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.936+0000 7f6972482640 10 monclient: monmap: 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:30.937315+0000 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:30.937315+0000 2024-09-16T04:16:30.938 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:30.939 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 1 Processor -- start 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 1 -- start start 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient: init 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient(hunting): picked mon.noname-b con 0x7f696c133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient(hunting): picked mon.noname-c con 0x7f696c06c430 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient(hunting): picked mon.noname-a con 0x7f696c13d0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c09b0d0 con 0x7f696c13d0e0 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c09b340 con 0x7f696c06c430 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c09b620 con 0x7f696c133910 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:30.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6972482640 10 monclient(hunting): authenticate will time out at 1411.562670s 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6970a1f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f696c13d0e0 0x7f696c13f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54848/0 (socket says 172.21.15.148:54848) 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f696bfff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f696c133910 0x7f696c133cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:34824/0 (socket says 172.21.15.148:34824) 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.937+0000 7f6970a1f640 1 -- 172.21.15.148:0/3900083299 learned_addr learned my addr 172.21.15.148:0/3900083299 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991734057 0 0) 0x7f696c09b0d0 con 0x7f696c13d0e0 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): my global_id is 65160 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6954003520 con 0x7f696c13d0e0 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2237724069 0 0) 0x7f696c09b340 con 0x7f696c06c430 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient(hunting): my global_id is 6706 2024-09-16T04:16:30.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f696c09b0d0 con 0x7f696c06c430 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2126412811 0 0) 0x7f696c09b0d0 con 0x7f696c06c430 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 >> v1:172.21.15.148:6790/0 conn(0x7f696c133910 legacy=0x7f696c133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 >> v1:172.21.15.148:6789/0 conn(0x7f696c13d0e0 legacy=0x7f696c13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: _finish_hunting 0 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 monclient: found mon.noname-c 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:16:30.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c0633c0 con 0x7f696c06c430 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: _finish_auth 0 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: _check_auth_tickets 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f69600015f0 con 0x7f696c06c430 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: dump: 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.950 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f6972482640 5 monclient: authenticate success, global_id 6706 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f6972482640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6960001950 con 0x7f696c06c430 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f6972482640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 1 -- 172.21.15.148:0/3900083299 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6960003610 con 0x7f696c06c430 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f6972482640 4 set_mon_vals no callback set 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.938+0000 7f696affd640 10 monclient: dump: 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.951 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 10 monclient: shutdown 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 1 -- 172.21.15.148:0/3900083299 >> v1:172.21.15.171:6789/0 conn(0x7f696c06c430 legacy=0x7f696c13b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 1 -- 172.21.15.148:0/3900083299 shutdown_connections 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 1 -- 172.21.15.148:0/3900083299 >> 172.21.15.148:0/3900083299 conn(0x7f696c062630 msgr2=0x7f696c062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 1 -- 172.21.15.148:0/3900083299 shutdown_connections 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 1 -- 172.21.15.148:0/3900083299 wait complete. 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 10 monclient: build_initial_monmap 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.939+0000 7f6972482640 10 monclient: monmap: 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:30.953 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 1 Processor -- start 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 1 -- start start 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 10 monclient: init 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 10 monclient(hunting): picked mon.noname-a con 0x7f696c06c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 10 monclient(hunting): picked mon.noname-b con 0x7f696c133910 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 10 monclient(hunting): picked mon.noname-c con 0x7f696c13d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.940+0000 7f6972482640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c13b120 con 0x7f696c06c430 2024-09-16T04:16:30.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c0a5180 con 0x7f696c133910 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c0a4dc0 con 0x7f696c13d0e0 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 10 monclient(hunting): authenticate will time out at 1411.566034s 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f696bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f696c06c430 0x7f696c13c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54850/0 (socket says 172.21.15.148:54850) 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f696bfff640 1 -- 172.21.15.148:0/1515998527 learned_addr learned my addr 172.21.15.148:0/1515998527 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2776932806 0 0) 0x7f696c0a4dc0 con 0x7f696c13d0e0 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): my global_id is 104099 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6958003520 con 0x7f696c13d0e0 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3032960971 0 0) 0x7f6958003520 con 0x7f696c13d0e0 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f696c0a4dc0 con 0x7f696c13d0e0 2024-09-16T04:16:30.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6964001b80 con 0x7f696c13d0e0 2024-09-16T04:16:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): dump: 2024-09-16T04:16:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 559217938 0 0) 0x7f696c0a4dc0 con 0x7f696c13d0e0 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.171:6789/0 conn(0x7f696c133910 legacy=0x7f696c1d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.148:6789/0 conn(0x7f696c06c430 legacy=0x7f696c13c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:16:30.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 monclient: found mon.c 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c06e000 con 0x7f696c13d0e0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: _finish_auth 0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 5 monclient: authenticate success, global_id 104099 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 10 monclient: _renew_subs 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f696c138e30 con 0x7f696c13d0e0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 10 monclient: _renew_subs 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f696c13cd60 con 0x7f696c13d0e0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6964004b60 con 0x7f696c13d0e0 2024-09-16T04:16:30.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6964004f30 con 0x7f696c13d0e0 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f696a7fc640 4 set_mon_vals no callback set 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.941+0000 7f6968ff9640 10 monclient: dump: 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f69640050d0 con 0x7f696c13d0e0 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6972482640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f694c0065c0 con 0x7f694c002860 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f6964047c90 con 0x7f696c13d0e0 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864310820 0 0) 0x7f694c0065c0 con 0x7f694c002860 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.942+0000 7f6968ff9640 10 monclient(hunting): my global_id is 65163 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f696c0a4dc0 con 0x7f694c002860 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f696b7fe640 10 monclient: get_auth_request con 0x7f6958038f80 auth_method 0 2024-09-16T04:16:30.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 580081999 0 0) 0x7f696c0a4dc0 con 0x7f694c002860 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f694c0065c0 con 0x7f694c002860 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f695c001670 con 0x7f694c002860 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 10 monclient: dump: 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.961 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2994876918 0 0) 0x7f694c0065c0 con 0x7f694c002860 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f694c00a150 con 0x7f694c002860 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f695c001aa0 con 0x7f694c002860 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.148:6789/0 conn(0x7f694c002860 legacy=0x7f694c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.943+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.148:6789/0 conn(0x7f694c002860 legacy=0x7f694c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6972482640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f694c009e40 con 0x7f694c009100 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3656997432 0 0) 0x7f694c009e40 con 0x7f694c009100 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient(hunting): my global_id is 65166 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f694c0065c0 con 0x7f694c009100 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 86376236 0 0) 0x7f694c0065c0 con 0x7f694c009100 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f694c009e40 con 0x7f694c009100 2024-09-16T04:16:30.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f695c004c30 con 0x7f694c009100 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 10 monclient: dump: 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3889765577 0 0) 0x7f694c009e40 con 0x7f694c009100 2024-09-16T04:16:30.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.950+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f694c00c3d0 con 0x7f694c009100 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4313 (unknown 0 0 3828870954) 0x7f695c004a50 con 0x7f694c009100 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6968ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6968ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.148:6789/0 conn(0x7f694c009100 legacy=0x7f694c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6968ff9640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.148:6789/0 conn(0x7f694c009100 legacy=0x7f694c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 44, 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 370149, 2024-09-16T04:16:30.966 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:30.967 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:30.968 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:30.969 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:30.970 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:30.971 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:30.972 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:30.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.974 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:30.975 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.976 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:30.977 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:30.978 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:30.978 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:30.978 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.171:6812/7327273 conn(0x7f6958038f80 legacy=0x7f695803b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 10 monclient: shutdown 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 >> v1:172.21.15.148:6790/0 conn(0x7f696c13d0e0 legacy=0x7f696c1d3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6970a1f640 1 -- 172.21.15.148:0/1515998527 reap_dead start 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6970a1f640 1 -- 172.21.15.148:0/1515998527 reap_dead start 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 shutdown_connections 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 >> 172.21.15.148:0/1515998527 conn(0x7f696c062630 msgr2=0x7f696c13d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 shutdown_connections 2024-09-16T04:16:30.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:30.951+0000 7f6972482640 1 -- 172.21.15.148:0/1515998527 wait complete. 2024-09-16T04:16:30.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:16:30.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:16:30.980 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:16:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.076+0000 7fbdeb1d8640 10 monclient: get_monmap_and_config 2024-09-16T04:16:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.076+0000 7fbdeb1d8640 10 monclient: build_initial_monmap 2024-09-16T04:16:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.076+0000 7fbdeb1d8640 10 monclient: monmap: 2024-09-16T04:16:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.078 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:31.077684+0000 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:31.077684+0000 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:31.079 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.077+0000 7fbdeb1d8640 1 Processor -- start 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.077+0000 7fbdeb1d8640 1 -- start start 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.077+0000 7fbdeb1d8640 10 monclient: init 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.077+0000 7fbdeb1d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.077+0000 7fbdeb1d8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.077+0000 7fbdeb1d8640 10 monclient(hunting): picked mon.noname-c con 0x7fbde409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 10 monclient(hunting): picked mon.noname-a con 0x7fbde409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:31.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 10 monclient(hunting): picked mon.noname-b con 0x7fbde409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde40a1010 con 0x7fbde409c660 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde409f030 con 0x7fbde409bb70 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde409dc60 con 0x7fbde409cc30 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 10 monclient(hunting): authenticate will time out at 1411.703071s 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde3fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbde409c660 0x7fbde4138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54882/0 (socket says 172.21.15.148:54882) 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde3fff640 1 -- 172.21.15.148:0/2344103869 learned_addr learned my addr 172.21.15.148:0/2344103869 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde9775640 1 --1- 172.21.15.148:0/2344103869 >> v1:172.21.15.148:6790/0 conn(0x7fbde409cc30 0x7fbde413c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:34836/0 (socket says 172.21.15.148:34836) 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187227754 0 0) 0x7fbde409dc60 con 0x7fbde409cc30 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): my global_id is 104102 2024-09-16T04:16:31.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdd4003520 con 0x7fbde409cc30 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515488248 0 0) 0x7fbde40a1010 con 0x7fbde409c660 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient(hunting): my global_id is 65169 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde409dc60 con 0x7fbde409c660 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2493864397 0 0) 0x7fbdd4003520 con 0x7fbde409cc30 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 >> v1:172.21.15.171:6789/0 conn(0x7fbde409bb70 legacy=0x7fbde409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 >> v1:172.21.15.148:6789/0 conn(0x7fbde409c660 legacy=0x7fbde4138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 monclient: found mon.noname-b 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:31.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde409df40 con 0x7fbde409cc30 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: _finish_auth 0 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbddc001990 con 0x7fbde409cc30 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 5 monclient: authenticate success, global_id 104102 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: dump: 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.090 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbddc001b80 con 0x7fbde409cc30 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 1 -- 172.21.15.148:0/2344103869 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbddc001f50 con 0x7fbde409cc30 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbde37fe640 10 monclient: dump: 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.078+0000 7fbdeb1d8640 4 set_mon_vals no callback set 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 10 monclient: shutdown 2024-09-16T04:16:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/2344103869 >> v1:172.21.15.148:6790/0 conn(0x7fbde409cc30 legacy=0x7fbde413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/2344103869 shutdown_connections 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/2344103869 >> 172.21.15.148:0/2344103869 conn(0x7fbde40a5640 msgr2=0x7fbde40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/2344103869 shutdown_connections 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/2344103869 wait complete. 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 10 monclient: build_initial_monmap 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.079+0000 7fbdeb1d8640 10 monclient: monmap: 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 1 Processor -- start 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 1 -- start start 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient: init 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient(hunting): picked mon.noname-b con 0x7fbde409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient(hunting): picked mon.noname-c con 0x7fbde409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient(hunting): picked mon.noname-a con 0x7fbde409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde40a4190 con 0x7fbde409cc30 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde4120e90 con 0x7fbde409bb70 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde40721a0 con 0x7fbde409c660 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbdeb1d8640 10 monclient(hunting): authenticate will time out at 1411.705795s 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbde9775640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbde409cc30 0x7fbde41cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54886/0 (socket says 172.21.15.148:54886) 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.080+0000 7fbde9775640 1 -- 172.21.15.148:0/3518256269 learned_addr learned my addr 172.21.15.148:0/3518256269 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574486144 0 0) 0x7fbde40721a0 con 0x7fbde409c660 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): my global_id is 104105 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc8003520 con 0x7fbde409c660 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766981937 0 0) 0x7fbde409df40 con 0x7fbde409cc30 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): my global_id is 65172 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde40721a0 con 0x7fbde409cc30 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2564479047 0 0) 0x7fbde4120e90 con 0x7fbde409bb70 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): my global_id is 6712 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde409df40 con 0x7fbde409bb70 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2806339060 0 0) 0x7fbde40721a0 con 0x7fbde409cc30 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbde4120e90 con 0x7fbde409cc30 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3609956454 0 0) 0x7fbdc8003520 con 0x7fbde409c660 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbde40721a0 con 0x7fbde409c660 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbddc001ed0 con 0x7fbde409cc30 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.097 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdd0001170 con 0x7fbde409c660 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4269420519 0 0) 0x7fbde409df40 con 0x7fbde409bb70 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc8003520 con 0x7fbde409bb70 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1807798219 0 0) 0x7fbde4120e90 con 0x7fbde409cc30 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.148:6790/0 conn(0x7fbde409c660 legacy=0x7fbde41cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.171:6789/0 conn(0x7fbde409bb70 legacy=0x7fbde4133a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: _finish_hunting 0 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 monclient: found mon.a 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde4120bb0 con 0x7fbde409cc30 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: _finish_auth 0 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 5 monclient: authenticate success, global_id 65172 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 10 monclient: _renew_subs 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbde4121170 con 0x7fbde409cc30 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 10 monclient: _renew_subs 2024-09-16T04:16:31.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbde4134180 con 0x7fbde409cc30 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbddc001b80 con 0x7fbde409cc30 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbddc004e10 con 0x7fbde409cc30 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde2ffd640 4 set_mon_vals no callback set 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.081+0000 7fbde17fa640 10 monclient: dump: 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbdeb1d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdc40065c0 con 0x7fbdc4002860 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbddc005030 con 0x7fbde409cc30 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fbddc047ba0 con 0x7fbde409cc30 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953233743 0 0) 0x7fbdc40065c0 con 0x7fbdc4002860 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.082+0000 7fbde17fa640 10 monclient(hunting): my global_id is 104108 2024-09-16T04:16:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde4120e90 con 0x7fbdc4002860 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde3fff640 10 monclient: get_auth_request con 0x7fbdc8039170 auth_method 0 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 768396028 0 0) 0x7fbde4120e90 con 0x7fbdc4002860 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc40065c0 con 0x7fbdc4002860 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdd80035f0 con 0x7fbdc4002860 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 10 monclient: dump: 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 981156018 0 0) 0x7fbdc40065c0 con 0x7fbdc4002860 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdc400a150 con 0x7fbdc4002860 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbdd8005900 con 0x7fbdc4002860 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.148:6790/0 conn(0x7fbdc4002860 legacy=0x7fbdc4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.083+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.148:6790/0 conn(0x7fbdc4002860 legacy=0x7fbdc4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbdeb1d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdc4009e40 con 0x7fbdc4009100 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941893534 0 0) 0x7fbdc4009e40 con 0x7fbdc4009100 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbde17fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbde17fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbde17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.089+0000 7fbde17fa640 10 monclient(hunting): my global_id is 104111 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc40065c0 con 0x7fbdc4009100 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2179985370 0 0) 0x7fbdc40065c0 con 0x7fbdc4009100 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc4009e40 con 0x7fbdc4009100 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbdd8003c40 con 0x7fbdc4009100 2024-09-16T04:16:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 10 monclient: dump: 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1659184916 0 0) 0x7fbdc4009e40 con 0x7fbdc4009100 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbdc400c3d0 con 0x7fbdc4009100 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3974 (unknown 0 0 4141961788) 0x7fbdd8005ab0 con 0x7fbdc4009100 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.148:6790/0 conn(0x7fbdc4009100 legacy=0x7fbdc4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde17fa640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.148:6790/0 conn(0x7fbdc4009100 legacy=0x7fbdc4009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:16:31.108 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 44, 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 526, 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:31.109 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:31.110 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:31.111 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:31.112 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.113 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.114 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.115 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:31.116 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:31.117 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.118 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:31.119 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.171:6812/7327273 conn(0x7fbdc8039170 legacy=0x7fbdc803b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbdeb1d8640 10 monclient: shutdown 2024-09-16T04:16:31.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbdeb1d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 >> v1:172.21.15.148:6789/0 conn(0x7fbde409cc30 legacy=0x7fbde41cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde9775640 1 -- 172.21.15.148:0/3518256269 reap_dead start 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbde9775640 1 -- 172.21.15.148:0/3518256269 reap_dead start 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 shutdown_connections 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.090+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 >> 172.21.15.148:0/3518256269 conn(0x7fbde40a5640 msgr2=0x7fbde41b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.091+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 shutdown_connections 2024-09-16T04:16:31.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.091+0000 7fbdeb1d8640 1 -- 172.21.15.148:0/3518256269 wait complete. 2024-09-16T04:16:31.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:16:31.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:16:31.121 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.217+0000 7fb5724b7640 10 monclient: get_monmap_and_config 2024-09-16T04:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.217+0000 7fb5724b7640 10 monclient: build_initial_monmap 2024-09-16T04:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.217+0000 7fb5724b7640 10 monclient: monmap: 2024-09-16T04:16:31.219 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:31.218884+0000 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:31.218884+0000 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:31.220 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.218+0000 7fb5724b7640 1 Processor -- start 2024-09-16T04:16:31.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.218+0000 7fb5724b7640 1 -- start start 2024-09-16T04:16:31.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.218+0000 7fb5724b7640 10 monclient: init 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient(hunting): picked mon.noname-c con 0x7fb56c0a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient(hunting): picked mon.noname-a con 0x7fb56c09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient(hunting): picked mon.noname-b con 0x7fb56c09b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c09e040 con 0x7fb56c09b170 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c09d660 con 0x7fb56c0a0e90 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c09c0d0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb5724b7640 10 monclient(hunting): authenticate will time out at 1411.844401s 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb563fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb56c09b170 0x7fb56c1383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54902/0 (socket says 172.21.15.148:54902) 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb563fff640 1 -- 172.21.15.148:0/3771180429 learned_addr learned my addr 172.21.15.148:0/3771180429 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:31.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516090964 0 0) 0x7fb554001110 con 0x7fb56c09b170 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): my global_id is 65175 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb558003520 con 0x7fb56c09b170 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417699910 0 0) 0x7fb56c09c0d0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient(hunting): my global_id is 104114 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb554001110 con 0x7fb56c09b5c0 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 185614464 0 0) 0x7fb554001110 con 0x7fb56c09b5c0 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 >> v1:172.21.15.171:6789/0 conn(0x7fb56c0a0e90 legacy=0x7fb56c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 >> v1:172.21.15.148:6789/0 conn(0x7fb56c09b170 legacy=0x7fb56c1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 monclient: found mon.noname-b 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.219+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c09d380 con 0x7fb56c09b5c0 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 5 monclient: authenticate success, global_id 104114 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb564001990 con 0x7fb56c09b5c0 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: dump: 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:31.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb564001b80 con 0x7fb56c09b5c0 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 1 -- 172.21.15.148:0/3771180429 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb564001f10 con 0x7fb56c09b5c0 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 4 set_mon_vals no callback set 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb56b7fe640 10 monclient: dump: 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.233 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 10 monclient: shutdown 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 1 -- 172.21.15.148:0/3771180429 >> v1:172.21.15.148:6790/0 conn(0x7fb56c09b5c0 legacy=0x7fb56c13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 1 -- 172.21.15.148:0/3771180429 shutdown_connections 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 1 -- 172.21.15.148:0/3771180429 >> 172.21.15.148:0/3771180429 conn(0x7fb56c0a5640 msgr2=0x7fb56c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 1 -- 172.21.15.148:0/3771180429 shutdown_connections 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.220+0000 7fb5724b7640 1 -- 172.21.15.148:0/3771180429 wait complete. 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient: build_initial_monmap 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient: monmap: 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:31.234 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 1 Processor -- start 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 1 -- start start 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient: init 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient(hunting): picked mon.noname-b con 0x7fb56c09b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:31.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.221+0000 7fb5724b7640 10 monclient(hunting): picked mon.noname-a con 0x7fb56c09b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient(hunting): picked mon.noname-c con 0x7fb56c0a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c1203c0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c0a4210 con 0x7fb56c09b170 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c0740a0 con 0x7fb56c0a0e90 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient(hunting): authenticate will time out at 1411.847047s 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb563fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb56c09b5c0 0x7fb56c1d09b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54904/0 (socket says 172.21.15.148:54904) 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb563fff640 1 -- 172.21.15.148:0/3814078700 learned_addr learned my addr 172.21.15.148:0/3814078700 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2829683386 0 0) 0x7fb56c1203c0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): my global_id is 65178 2024-09-16T04:16:31.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb54c003520 con 0x7fb56c09b5c0 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446504726 0 0) 0x7fb56c0740a0 con 0x7fb56c0a0e90 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): my global_id is 104117 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb56c1203c0 con 0x7fb56c0a0e90 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826418409 0 0) 0x7fb56c0a4210 con 0x7fb56c09b170 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): my global_id is 6718 2024-09-16T04:16:31.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb56c0740a0 con 0x7fb56c09b170 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1379709759 0 0) 0x7fb54c003520 con 0x7fb56c09b5c0 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb56c0a4210 con 0x7fb56c09b5c0 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb554001420 con 0x7fb56c09b5c0 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.238 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 886262560 0 0) 0x7fb56c1203c0 con 0x7fb56c0a0e90 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb54c003520 con 0x7fb56c0a0e90 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb564001ed0 con 0x7fb56c0a0e90 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.239 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 620481853 0 0) 0x7fb56c0740a0 con 0x7fb56c09b170 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb56c1203c0 con 0x7fb56c09b170 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb55c001830 con 0x7fb56c09b170 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.240 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1333435325 0 0) 0x7fb56c0a4210 con 0x7fb56c09b5c0 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.148:6790/0 conn(0x7fb56c0a0e90 legacy=0x7fb56c1d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.171:6789/0 conn(0x7fb56c09b170 legacy=0x7fb56c138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: _finish_hunting 0 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 monclient: found mon.a 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c1200e0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: _finish_auth 0 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: _check_auth_tickets 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:31.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 5 monclient: authenticate success, global_id 65178 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient: _renew_subs 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c139170 con 0x7fb56c09b5c0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient: _renew_subs 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb56c140310 con 0x7fb56c09b5c0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb554001810 con 0x7fb56c09b5c0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb554004c30 con 0x7fb56c09b5c0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb56affd640 4 set_mon_vals no callback set 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.222+0000 7fb5697fa640 10 monclient: dump: 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.223+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb55400fec0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.223+0000 7fb5724b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.223+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb56c072b20 con 0x7fb56c13c650 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.223+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fb5540476f0 con 0x7fb56c09b5c0 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb570a54640 10 monclient: get_auth_request con 0x7fb54c039160 auth_method 0 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677524272 0 0) 0x7fb56c072b20 con 0x7fb56c13c650 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient(hunting): my global_id is 6721 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb56c0a4210 con 0x7fb56c13c650 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2430606988 0 0) 0x7fb56c0a4210 con 0x7fb56c13c650 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb56c072b20 con 0x7fb56c13c650 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb55c003790 con 0x7fb56c13c650 2024-09-16T04:16:31.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 10 monclient: dump: 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2482750131 0 0) 0x7fb56c072b20 con 0x7fb56c13c650 2024-09-16T04:16:31.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb56c0008d0 con 0x7fb56c13c650 2024-09-16T04:16:31.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.224+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb55c005a10 con 0x7fb56c13c650 2024-09-16T04:16:31.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.225+0000 7fb5697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.225+0000 7fb5697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:31.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.225+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.171:6789/0 conn(0x7fb56c13c650 legacy=0x7fb56c13ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.225+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.171:6789/0 conn(0x7fb56c13c650 legacy=0x7fb56c13ca30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:31.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5724b7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:31.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb56c140600 con 0x7fb56c13d140 2024-09-16T04:16:31.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458148970 0 0) 0x7fb56c140600 con 0x7fb56c13d140 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient(hunting): my global_id is 6724 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb56c072b20 con 0x7fb56c13d140 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1169947019 0 0) 0x7fb56c072b20 con 0x7fb56c13d140 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb56c140600 con 0x7fb56c13d140 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb55c005170 con 0x7fb56c13d140 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.231+0000 7fb5697fa640 10 monclient: dump: 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.249 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2359173283 0 0) 0x7fb56c140600 con 0x7fb56c13d140 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb56c13d520 con 0x7fb56c13d140 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 3614881258) 0x7fb55c004620 con 0x7fb56c13d140 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.171:6789/0 conn(0x7fb56c13d140 legacy=0x7fb56c1d9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5697fa640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.171:6789/0 conn(0x7fb56c13d140 legacy=0x7fb56c1d9820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 44, 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 615270, 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:31.251 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:31.252 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:31.253 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:31.254 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:31.255 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:31.256 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.257 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:31.258 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:31.259 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.260 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:31.261 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:31.262 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.171:6812/7327273 conn(0x7fb54c039160 legacy=0x7fb54c03b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 10 monclient: shutdown 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 >> v1:172.21.15.148:6789/0 conn(0x7fb56c09b5c0 legacy=0x7fb56c1d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb570a54640 1 -- 172.21.15.148:0/3814078700 reap_dead start 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb570a54640 1 -- 172.21.15.148:0/3814078700 reap_dead start 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 shutdown_connections 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 >> 172.21.15.148:0/3814078700 conn(0x7fb56c0a5640 msgr2=0x7fb56c070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 shutdown_connections 2024-09-16T04:16:31.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.232+0000 7fb5724b7640 1 -- 172.21.15.148:0/3814078700 wait complete. 2024-09-16T04:16:31.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:16:31.265 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:16:31.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:16:31.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.361+0000 7f6b8ac4c640 10 monclient: get_monmap_and_config 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.361+0000 7f6b8ac4c640 10 monclient: build_initial_monmap 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.361+0000 7f6b8ac4c640 10 monclient: monmap: 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:31.362043+0000 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:31.362043+0000 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.363 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:31.364 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:31.364 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:31.364 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.361+0000 7f6b8ac4c640 1 Processor -- start 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.361+0000 7f6b8ac4c640 1 -- start start 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.361+0000 7f6b8ac4c640 10 monclient: init 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient(hunting): picked mon.noname-b con 0x7f6b8409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient(hunting): picked mon.noname-a con 0x7f6b8409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient(hunting): picked mon.noname-c con 0x7f6b8409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b840a1010 con 0x7f6b8409c660 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8409f030 con 0x7f6b8409cc30 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8409dc60 con 0x7f6b8409bb70 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:31.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b8ac4c640 10 monclient(hunting): authenticate will time out at 1411.987456s 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6b8409c660 0x7f6b84138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54908/0 (socket says 172.21.15.148:54908) 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b889e8640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6b8409bb70 0x7f6b8409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:34898/0 (socket says 172.21.15.148:34898) 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7bfff640 1 -- 172.21.15.148:0/3245616987 learned_addr learned my addr 172.21.15.148:0/3245616987 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745976991 0 0) 0x7f6b840a1010 con 0x7f6b8409c660 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): my global_id is 65181 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b6c003520 con 0x7f6b8409c660 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822767363 0 0) 0x7f6b8409dc60 con 0x7f6b8409bb70 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): my global_id is 104120 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b840a1010 con 0x7f6b8409bb70 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4164579030 0 0) 0x7f6b8409f030 con 0x7f6b8409cc30 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 10 monclient(hunting): my global_id is 6727 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.362+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b8409dc60 con 0x7f6b8409cc30 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1267660408 0 0) 0x7f6b6c003520 con 0x7f6b8409c660 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 >> v1:172.21.15.148:6790/0 conn(0x7f6b8409bb70 legacy=0x7f6b8409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 >> v1:172.21.15.171:6789/0 conn(0x7f6b8409cc30 legacy=0x7f6b8413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8409df40 con 0x7f6b8409c660 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6b74001610 con 0x7f6b8409c660 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 5 monclient: authenticate success, global_id 65181 2024-09-16T04:16:31.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: dump: 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.377 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b74001840 con 0x7f6b8409c660 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 1 -- 172.21.15.148:0/3245616987 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6b74001c50 con 0x7f6b8409c660 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 4 set_mon_vals no callback set 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b7b7fe640 10 monclient: dump: 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.378 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 10 monclient: shutdown 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/3245616987 >> v1:172.21.15.148:6789/0 conn(0x7f6b8409c660 legacy=0x7f6b84138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/3245616987 shutdown_connections 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/3245616987 >> 172.21.15.148:0/3245616987 conn(0x7f6b840a5640 msgr2=0x7f6b840a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:31.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/3245616987 shutdown_connections 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.363+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/3245616987 wait complete. 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.364+0000 7f6b8ac4c640 10 monclient: build_initial_monmap 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.364+0000 7f6b8ac4c640 10 monclient: monmap: 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:31.380 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.364+0000 7f6b8ac4c640 1 Processor -- start 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.364+0000 7f6b8ac4c640 1 -- start start 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.364+0000 7f6b8ac4c640 10 monclient: init 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient(hunting): picked mon.noname-a con 0x7f6b8409bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient(hunting): picked mon.noname-c con 0x7f6b8409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient(hunting): picked mon.noname-b con 0x7f6b8409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b840740a0 con 0x7f6b8409bb70 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8413cb30 con 0x7f6b8409cc30 2024-09-16T04:16:31.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b840a3fb0 con 0x7f6b8409c660 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b8ac4c640 10 monclient(hunting): authenticate will time out at 1411.990382s 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b889e8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6b8409bb70 0x7f6b84131c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:54924/0 (socket says 172.21.15.148:54924) 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b889e8640 1 -- 172.21.15.148:0/250276556 learned_addr learned my addr 172.21.15.148:0/250276556 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4180235018 0 0) 0x7f6b800018f0 con 0x7f6b8409bb70 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): my global_id is 65184 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b70003520 con 0x7f6b8409bb70 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3627685192 0 0) 0x7f6b840a3fb0 con 0x7f6b8409c660 2024-09-16T04:16:31.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): my global_id is 104123 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b800018f0 con 0x7f6b8409c660 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3057699643 0 0) 0x7f6b70003520 con 0x7f6b8409bb70 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b840a3fb0 con 0x7f6b8409bb70 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6b80001c20 con 0x7f6b8409bb70 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.383 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1272490204 0 0) 0x7f6b800018f0 con 0x7f6b8409c660 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b70003520 con 0x7f6b8409c660 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 881747318 0 0) 0x7f6b8413cb30 con 0x7f6b8409cc30 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:31.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): my global_id is 6730 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b800018f0 con 0x7f6b8409cc30 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6b74001b90 con 0x7f6b8409c660 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.365+0000 7f6b797fa640 10 monclient(hunting): dump: 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.385 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3172079217 0 0) 0x7f6b840a3fb0 con 0x7f6b8409bb70 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 >> v1:172.21.15.148:6790/0 conn(0x7f6b8409c660 legacy=0x7f6b84132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 >> v1:172.21.15.171:6789/0 conn(0x7f6b8409cc30 legacy=0x7f6b84132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: _finish_hunting 0 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 monclient: found mon.a 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b841359f0 con 0x7f6b8409bb70 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: _finish_auth 0 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: _check_auth_tickets 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 5 monclient: authenticate success, global_id 65184 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 10 monclient: _renew_subs 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b84135cd0 con 0x7f6b8409bb70 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 10 monclient: _renew_subs 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b841d27a0 con 0x7f6b8409bb70 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6b80004a90 con 0x7f6b8409bb70 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:31.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6b80004de0 con 0x7f6b8409bb70 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b7affd640 4 set_mon_vals no callback set 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 10 monclient: dump: 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:31.388 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.366+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6b80010210 con 0x7f6b8409bb70 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.367+0000 7f6b8ac4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.367+0000 7f6b8ac4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.367+0000 7f6b8ac4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.367+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b64005080 con 0x7f6b8409bb70 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.367+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f6b80047b40 con 0x7f6b8409bb70 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.367+0000 7f6b7bfff640 10 monclient: get_auth_request con 0x7f6b700390e0 auth_method 0 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.371+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6b840740a0 con 0x7f6b8409bb70 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.371+0000 7f6b797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:31.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.371+0000 7f6b797fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b8ac4c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b8ac4c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:16:31.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b8ac4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:31.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6b64005b80 con 0x7f6b8409bb70 2024-09-16T04:16:31.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b797fa640 1 -- 172.21.15.148:0/250276556 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f6b800199f0 con 0x7f6b8409bb70 2024-09-16T04:16:31.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:16:31.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.603+0000 7f6b797fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:31.607 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.604+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.607 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.604+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=3503511220,config=692619927,health=874629345,kv=2270513034,logm=2279073954}) 2024-09-16T04:16:31.607 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.604+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=9,logm=58} crc {auth=3106392227,config=692619927,health=3070137136,kv=3693054442,logm=167666895}) 2024-09-16T04:16:31.607 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.604+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.604+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=3503511220,config=692619927,health=874629345,kv=2270513034,logm=2279073954}) 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.604+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=10,logm=56} crc {auth=136702132,config=692619927,health=3070137136,kv=3976697123,logm=2392736574}) 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.605+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.605+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757063356}) 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.605+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3544426982}) 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.605+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.605+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757063356}) 2024-09-16T04:16:31.608 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.605+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619398919}) 2024-09-16T04:16:31.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 >> v1:172.21.15.171:6812/7327273 conn(0x7f6b700390e0 legacy=0x7f6b7003b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 10 monclient: shutdown 2024-09-16T04:16:31.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:31.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 >> v1:172.21.15.148:6789/0 conn(0x7f6b8409bb70 legacy=0x7f6b84131c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:31.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 shutdown_connections 2024-09-16T04:16:31.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 >> 172.21.15.148:0/250276556 conn(0x7f6b840a5640 msgr2=0x7f6b8409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:31.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 shutdown_connections 2024-09-16T04:16:31.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f6b8ac4c640 1 -- 172.21.15.148:0/250276556 wait complete. 2024-09-16T04:16:31.615 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.615 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762640775}) 2024-09-16T04:16:31.615 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15986455}) 2024-09-16T04:16:31.615 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.615 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762640775}) 2024-09-16T04:16:31.615 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.606+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269171336}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.607+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.607+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586208054}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.607+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2715861221}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.607+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.607+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586208054}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.607+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459555444}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310545137}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1435213156}) 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.616 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310545137}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463910683}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567741563}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3072963148}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567741563}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.608+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102516810}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.609+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.609+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664549281}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.609+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343362909}) 2024-09-16T04:16:31.617 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.609+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.609+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2664549281}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.609+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673125849}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588035983}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826040415}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588035983}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762885496}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360537989}) 2024-09-16T04:16:31.618 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318471084}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360537989}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.610+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696726746}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157678001}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1068383188}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157678001}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4135802197}) 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.619 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388763959}) 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1881336599}) 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388763959}) 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.611+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56526957}) 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.612+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.612+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208241709}) 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.612+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711572989}) 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.612+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.620 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.612+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208241709}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.612+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887024073}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298874531}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070155932}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298874531}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733440992}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235674095}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999687839}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235674095}) 2024-09-16T04:16:31.621 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.613+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2100797412}) 2024-09-16T04:16:31.624 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.614+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.624 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.614+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960779298}) 2024-09-16T04:16:31.624 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.614+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766875467}) 2024-09-16T04:16:31.624 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.614+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.624 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.614+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960779298}) 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.614+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=313494905}) 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019115453}) 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481538431}) 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019115453}) 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3963315299}) 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.625 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623523669}) 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21945136}) 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623523669}) 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.615+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664224435}) 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788517229}) 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4106147818}) 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.626 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788517229}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909417797}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792457071}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792174192}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792457071}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.616+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3351827719}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.617+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.617+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626643336}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.617+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47193621}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.617+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.617+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626643336}) 2024-09-16T04:16:31.627 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.617+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742052234}) 2024-09-16T04:16:31.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665688043}) 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016517407}) 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665688043}) 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2588196196}) 2024-09-16T04:16:31.640 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319848830}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=155846192}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319848830}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.618+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=112025215}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.619+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.619+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849471709}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.619+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2977295000}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.619+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.619+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849471709}) 2024-09-16T04:16:31.641 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.619+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788167658}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137657827}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=174278814}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137657827}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628357042}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229827288}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=869609853}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229827288}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.620+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3513153543}) 2024-09-16T04:16:31.642 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197955669}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2246955821}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197955669}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266167954}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746685448}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375586512}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=746685448}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.621+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633092981}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.622+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.622+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821489610}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.622+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4135493489}) 2024-09-16T04:16:31.643 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.622+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.622+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821489610}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.622+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910237181}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220499358}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376585079}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220499358}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2879334977}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661924362}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315142137}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661924362}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.623+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125733389}) 2024-09-16T04:16:31.644 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.624+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.624+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693001717}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.624+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711625106}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.624+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.624+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693001717}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.624+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280874720}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86020629}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327296324}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86020629}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209077586}) 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.645 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640407021}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064487277}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640407021}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.625+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3065189327}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595958791}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753618659}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595958791}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342666836}) 2024-09-16T04:16:31.646 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076056112}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154361062}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076056112}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.626+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837625258}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.627+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.627+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547727488}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.627+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563893839}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.627+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.627+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547727488}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.627+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275345857}) 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.647 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=842267861}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2194250838}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=842267861}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802353780}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528338975}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413096065}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528338975}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.628+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868770267}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.629+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.629+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216470767}) 2024-09-16T04:16:31.648 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.629+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596405368}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.629+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.629+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216470767}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.629+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3886027548}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226365958}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801695804}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226365958}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725794668}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050011570}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119038227}) 2024-09-16T04:16:31.649 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050011570}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.630+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=50087808}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.631+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.631+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11} crc {logm=238169052,mdsmap=1097804592,mgr=527166131,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2734109742,mon_config_key=2589496441}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.631+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2} crc {logm=858863441,mdsmap=1097804592,mgr=4222608532,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2734109742,mon_config_key=2528046720,monmap=48254013}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.631+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.631+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11} crc {logm=238169052,mdsmap=1097804592,mgr=527166131,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2734109742,mon_config_key=2589496441}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.631+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2} crc {logm=2254449284,mdsmap=1097804592,mgr=1586861337,mgr_metadata=582288965,mgrstat=2734109742,mon_config_key=3438290323,monmap=48254013}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.633+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.633+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=937654316}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.633+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=89} crc {monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=3823785925,osdmap=1921702359}) 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.633+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.650 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.633+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=88} crc {monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=937654316}) 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.633+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=2633035914,osd_metadata=2197806543,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=3936404593}) 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.634+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.634+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=2041506814}) 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.634+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=3193042709}) 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.634+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.634+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=2041506814}) 2024-09-16T04:16:31.651 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:16:31.634+0000 7f3aabf6b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=2132933613}) 2024-09-16T04:16:32.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:16:32.630 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:16:32.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.737+0000 7f8915981640 10 monclient: get_monmap_and_config 2024-09-16T04:16:32.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.737+0000 7f8915981640 10 monclient: build_initial_monmap 2024-09-16T04:16:32.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.737+0000 7f8915981640 10 monclient: monmap: 2024-09-16T04:16:32.739 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:32.739 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:32.739 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:32.738322+0000 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:32.738322+0000 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:32.740 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.737+0000 7f8915981640 1 Processor -- start 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 1 -- start start 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient: init 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient(hunting): picked mon.noname-a con 0x7f89100a22b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient(hunting): picked mon.noname-c con 0x7f89100a1b70 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient(hunting): picked mon.noname-b con 0x7f891013d0d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89100a0440 con 0x7f89100a22b0 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89100a4d90 con 0x7f89100a1b70 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89100a08f0 con 0x7f891013d0d0 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:32.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f8915981640 10 monclient(hunting): authenticate will time out at 1413.363561s 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890effd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f89100a22b0 0x7f89100a2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60366/0 (socket says 172.21.15.148:60366) 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890f7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f891013d0d0 0x7f891013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:52614/0 (socket says 172.21.15.148:52614) 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890effd640 1 -- 172.21.15.148:0/4256850366 learned_addr learned my addr 172.21.15.148:0/4256850366 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460806520 0 0) 0x7f89100a0440 con 0x7f89100a22b0 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): my global_id is 65187 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f8003520 con 0x7f89100a22b0 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3045721646 0 0) 0x7f89100a08f0 con 0x7f891013d0d0 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:32.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 10 monclient(hunting): my global_id is 104126 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.738+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89100a0440 con 0x7f891013d0d0 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1237125028 0 0) 0x7f88f8003520 con 0x7f89100a22b0 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 >> v1:172.21.15.148:6790/0 conn(0x7f891013d0d0 legacy=0x7f891013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 >> v1:172.21.15.171:6789/0 conn(0x7f89100a1b70 legacy=0x7f891013b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 monclient: found mon.noname-a 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89100a2da0 con 0x7f89100a22b0 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: _finish_auth 0 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:16:32.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8900001610 con 0x7f89100a22b0 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 5 monclient: authenticate success, global_id 65187 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: dump: 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:32.752 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89000018f0 con 0x7f89100a22b0 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 1 -- 172.21.15.148:0/4256850366 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8900001d00 con 0x7f89100a22b0 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 4 set_mon_vals no callback set 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f890dffb640 10 monclient: dump: 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:32.753 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 10 monclient: shutdown 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 1 -- 172.21.15.148:0/4256850366 >> v1:172.21.15.148:6789/0 conn(0x7f89100a22b0 legacy=0x7f89100a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 1 -- 172.21.15.148:0/4256850366 shutdown_connections 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 1 -- 172.21.15.148:0/4256850366 >> 172.21.15.148:0/4256850366 conn(0x7f89100732c0 msgr2=0x7f89100736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:32.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 1 -- 172.21.15.148:0/4256850366 shutdown_connections 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.739+0000 7f8915981640 1 -- 172.21.15.148:0/4256850366 wait complete. 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.740+0000 7f8915981640 10 monclient: build_initial_monmap 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.740+0000 7f8915981640 10 monclient: monmap: 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:32.755 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.740+0000 7f8915981640 1 Processor -- start 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 1 -- start start 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient: init 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient(hunting): picked mon.noname-a con 0x7f89100a1b70 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient(hunting): picked mon.noname-c con 0x7f89100a22b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient(hunting): picked mon.noname-b con 0x7f891013d0d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891013ceb0 con 0x7f89100a1b70 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8910124100 con 0x7f891013d0d0 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89101402d0 con 0x7f89100a22b0 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f8915981640 10 monclient(hunting): authenticate will time out at 1413.366584s 2024-09-16T04:16:32.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f890effd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f89100a1b70 0x7f891013c7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60380/0 (socket says 172.21.15.148:60380) 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f890effd640 1 -- 172.21.15.148:0/2679608543 learned_addr learned my addr 172.21.15.148:0/2679608543 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4135962771 0 0) 0x7f891013ceb0 con 0x7f89100a1b70 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): my global_id is 65190 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88fc003520 con 0x7f89100a1b70 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756108037 0 0) 0x7f89101402d0 con 0x7f89100a22b0 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 10 monclient(hunting): my global_id is 104129 2024-09-16T04:16:32.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f891013ceb0 con 0x7f89100a22b0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.741+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3353992348 0 0) 0x7f8910124100 con 0x7f891013d0d0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): my global_id is 6736 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89101402d0 con 0x7f891013d0d0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3863684616 0 0) 0x7f891013ceb0 con 0x7f89100a22b0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8910124100 con 0x7f89100a22b0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f89040017b0 con 0x7f89100a22b0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): dump: 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:32.758 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1883825905 0 0) 0x7f88fc003520 con 0x7f89100a1b70 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f891013ceb0 con 0x7f89100a1b70 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8900001c40 con 0x7f89100a1b70 2024-09-16T04:16:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient(hunting): dump: 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2515456621 0 0) 0x7f89101402d0 con 0x7f891013d0d0 2024-09-16T04:16:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88fc003520 con 0x7f891013d0d0 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1779269696 0 0) 0x7f8910124100 con 0x7f89100a22b0 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 >> v1:172.21.15.171:6789/0 conn(0x7f891013d0d0 legacy=0x7f89101d3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 >> v1:172.21.15.148:6789/0 conn(0x7f89100a1b70 legacy=0x7f891013c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 monclient: found mon.c 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89100a81e0 con 0x7f89100a22b0 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: _finish_auth 0 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 5 monclient: authenticate success, global_id 104129 2024-09-16T04:16:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 10 monclient: _renew_subs 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89100a0130 con 0x7f89100a22b0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 10 monclient: _renew_subs 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8910123e20 con 0x7f89100a22b0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89040035f0 con 0x7f89100a22b0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8904003940 con 0x7f89100a22b0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f890d7fa640 4 set_mon_vals no callback set 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.742+0000 7f88f7fff640 10 monclient: dump: 2024-09-16T04:16:32.762 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f890400ebd0 con 0x7f89100a22b0 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f8915981640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f8915981640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f8915981640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:32.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f8904046540 con 0x7f89100a22b0 2024-09-16T04:16:32.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f0005080 con 0x7f89100a22b0 2024-09-16T04:16:32.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.743+0000 7f890effd640 10 monclient: get_auth_request con 0x7f88fc0391e0 auth_method 0 2024-09-16T04:16:32.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.747+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8904014800 con 0x7f89100a22b0 2024-09-16T04:16:32.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.747+0000 7f88f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:32.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.747+0000 7f88f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:32.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.980+0000 7f8915981640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:16:32.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.980+0000 7f8915981640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:16:32.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.980+0000 7f8915981640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:32.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.980+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f88f0005370 con 0x7f89100a22b0 2024-09-16T04:16:32.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.981+0000 7f88f7fff640 1 -- 172.21.15.148:0/2679608543 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 750260278) 0x7f8904014100 con 0x7f89100a22b0 2024-09-16T04:16:32.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.981+0000 7f88f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:16:32.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.981+0000 7f88f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:32.983 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 >> v1:172.21.15.171:6812/7327273 conn(0x7f88fc0391e0 legacy=0x7f88fc03b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 10 monclient: shutdown 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 >> v1:172.21.15.148:6790/0 conn(0x7f89100a22b0 legacy=0x7f89101d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 shutdown_connections 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 >> 172.21.15.148:0/2679608543 conn(0x7f89100732c0 msgr2=0x7f89100a6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 shutdown_connections 2024-09-16T04:16:32.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:32.983+0000 7f8915981640 1 -- 172.21.15.148:0/2679608543 wait complete. 2024-09-16T04:16:32.995 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:16:32.995 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:16:32.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:16:32.995 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.092+0000 7f7f4bbd6640 10 monclient: get_monmap_and_config 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.092+0000 7f7f4bbd6640 10 monclient: build_initial_monmap 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.092+0000 7f7f4bbd6640 10 monclient: monmap: 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:33.093477+0000 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:33.093477+0000 2024-09-16T04:16:33.094 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:33.095 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.095 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:33.095 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:33.095 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:33.095 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 1 Processor -- start 2024-09-16T04:16:33.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 1 -- start start 2024-09-16T04:16:33.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient: init 2024-09-16T04:16:33.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4409c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f440a1010 con 0x7f7f4409cc30 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4409f030 con 0x7f7f4409c660 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4409dc60 con 0x7f7f4409bb70 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4bbd6640 10 monclient(hunting): authenticate will time out at 1413.718810s 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4a173640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7f4409cc30 0x7f7f4413c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60382/0 (socket says 172.21.15.148:60382) 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f49972640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7f4409bb70 0x7f7f4409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:52630/0 (socket says 172.21.15.148:52630) 2024-09-16T04:16:33.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.093+0000 7f7f4a173640 1 -- 172.21.15.148:0/2437207177 learned_addr learned my addr 172.21.15.148:0/2437207177 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089435533 0 0) 0x7f7f4409dc60 con 0x7f7f4409bb70 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): my global_id is 104132 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f30003520 con 0x7f7f4409bb70 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840109503 0 0) 0x7f7f440a1010 con 0x7f7f4409cc30 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): my global_id is 65193 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f4409dc60 con 0x7f7f4409cc30 2024-09-16T04:16:33.105 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278001785 0 0) 0x7f7f4409f030 con 0x7f7f4409c660 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient(hunting): my global_id is 6739 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f440a1010 con 0x7f7f4409c660 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2877524601 0 0) 0x7f7f4409dc60 con 0x7f7f4409cc30 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 >> v1:172.21.15.148:6790/0 conn(0x7f7f4409bb70 legacy=0x7f7f4409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 >> v1:172.21.15.171:6789/0 conn(0x7f7f4409c660 legacy=0x7f7f44138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: _finish_hunting 0 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 monclient: found mon.noname-a 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:16:33.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4409df40 con 0x7f7f4409cc30 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: _finish_auth 0 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: _check_auth_tickets 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f3c0018f0 con 0x7f7f4409cc30 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f4bbd6640 5 monclient: authenticate success, global_id 65193 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f4bbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: dump: 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.107 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f4bbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c001b60 con 0x7f7f4409cc30 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 1 -- 172.21.15.148:0/2437207177 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f3c001f70 con 0x7f7f4409cc30 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f48970640 10 monclient: dump: 2024-09-16T04:16:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f4bbd6640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.094+0000 7f7f4bbd6640 4 set_mon_vals no callback set 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 10 monclient: shutdown 2024-09-16T04:16:33.109 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/2437207177 >> v1:172.21.15.148:6789/0 conn(0x7f7f4409cc30 legacy=0x7f7f4413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/2437207177 shutdown_connections 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/2437207177 >> 172.21.15.148:0/2437207177 conn(0x7f7f440a5640 msgr2=0x7f7f440a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/2437207177 shutdown_connections 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/2437207177 wait complete. 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 10 monclient: build_initial_monmap 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.095+0000 7f7f4bbd6640 10 monclient: monmap: 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:33.110 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 1 Processor -- start 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 1 -- start start 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient: init 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:33.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f44120d90 con 0x7f7f4409cc30 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4411f400 con 0x7f7f4409bb70 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f441206d0 con 0x7f7f4409c660 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4bbd6640 10 monclient(hunting): authenticate will time out at 1413.721513s 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4a173640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7f4409cc30 0x7f7f441cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60388/0 (socket says 172.21.15.148:60388) 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f49171640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7f4409c660 0x7f7f4411ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:52634/0 (socket says 172.21.15.148:52634) 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f4a173640 1 -- 172.21.15.148:0/308050572 learned_addr learned my addr 172.21.15.148:0/308050572 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1044091229 0 0) 0x7f7f441206d0 con 0x7f7f4409c660 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): my global_id is 104135 2024-09-16T04:16:33.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f28003520 con 0x7f7f4409c660 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2535548302 0 0) 0x7f7f44120d90 con 0x7f7f4409cc30 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): my global_id is 65196 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f441206d0 con 0x7f7f4409cc30 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185198500 0 0) 0x7f7f4411f400 con 0x7f7f4409bb70 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 10 monclient(hunting): my global_id is 6742 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.096+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f44120d90 con 0x7f7f4409bb70 2024-09-16T04:16:33.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1612604002 0 0) 0x7f7f28003520 con 0x7f7f4409c660 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f4411f400 con 0x7f7f4409c660 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3849924597 0 0) 0x7f7f441206d0 con 0x7f7f4409cc30 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f28003520 con 0x7f7f4409cc30 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f340018b0 con 0x7f7f4409c660 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): dump: 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.114 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f3c0017f0 con 0x7f7f4409cc30 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient(hunting): dump: 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.115 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1368003423 0 0) 0x7f7f4411f400 con 0x7f7f4409c660 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.171:6789/0 conn(0x7f7f4409bb70 legacy=0x7f7f4411e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.148:6789/0 conn(0x7f7f4409cc30 legacy=0x7f7f441cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 monclient: found mon.c 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f44072040 con 0x7f7f4409c660 2024-09-16T04:16:33.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 5 monclient: authenticate success, global_id 104135 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 10 monclient: _renew_subs 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f440a3fb0 con 0x7f7f4409c660 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 10 monclient: _renew_subs 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f44121f60 con 0x7f7f4409c660 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f34001c60 con 0x7f7f4409c660 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f34003a60 con 0x7f7f4409c660 2024-09-16T04:16:33.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3bfff640 4 set_mon_vals no callback set 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 10 monclient: dump: 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.097+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7f34003c80 con 0x7f7f4409c660 2024-09-16T04:16:33.118 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f4bbd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7f2c0065c0 con 0x7f7f2c002860 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f7f34047a70 con 0x7f7f4409c660 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775636188 0 0) 0x7f7f2c0065c0 con 0x7f7f2c002860 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient(hunting): my global_id is 65199 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f4411f400 con 0x7f7f2c002860 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f4a173640 10 monclient: get_auth_request con 0x7f7f28038fd0 auth_method 0 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3626555337 0 0) 0x7f7f4411f400 con 0x7f7f2c002860 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f2c0065c0 con 0x7f7f2c002860 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f40001960 con 0x7f7f2c002860 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:33.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 10 monclient: dump: 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1904033115 0 0) 0x7f7f2c0065c0 con 0x7f7f2c002860 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.098+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f2c00a150 con 0x7f7f2c002860 2024-09-16T04:16:33.120 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.099+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7f400017c0 con 0x7f7f2c002860 2024-09-16T04:16:33.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.099+0000 7f7f3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:33.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.099+0000 7f7f3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:33.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.099+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.148:6789/0 conn(0x7f7f2c002860 legacy=0x7f7f2c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.121 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.099+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.148:6789/0 conn(0x7f7f2c002860 legacy=0x7f7f2c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:33.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f4bbd6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7f2c009e40 con 0x7f7f2c009100 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336550075 0 0) 0x7f7f2c009e40 con 0x7f7f2c009100 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient(hunting): my global_id is 65202 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f2c0065c0 con 0x7f7f2c009100 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1254366306 0 0) 0x7f7f2c0065c0 con 0x7f7f2c009100 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f2c009e40 con 0x7f7f2c009100 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7f40004b70 con 0x7f7f2c009100 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.123 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient: dump: 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2873482130 0 0) 0x7f7f2c009e40 con 0x7f7f2c009100 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f2c00c3d0 con 0x7f7f2c009100 2024-09-16T04:16:33.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4313 (unknown 0 0 2837648346) 0x7f7f40004bd0 con 0x7f7f2c009100 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.148:6789/0 conn(0x7f7f2c009100 legacy=0x7f7f2c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.105+0000 7f7f3a7fc640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.148:6789/0 conn(0x7f7f2c009100 legacy=0x7f7f2c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 44, 2024-09-16T04:16:33.125 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 372303, 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.126 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.127 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:33.128 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:33.129 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.130 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.131 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.132 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:33.133 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.134 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:16:33.135 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:16:33.136 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:33.137 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.171:6812/7327273 conn(0x7f7f28038fd0 legacy=0x7f7f2803b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 10 monclient: shutdown 2024-09-16T04:16:33.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 >> v1:172.21.15.148:6790/0 conn(0x7f7f4409c660 legacy=0x7f7f4411ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4a173640 1 -- 172.21.15.148:0/308050572 reap_dead start 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4a173640 1 -- 172.21.15.148:0/308050572 reap_dead start 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 shutdown_connections 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 >> 172.21.15.148:0/308050572 conn(0x7f7f440a5640 msgr2=0x7f7f441338f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 shutdown_connections 2024-09-16T04:16:33.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.106+0000 7f7f4bbd6640 1 -- 172.21.15.148:0/308050572 wait complete. 2024-09-16T04:16:33.139 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:16:33.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:16:33.140 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.237+0000 7f5b3ebf6640 10 monclient: get_monmap_and_config 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.237+0000 7f5b3ebf6640 10 monclient: build_initial_monmap 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.237+0000 7f5b3ebf6640 10 monclient: monmap: 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:33.238475+0000 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:33.238475+0000 2024-09-16T04:16:33.239 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:33.240 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.240 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:33.240 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:33.240 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:33.240 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 1 Processor -- start 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 1 -- start start 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient: init 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient(hunting): picked mon.noname-c con 0x7f5b38098400 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient(hunting): picked mon.noname-b con 0x7f5b380987e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient(hunting): picked mon.noname-a con 0x7f5b38098bf0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3809ced0 con 0x7f5b38098bf0 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b38137480 con 0x7f5b38098400 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3809cbf0 con 0x7f5b380987e0 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:33.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3ebf6640 10 monclient(hunting): authenticate will time out at 1413.863762s 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3d193640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5b38098bf0 0x7f5b3813f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60410/0 (socket says 172.21.15.148:60410) 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b37fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5b380987e0 0x7f5b3813bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:52640/0 (socket says 172.21.15.148:52640) 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.238+0000 7f5b3d193640 1 -- 172.21.15.148:0/3233561414 learned_addr learned my addr 172.21.15.148:0/3233561414 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526798419 0 0) 0x7f5b3809cbf0 con 0x7f5b380987e0 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): my global_id is 104138 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b28003520 con 0x7f5b380987e0 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267723064 0 0) 0x7f5b30001490 con 0x7f5b38098bf0 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient(hunting): my global_id is 65205 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b3809cbf0 con 0x7f5b38098bf0 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2461677884 0 0) 0x7f5b28003520 con 0x7f5b380987e0 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 >> v1:172.21.15.171:6789/0 conn(0x7f5b38098400 legacy=0x7f5b38136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 >> v1:172.21.15.148:6789/0 conn(0x7f5b38098bf0 legacy=0x7f5b3813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: _finish_hunting 0 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 monclient: found mon.noname-b 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b380a16d0 con 0x7f5b380987e0 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: _finish_auth 0 2024-09-16T04:16:33.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: _check_auth_tickets 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b24001540 con 0x7f5b380987e0 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 5 monclient: authenticate success, global_id 104138 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: dump: 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.252 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b24001730 con 0x7f5b380987e0 2024-09-16T04:16:33.253 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 1 -- 172.21.15.148:0/3233561414 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b24001b40 con 0x7f5b380987e0 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 4 set_mon_vals no callback set 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b377fe640 10 monclient: dump: 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.254 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 10 monclient: shutdown 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.239+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/3233561414 >> v1:172.21.15.148:6790/0 conn(0x7f5b380987e0 legacy=0x7f5b3813bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/3233561414 shutdown_connections 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/3233561414 >> 172.21.15.148:0/3233561414 conn(0x7f5b380a5860 msgr2=0x7f5b380a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/3233561414 shutdown_connections 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/3233561414 wait complete. 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 10 monclient: build_initial_monmap 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 10 monclient: monmap: 2024-09-16T04:16:33.255 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 1 Processor -- start 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 1 -- start start 2024-09-16T04:16:33.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.240+0000 7f5b3ebf6640 10 monclient: init 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient(hunting): picked mon.noname-c con 0x7f5b38098400 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient(hunting): picked mon.noname-a con 0x7f5b380987e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient(hunting): picked mon.noname-b con 0x7f5b38098bf0 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b38140310 con 0x7f5b380987e0 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3813e7a0 con 0x7f5b38098bf0 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b381405f0 con 0x7f5b38098400 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3ebf6640 10 monclient(hunting): authenticate will time out at 1413.866552s 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3c992640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5b38098400 0x7f5b38135820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:52656/0 (socket says 172.21.15.148:52656) 2024-09-16T04:16:33.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b3c992640 1 -- 172.21.15.148:0/1661422137 learned_addr learned my addr 172.21.15.148:0/1661422137 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151071902 0 0) 0x7f5b38140310 con 0x7f5b380987e0 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): my global_id is 65208 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b1c003520 con 0x7f5b380987e0 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1012752943 0 0) 0x7f5b381405f0 con 0x7f5b38098400 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): my global_id is 104141 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b38140310 con 0x7f5b38098400 2024-09-16T04:16:33.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203704877 0 0) 0x7f5b3813e7a0 con 0x7f5b38098bf0 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 10 monclient(hunting): my global_id is 6748 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.241+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b381405f0 con 0x7f5b38098bf0 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1947445221 0 0) 0x7f5b38140310 con 0x7f5b38098400 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b3813e7a0 con 0x7f5b38098400 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3784132534 0 0) 0x7f5b1c003520 con 0x7f5b380987e0 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b38140310 con 0x7f5b380987e0 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b2c001890 con 0x7f5b38098400 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:33.259 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): dump: 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b24001a80 con 0x7f5b380987e0 2024-09-16T04:16:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): dump: 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.261 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3476200522 0 0) 0x7f5b381405f0 con 0x7f5b38098bf0 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b1c003520 con 0x7f5b38098bf0 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b30001760 con 0x7f5b38098bf0 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient(hunting): dump: 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.262 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2262411955 0 0) 0x7f5b3813e7a0 con 0x7f5b38098400 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.171:6789/0 conn(0x7f5b38098bf0 legacy=0x7f5b381d47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.148:6789/0 conn(0x7f5b380987e0 legacy=0x7f5b381d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: _finish_hunting 0 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 monclient: found mon.c 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b380a4b50 con 0x7f5b38098400 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: _finish_auth 0 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: _check_auth_tickets 2024-09-16T04:16:33.263 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 5 monclient: authenticate success, global_id 104141 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 10 monclient: _renew_subs 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3809ef10 con 0x7f5b38098400 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 10 monclient: _renew_subs 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b38140030 con 0x7f5b38098400 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b2c001c60 con 0x7f5b38098400 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b2c003a60 con 0x7f5b38098400 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b36ffd640 4 set_mon_vals no callback set 2024-09-16T04:16:33.264 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.242+0000 7f5b357fa640 10 monclient: dump: 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.265 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b3ebf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b180065c0 con 0x7f5b18002860 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5b2c003a60 con 0x7f5b38098400 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f5b2c047710 con 0x7f5b38098400 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901792619 0 0) 0x7f5b180065c0 con 0x7f5b18002860 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient(hunting): my global_id is 104144 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b3813e7a0 con 0x7f5b18002860 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b3d193640 10 monclient: get_auth_request con 0x7f5b1c039140 auth_method 0 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1933492459 0 0) 0x7f5b3813e7a0 con 0x7f5b18002860 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b180065c0 con 0x7f5b18002860 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b24004c80 con 0x7f5b18002860 2024-09-16T04:16:33.266 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 10 monclient: dump: 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1601756847 0 0) 0x7f5b180065c0 con 0x7f5b18002860 2024-09-16T04:16:33.267 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.243+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b1800a150 con 0x7f5b18002860 2024-09-16T04:16:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.244+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5b24006210 con 0x7f5b18002860 2024-09-16T04:16:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.244+0000 7f5b357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.244+0000 7f5b357fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.244+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.148:6790/0 conn(0x7f5b18002860 legacy=0x7f5b18004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.244+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.148:6790/0 conn(0x7f5b18002860 legacy=0x7f5b18004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b3ebf6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:33.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b18009e40 con 0x7f5b18009100 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140242846 0 0) 0x7f5b18009e40 con 0x7f5b18009100 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient(hunting): my global_id is 104147 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b180065c0 con 0x7f5b18009100 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3756192078 0 0) 0x7f5b180065c0 con 0x7f5b18009100 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b18009e40 con 0x7f5b18009100 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b240067d0 con 0x7f5b18009100 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient: dump: 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4096736080 0 0) 0x7f5b18009e40 con 0x7f5b18009100 2024-09-16T04:16:33.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b1800c3d0 con 0x7f5b18009100 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3975 (unknown 0 0 764361394) 0x7f5b240066a0 con 0x7f5b18009100 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.148:6790/0 conn(0x7f5b18009100 legacy=0x7f5b18009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.250+0000 7f5b357fa640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.148:6790/0 conn(0x7f5b18009100 legacy=0x7f5b18009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:16:33.272 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 44, 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 2686, 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.273 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:33.274 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.275 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:33.276 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.277 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.278 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.279 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:33.280 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:33.281 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:33.282 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.283 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:33.284 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.171:6812/7327273 conn(0x7f5b1c039140 legacy=0x7f5b1c03b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 10 monclient: shutdown 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 >> v1:172.21.15.148:6790/0 conn(0x7f5b38098400 legacy=0x7f5b38135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3d193640 1 -- 172.21.15.148:0/1661422137 reap_dead start 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3d193640 1 -- 172.21.15.148:0/1661422137 reap_dead start 2024-09-16T04:16:33.285 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 shutdown_connections 2024-09-16T04:16:33.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 >> 172.21.15.148:0/1661422137 conn(0x7f5b380a5860 msgr2=0x7f5b3809b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:33.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 shutdown_connections 2024-09-16T04:16:33.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.251+0000 7f5b3ebf6640 1 -- 172.21.15.148:0/1661422137 wait complete. 2024-09-16T04:16:33.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:16:33.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:16:33.286 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:16:33.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.383+0000 7f034a6af640 10 monclient: get_monmap_and_config 2024-09-16T04:16:33.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.383+0000 7f034a6af640 10 monclient: build_initial_monmap 2024-09-16T04:16:33.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.383+0000 7f034a6af640 10 monclient: monmap: 2024-09-16T04:16:33.385 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:33.385 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:33.384782+0000 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:33.384782+0000 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:33.386 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.384+0000 7f034a6af640 1 Processor -- start 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.384+0000 7f034a6af640 1 -- start start 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.384+0000 7f034a6af640 10 monclient: init 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient(hunting): picked mon.noname-b con 0x7f034409ed10 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient(hunting): picked mon.noname-c con 0x7f0344136d20 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient(hunting): picked mon.noname-a con 0x7f0344137190 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03440a1b60 con 0x7f0344137190 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034409cbf0 con 0x7f0344136d20 2024-09-16T04:16:33.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034409ced0 con 0x7f034409ed10 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient(hunting): authenticate will time out at 1414.010166s 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0343fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f034409ed10 0x7f034409f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:52668/0 (socket says 172.21.15.148:52668) 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0348c4c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f0344137190 0x7f034413f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60426/0 (socket says 172.21.15.148:60426) 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0343fff640 1 -- 172.21.15.148:0/2257782037 learned_addr learned my addr 172.21.15.148:0/2257782037 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235271149 0 0) 0x7f034409ced0 con 0x7f034409ed10 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): my global_id is 104150 2024-09-16T04:16:33.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f032c003520 con 0x7f034409ed10 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212047367 0 0) 0x7f03440a1b60 con 0x7f0344137190 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient(hunting): my global_id is 65211 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f034409ced0 con 0x7f0344137190 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3245879580 0 0) 0x7f032c003520 con 0x7f034409ed10 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 >> v1:172.21.15.171:6789/0 conn(0x7f0344136d20 legacy=0x7f034413bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 >> v1:172.21.15.148:6789/0 conn(0x7f0344137190 legacy=0x7f034413f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 monclient: found mon.noname-b 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:33.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034409d1b0 con 0x7f034409ed10 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: _finish_auth 0 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 5 monclient: authenticate success, global_id 104150 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0334001610 con 0x7f034409ed10 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: dump: 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.398 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0334001800 con 0x7f034409ed10 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 1 -- 172.21.15.148:0/2257782037 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0334001c10 con 0x7f034409ed10 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f034a6af640 4 set_mon_vals no callback set 2024-09-16T04:16:33.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.385+0000 7f0342ffd640 10 monclient: dump: 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.400 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 10 monclient: shutdown 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 1 -- 172.21.15.148:0/2257782037 >> v1:172.21.15.148:6790/0 conn(0x7f034409ed10 legacy=0x7f034409f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 1 -- 172.21.15.148:0/2257782037 shutdown_connections 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 1 -- 172.21.15.148:0/2257782037 >> 172.21.15.148:0/2257782037 conn(0x7f03440a5860 msgr2=0x7f03440a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 1 -- 172.21.15.148:0/2257782037 shutdown_connections 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 1 -- 172.21.15.148:0/2257782037 wait complete. 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.386+0000 7f034a6af640 10 monclient: build_initial_monmap 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient: monmap: 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:33.401 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 1 Processor -- start 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 1 -- start start 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient: init 2024-09-16T04:16:33.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient(hunting): picked mon.noname-a con 0x7f034409ed10 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient(hunting): picked mon.noname-c con 0x7f0344136d20 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient(hunting): picked mon.noname-b con 0x7f0344137190 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0344098530 con 0x7f034409ed10 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03440a3870 con 0x7f0344137190 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03440a34d0 con 0x7f0344136d20 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.387+0000 7f034a6af640 10 monclient(hunting): authenticate will time out at 1414.012980s 2024-09-16T04:16:33.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0343fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f034409ed10 0x7f0344097710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60436/0 (socket says 172.21.15.148:60436) 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0343fff640 1 -- 172.21.15.148:0/385395149 learned_addr learned my addr 172.21.15.148:0/385395149 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561541654 0 0) 0x7f0344098530 con 0x7f034409ed10 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): my global_id is 65214 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0330003520 con 0x7f034409ed10 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487608718 0 0) 0x7f03440a34d0 con 0x7f0344136d20 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): my global_id is 104153 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0344098530 con 0x7f0344136d20 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265105063 0 0) 0x7f03440a3870 con 0x7f0344137190 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): my global_id is 6754 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03440a34d0 con 0x7f0344137190 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3489769166 0 0) 0x7f0330003520 con 0x7f034409ed10 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03440a3870 con 0x7f034409ed10 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0334001c10 con 0x7f034409ed10 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:16:33.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): dump: 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 479483381 0 0) 0x7f0344098530 con 0x7f0344136d20 2024-09-16T04:16:33.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0330003520 con 0x7f0344136d20 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0338001830 con 0x7f0344136d20 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): dump: 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.407 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3813916572 0 0) 0x7f03440a34d0 con 0x7f0344137190 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0344098530 con 0x7f0344137190 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f033c001470 con 0x7f0344137190 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient(hunting): dump: 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.408 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 62748771 0 0) 0x7f03440a3870 con 0x7f034409ed10 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.148:6790/0 conn(0x7f0344136d20 legacy=0x7f0344097e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.171:6789/0 conn(0x7f0344137190 legacy=0x7f03441d3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 monclient: found mon.a 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034409c630 con 0x7f034409ed10 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: _finish_auth 0 2024-09-16T04:16:33.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 5 monclient: authenticate success, global_id 65214 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 10 monclient: _renew_subs 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f034408cb60 con 0x7f034409ed10 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 10 monclient: _renew_subs 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f034408d820 con 0x7f034409ed10 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0334001800 con 0x7f034409ed10 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0334004de0 con 0x7f034409ed10 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f03427fc640 4 set_mon_vals no callback set 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.388+0000 7f0340ff9640 10 monclient: dump: 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.389+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f0334004de0 con 0x7f034409ed10 2024-09-16T04:16:33.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.389+0000 7f034a6af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.389+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f03240065c0 con 0x7f0324002860 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.389+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f0334047900 con 0x7f034409ed10 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.389+0000 7f0348c4c640 10 monclient: get_auth_request con 0x7f0330039170 auth_method 0 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444781074 0 0) 0x7f03240065c0 con 0x7f0324002860 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient(hunting): my global_id is 6757 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03440a3870 con 0x7f0324002860 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2192902105 0 0) 0x7f03440a3870 con 0x7f0324002860 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03240065c0 con 0x7f0324002860 2024-09-16T04:16:33.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0338003980 con 0x7f0324002860 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient: dump: 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.413 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1504699399 0 0) 0x7f03240065c0 con 0x7f0324002860 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f032400a150 con 0x7f0324002860 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0338003bf0 con 0x7f0324002860 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.171:6789/0 conn(0x7f0324002860 legacy=0x7f0324004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.390+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.171:6789/0 conn(0x7f0324002860 legacy=0x7f0324004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:33.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.396+0000 7f034a6af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:33.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.396+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0324009e20 con 0x7f03240090e0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062225913 0 0) 0x7f0324009e20 con 0x7f03240090e0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient(hunting): my global_id is 6760 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03240065c0 con 0x7f03240090e0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2205121870 0 0) 0x7f03240065c0 con 0x7f03240090e0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0324009e20 con 0x7f03240090e0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f03380041b0 con 0x7f03240090e0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 10 monclient: dump: 2024-09-16T04:16:33.416 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4076822182 0 0) 0x7f0324009e20 con 0x7f03240090e0 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.397+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f032400c390 con 0x7f03240090e0 2024-09-16T04:16:33.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 2129001231) 0x7f0338005400 con 0x7f03240090e0 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0340ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0340ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.171:6789/0 conn(0x7f03240090e0 legacy=0x7f0324009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0340ff9640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.171:6789/0 conn(0x7f03240090e0 legacy=0x7f0324009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 44, 2024-09-16T04:16:33.418 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 617436, 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.419 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.420 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.421 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:33.422 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:33.423 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:33.424 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.425 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.426 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.427 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.428 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.429 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:33.430 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.171:6812/7327273 conn(0x7f0330039170 legacy=0x7f033003b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 10 monclient: shutdown 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 >> v1:172.21.15.148:6789/0 conn(0x7f034409ed10 legacy=0x7f0344097710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0348c4c640 1 -- 172.21.15.148:0/385395149 reap_dead start 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f0348c4c640 1 -- 172.21.15.148:0/385395149 reap_dead start 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 shutdown_connections 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 >> 172.21.15.148:0/385395149 conn(0x7f03440a5860 msgr2=0x7f03440a1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 shutdown_connections 2024-09-16T04:16:33.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:33.398+0000 7f034a6af640 1 -- 172.21.15.148:0/385395149 wait complete. 2024-09-16T04:16:33.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:16:33.433 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:16:33.433 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T04:16:33.433 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:16:33.433 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:16:33.433 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:16:33.433 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:16:33.446 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:16:33.446 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T04:16:33.446 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T04:16:33.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:16:48.447 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T04:16:48.448 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T04:16:48.448 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:16:48.449 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:16:48.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.554+0000 7f33bb6d2640 10 monclient: get_monmap_and_config 2024-09-16T04:16:48.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.554+0000 7f33bb6d2640 10 monclient: build_initial_monmap 2024-09-16T04:16:48.557 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.554+0000 7f33bb6d2640 10 monclient: monmap: 2024-09-16T04:16:48.557 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:48.557 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:48.556454+0000 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:48.556454+0000 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:48.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 1 Processor -- start 2024-09-16T04:16:48.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 1 -- start start 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient: init 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient(hunting): picked mon.noname-a con 0x7f33b40a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient(hunting): picked mon.noname-b con 0x7f33b409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient(hunting): picked mon.noname-c con 0x7f33b409b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b409e040 con 0x7f33b40a0e90 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b409d660 con 0x7f33b409b5c0 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b409c0d0 con 0x7f33b409b170 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33bb6d2640 10 monclient(hunting): authenticate will time out at 1429.181841s 2024-09-16T04:16:48.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33b946e640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33b409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:48.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33b8c6d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f33b409b170 0x7f33b41383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39264/0 (socket says 172.21.15.148:39264) 2024-09-16T04:16:48.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.555+0000 7f33b8c6d640 1 -- 172.21.15.148:0/1796723921 learned_addr learned my addr 172.21.15.148:0/1796723921 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:48.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:48.757+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33b409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:49.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:49.158+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33b409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:49.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:49.959+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33b409aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:51.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.556+0000 7f33aaffd640 10 monclient(hunting): tick 2024-09-16T04:16:51.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.556+0000 7f33aaffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:16:51.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.556+0000 7f33aaffd640 1 monclient(hunting): continuing hunt 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.556+0000 7f33aaffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.556+0000 7f33aaffd640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6790/0 conn(0x7f33b409b170 legacy=0x7f33b41383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.171:6789/0 conn(0x7f33b409b5c0 legacy=0x7f33b413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33b409aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 10 monclient(hunting): picked mon.noname-a con 0x7f33b40a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 10 monclient(hunting): picked mon.noname-b con 0x7f33b409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 10 monclient(hunting): picked mon.noname-c con 0x7f33b409b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 1 -- 172.21.15.148:0/1796723921 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b409d380 con 0x7f33b40a0e90 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 1 -- 172.21.15.148:0/1796723921 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f339800edc0 con 0x7f33b409b5c0 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 1 -- 172.21.15.148:0/1796723921 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3398010010 con 0x7f33b409b170 2024-09-16T04:16:51.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33980056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:51.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.557+0000 7f33aaffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:51.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:51.758+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33980056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:52.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:52.159+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33980056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:52.960+0000 7f33b946e640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33980056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1344905185 0 0) 0x7f3398010010 con 0x7f33b409b170 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient(hunting): my global_id is 104156 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33a0005730 con 0x7f33b409b170 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 123181745 0 0) 0x7f33a0005730 con 0x7f33b409b170 2024-09-16T04:16:53.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.171:6789/0 conn(0x7f33b409b5c0 legacy=0x7f339800c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6789/0 conn(0x7f33b40a0e90 legacy=0x7f33980056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: _finish_hunting 0 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 monclient: found mon.noname-b 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33980123d0 con 0x7f33b409b170 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: _finish_auth 0 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: _check_auth_tickets 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33a4001550 con 0x7f33b409b170 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:53.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: dump: 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33bb6d2640 5 monclient: authenticate success, global_id 104156 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33bb6d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33a4001830 con 0x7f33b409b170 2024-09-16T04:16:53.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33bb6d2640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 1 -- 172.21.15.148:0/1796723921 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33a4001c40 con 0x7f33b409b170 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33bb6d2640 4 set_mon_vals no callback set 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.452+0000 7f33abfff640 10 monclient: dump: 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.458 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 10 monclient: shutdown 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 1 -- 172.21.15.148:0/1796723921 >> v1:172.21.15.148:6790/0 conn(0x7f33b409b170 legacy=0x7f3398008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 1 -- 172.21.15.148:0/1796723921 shutdown_connections 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 1 -- 172.21.15.148:0/1796723921 >> 172.21.15.148:0/1796723921 conn(0x7f33b40a5640 msgr2=0x7f33b40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 1 -- 172.21.15.148:0/1796723921 shutdown_connections 2024-09-16T04:16:53.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.453+0000 7f33bb6d2640 1 -- 172.21.15.148:0/1796723921 wait complete. 2024-09-16T04:16:53.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.454+0000 7f33bb6d2640 10 monclient: build_initial_monmap 2024-09-16T04:16:53.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.454+0000 7f33bb6d2640 10 monclient: monmap: 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.454+0000 7f33bb6d2640 1 Processor -- start 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 1 -- start start 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient: init 2024-09-16T04:16:53.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient(hunting): picked mon.noname-b con 0x7f33b409b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient(hunting): picked mon.noname-a con 0x7f33b409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient(hunting): picked mon.noname-c con 0x7f33b40a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b41c5960 con 0x7f33b409b5c0 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b41aa0f0 con 0x7f33b409b170 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33b8c6d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f33b409b5c0 legacy=0x7f33b41d0580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b41acbb0 con 0x7f33b40a0e90 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33bb6d2640 10 monclient(hunting): authenticate will time out at 1434.080712s 2024-09-16T04:16:53.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33b9c6f640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f33b40a0e90 0x7f33b41d3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59498/0 (socket says 172.21.15.148:59498) 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33b946e640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f33b409b170 0x7f33b41c5250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:53874/0 (socket says 172.21.15.148:53874) 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.455+0000 7f33b9c6f640 1 -- 172.21.15.148:0/4200290038 learned_addr learned my addr 172.21.15.148:0/4200290038 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.460+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912915600 0 0) 0x7f33b41aa0f0 con 0x7f33b409b170 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.460+0000 7f33a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.460+0000 7f33a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.460+0000 7f33a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.460+0000 7f33a9ffb640 10 monclient(hunting): my global_id is 6763 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.460+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33980086b0 con 0x7f33b409b170 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723728205 0 0) 0x7f33b41acbb0 con 0x7f33b40a0e90 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): my global_id is 104159 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b41aa0f0 con 0x7f33b40a0e90 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4160788614 0 0) 0x7f33980086b0 con 0x7f33b409b170 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33b41acbb0 con 0x7f33b409b170 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33b00042f0 con 0x7f33b409b170 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): dump: 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.469 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 285807833 0 0) 0x7f33b41aa0f0 con 0x7f33b40a0e90 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f33980086b0 con 0x7f33b40a0e90 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33ac001ad0 con 0x7f33b40a0e90 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient(hunting): dump: 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.470 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.471 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3257043915 0 0) 0x7f33b41acbb0 con 0x7f33b409b170 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 >> v1:172.21.15.148:6790/0 conn(0x7f33b40a0e90 legacy=0x7f33b41d3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 >> v1:172.21.15.148:6789/0 conn(0x7f33b409b5c0 legacy=0x7f33b41d0580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 monclient: found mon.b 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:53.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b41aa8e0 con 0x7f33b409b170 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33bb6d2640 5 monclient: authenticate success, global_id 6763 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33bb6d2640 10 monclient: _renew_subs 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33bb6d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f33b41ac8d0 con 0x7f33b409b170 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f33b0004800 con 0x7f33b409b170 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f33b0004b50 con 0x7f33b409b170 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33a9ffb640 10 monclient: dump: 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.461+0000 7f33aaffd640 4 set_mon_vals no callback set 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.462+0000 7f33bb6d2640 10 monclient: _renew_subs 2024-09-16T04:16:53.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.462+0000 7f33bb6d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.462+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f33b41c7cc0 con 0x7f33b409b170 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.462+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f33b0005d20 con 0x7f33b409b170 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.462+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f33b0047a30 con 0x7f33b409b170 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.463+0000 7f33b8c6d640 10 monclient: get_auth_request con 0x7f3398039610 auth_method 0 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.463+0000 7f33bb6d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.463+0000 7f33bb6d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.463+0000 7f33bb6d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:53.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.463+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b40740a0 con 0x7f33b409b170 2024-09-16T04:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.468+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f33b0015c30 con 0x7f33b409b170 2024-09-16T04:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.468+0000 7f33a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.468+0000 7f33a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.702+0000 7f33bb6d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:16:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.702+0000 7f33bb6d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:16:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.702+0000 7f33bb6d2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:16:53.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.702+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f33b406bc20 con 0x7f33b409b170 2024-09-16T04:16:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.703+0000 7f33a9ffb640 1 -- 172.21.15.148:0/4200290038 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1811339209) 0x7f33b0015530 con 0x7f33b409b170 2024-09-16T04:16:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.703+0000 7f33a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:16:53.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.703+0000 7f33a9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:53.705 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":46,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 >> v1:172.21.15.171:6812/7327273 conn(0x7f3398039610 legacy=0x7f339803bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 10 monclient: shutdown 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 >> v1:172.21.15.171:6789/0 conn(0x7f33b409b170 legacy=0x7f33b41c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 shutdown_connections 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 >> 172.21.15.148:0/4200290038 conn(0x7f33b40a5640 msgr2=0x7f33b4070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 shutdown_connections 2024-09-16T04:16:53.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.705+0000 7f33bb6d2640 1 -- 172.21.15.148:0/4200290038 wait complete. 2024-09-16T04:16:53.718 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:16:53.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:16:53.719 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:16:53.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.814+0000 7f8889d5b640 10 monclient: get_monmap_and_config 2024-09-16T04:16:53.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.814+0000 7f8889d5b640 10 monclient: build_initial_monmap 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.814+0000 7f8889d5b640 10 monclient: monmap: 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:53.815713+0000 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:53.815713+0000 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:53.817 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.815+0000 7f8889d5b640 1 Processor -- start 2024-09-16T04:16:53.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.815+0000 7f8889d5b640 1 -- start start 2024-09-16T04:16:53.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.815+0000 7f8889d5b640 10 monclient: init 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient(hunting): picked mon.noname-b con 0x7f88840a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient(hunting): picked mon.noname-c con 0x7f888409b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient(hunting): picked mon.noname-a con 0x7f888409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888409e040 con 0x7f888409b5c0 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888409d660 con 0x7f888409b170 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888409c0d0 con 0x7f88840a0e90 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient(hunting): authenticate will time out at 1434.441202s 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8883fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f888409b5c0 legacy=0x7f888413bb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:53.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f88837fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88840a0e90 0x7f888409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59504/0 (socket says 172.21.15.148:59504) 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f88837fe640 1 -- 172.21.15.148:0/3194985741 learned_addr learned my addr 172.21.15.148:0/3194985741 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2763027102 0 0) 0x7f888409c0d0 con 0x7f88840a0e90 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient(hunting): my global_id is 104162 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8870003520 con 0x7f88840a0e90 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3190831185 0 0) 0x7f8870003520 con 0x7f88840a0e90 2024-09-16T04:16:53.827 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 >> v1:172.21.15.171:6789/0 conn(0x7f888409b170 legacy=0x7f88841383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 >> v1:172.21.15.148:6789/0 conn(0x7f888409b5c0 legacy=0x7f888413bb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 monclient: found mon.noname-b 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888409d380 con 0x7f88840a0e90 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: _finish_auth 0 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8874001610 con 0x7f88840a0e90 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:53.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: dump: 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 5 monclient: authenticate success, global_id 104162 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:53.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8874001900 con 0x7f88840a0e90 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 1 -- 172.21.15.148:0/3194985741 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8874001cd0 con 0x7f88840a0e90 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8889d5b640 4 set_mon_vals no callback set 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.816+0000 7f8882ffd640 10 monclient: dump: 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.830 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 10 monclient: shutdown 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 1 -- 172.21.15.148:0/3194985741 >> v1:172.21.15.148:6790/0 conn(0x7f88840a0e90 legacy=0x7f888409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 1 -- 172.21.15.148:0/3194985741 shutdown_connections 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 1 -- 172.21.15.148:0/3194985741 >> 172.21.15.148:0/3194985741 conn(0x7f88840a5640 msgr2=0x7f88840a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 1 -- 172.21.15.148:0/3194985741 shutdown_connections 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.817+0000 7f8889d5b640 1 -- 172.21.15.148:0/3194985741 wait complete. 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient: build_initial_monmap 2024-09-16T04:16:53.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient: monmap: 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:53.832 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 1 Processor -- start 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 1 -- start start 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient: init 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient(hunting): picked mon.noname-c con 0x7f888409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient(hunting): picked mon.noname-a con 0x7f888409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:53.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient(hunting): picked mon.noname-b con 0x7f88840a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88841acea0 con 0x7f888409b5c0 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88841a6770 con 0x7f88840a0e90 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88841cc390 con 0x7f888409b170 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f887bfff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f888409b5c0 legacy=0x7f88841d07f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f8889d5b640 10 monclient(hunting): authenticate will time out at 1434.443866s 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f88837fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f888409b170 0x7f88841cbc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59510/0 (socket says 172.21.15.148:59510) 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.818+0000 7f88837fe640 1 -- 172.21.15.148:0/4230598655 learned_addr learned my addr 172.21.15.148:0/4230598655 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372489984 0 0) 0x7f88841cc390 con 0x7f888409b170 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): my global_id is 104165 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8864003520 con 0x7f888409b170 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352551729 0 0) 0x7f88841a6770 con 0x7f88840a0e90 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): my global_id is 6769 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88841cc390 con 0x7f88840a0e90 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1769736107 0 0) 0x7f8864003520 con 0x7f888409b170 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88841a6770 con 0x7f888409b170 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88740015d0 con 0x7f888409b170 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:53.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): dump: 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:53.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1340633895 0 0) 0x7f88841a6770 con 0x7f888409b170 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.171:6789/0 conn(0x7f88840a0e90 legacy=0x7f88841d3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.148:6789/0 conn(0x7f888409b5c0 legacy=0x7f88841d07f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 monclient: found mon.c 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88841ac590 con 0x7f888409b170 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: _finish_auth 0 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:53.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 5 monclient: authenticate success, global_id 104165 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 10 monclient: _renew_subs 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88841c5b20 con 0x7f888409b170 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 10 monclient: _renew_subs 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f88841ab0a0 con 0x7f888409b170 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8874001900 con 0x7f888409b170 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8874004cc0 con 0x7f888409b170 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f88827fc640 4 set_mon_vals no callback set 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.819+0000 7f8880ff9640 10 monclient: dump: 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8889d5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f88740100f0 con 0x7f888409b170 2024-09-16T04:16:53.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88680065c0 con 0x7f8868002860 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f8874047b60 con 0x7f888409b170 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601856559 0 0) 0x7f88680065c0 con 0x7f8868002860 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient(hunting): my global_id is 104168 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88841a6770 con 0x7f8868002860 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8883fff640 10 monclient: get_auth_request con 0x7f8864038f60 auth_method 0 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1388860633 0 0) 0x7f88841a6770 con 0x7f8868002860 2024-09-16T04:16:53.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88680065c0 con 0x7f8868002860 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f886c0016f0 con 0x7f8868002860 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.820+0000 7f8880ff9640 10 monclient: dump: 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.841 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2399552614 0 0) 0x7f88680065c0 con 0x7f8868002860 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f886800a150 con 0x7f8868002860 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f886c001a50 con 0x7f8868002860 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.148:6790/0 conn(0x7f8868002860 legacy=0x7f8868004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.821+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.148:6790/0 conn(0x7f8868002860 legacy=0x7f8868004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:53.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8889d5b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:53.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8868009e40 con 0x7f8868009100 2024-09-16T04:16:53.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738587267 0 0) 0x7f8868009e40 con 0x7f8868009100 2024-09-16T04:16:53.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient(hunting): my global_id is 104171 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88680065c0 con 0x7f8868009100 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2377506447 0 0) 0x7f88680065c0 con 0x7f8868009100 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8868009e40 con 0x7f8868009100 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f886c0035f0 con 0x7f8868009100 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 10 monclient: dump: 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.845 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 462732054 0 0) 0x7f8868009e40 con 0x7f8868009100 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.827+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f886800c3d0 con 0x7f8868009100 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4132 (unknown 0 0 1997374026) 0x7f886c001cf0 con 0x7f8868009100 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8880ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8880ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.148:6790/0 conn(0x7f8868009100 legacy=0x7f8868009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8880ff9640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.148:6790/0 conn(0x7f8868009100 legacy=0x7f8868009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 46, 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 23265, 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:53.847 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:53.848 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:53.849 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:53.850 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:53.851 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.852 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:53.853 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:53.854 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:53.855 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:16:53.856 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.857 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:53.858 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.171:6812/7327273 conn(0x7f8864038f60 legacy=0x7f886403b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 10 monclient: shutdown 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 >> v1:172.21.15.148:6790/0 conn(0x7f888409b170 legacy=0x7f88841cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8883fff640 1 -- 172.21.15.148:0/4230598655 reap_dead start 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8883fff640 1 -- 172.21.15.148:0/4230598655 reap_dead start 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 shutdown_connections 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 >> 172.21.15.148:0/4230598655 conn(0x7f88840a5640 msgr2=0x7f88840a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 shutdown_connections 2024-09-16T04:16:53.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.828+0000 7f8889d5b640 1 -- 172.21.15.148:0/4230598655 wait complete. 2024-09-16T04:16:53.860 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:16:53.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:16:53.860 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:16:53.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.957+0000 7f97d09fd640 10 monclient: get_monmap_and_config 2024-09-16T04:16:53.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.957+0000 7f97d09fd640 10 monclient: build_initial_monmap 2024-09-16T04:16:53.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.957+0000 7f97d09fd640 10 monclient: monmap: 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:16:53.958545+0000 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:16:53.958545+0000 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 1 Processor -- start 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 1 -- start start 2024-09-16T04:16:53.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 10 monclient: init 2024-09-16T04:16:53.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:53.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:16:53.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 10 monclient(hunting): picked mon.noname-c con 0x7f97cc0a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:53.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 10 monclient(hunting): picked mon.noname-a con 0x7f97cc09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:53.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 10 monclient(hunting): picked mon.noname-b con 0x7f97cc09b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97d09fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc09e040 con 0x7f97cc09b170 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.958+0000 7f97c9d9b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f97cc09b170 legacy=0x7f97cc1383f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc09d660 con 0x7f97cc0a0e90 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc09c0d0 con 0x7f97cc09b5c0 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 10 monclient(hunting): authenticate will time out at 1434.584095s 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97cad9d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f97cc09b5c0 0x7f97cc13bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59548/0 (socket says 172.21.15.148:59548) 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97cad9d640 1 -- 172.21.15.148:0/314874799 learned_addr learned my addr 172.21.15.148:0/314874799 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102006247 0 0) 0x7f97cc09c0d0 con 0x7f97cc09b5c0 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): my global_id is 104174 2024-09-16T04:16:53.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b4003520 con 0x7f97cc09b5c0 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591666199 0 0) 0x7f97bc001110 con 0x7f97cc0a0e90 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient(hunting): my global_id is 6772 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97cc09c0d0 con 0x7f97cc0a0e90 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3613621274 0 0) 0x7f97b4003520 con 0x7f97cc09b5c0 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 >> v1:172.21.15.171:6789/0 conn(0x7f97cc0a0e90 legacy=0x7f97cc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 >> v1:172.21.15.148:6789/0 conn(0x7f97cc09b170 legacy=0x7f97cc1383f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: _finish_hunting 0 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 monclient: found mon.noname-b 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:53.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc09d380 con 0x7f97cc09b5c0 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: _finish_auth 0 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: _check_auth_tickets 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97c4001990 con 0x7f97cc09b5c0 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: dump: 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.971 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 5 monclient: authenticate success, global_id 104174 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97c4001bf0 con 0x7f97cc09b5c0 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 1 -- 172.21.15.148:0/314874799 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97c4001f40 con 0x7f97cc09b5c0 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 10 monclient: get_monmap_and_config success 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97d09fd640 4 set_mon_vals no callback set 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.959+0000 7f97c959a640 10 monclient: dump: 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 10 monclient: shutdown 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 1 -- 172.21.15.148:0/314874799 >> v1:172.21.15.148:6790/0 conn(0x7f97cc09b5c0 legacy=0x7f97cc13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 1 -- 172.21.15.148:0/314874799 shutdown_connections 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 1 -- 172.21.15.148:0/314874799 >> 172.21.15.148:0/314874799 conn(0x7f97cc0a5640 msgr2=0x7f97cc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 1 -- 172.21.15.148:0/314874799 shutdown_connections 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.960+0000 7f97d09fd640 1 -- 172.21.15.148:0/314874799 wait complete. 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient: build_initial_monmap 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient: monmap: 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:16:53.974 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 1 Processor -- start 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 1 -- start start 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient: init 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient: _reopen_session rank -1 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient(hunting): picked mon.noname-c con 0x7f97cc09b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient(hunting): picked mon.noname-b con 0x7f97cc09b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient(hunting): picked mon.noname-a con 0x7f97cc0a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc1c1c20 con 0x7f97cc0a0e90 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc1a6400 con 0x7f97cc09b5c0 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97cad9d640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f97cc0a0e90 legacy=0x7f97cc1cff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:16:53.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97cc1a8e70 con 0x7f97cc09b170 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient(hunting): _renew_subs 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97ca59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f97cc09b170 0x7f97cc1c1510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59550/0 (socket says 172.21.15.148:59550) 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97ca59c640 1 -- 172.21.15.148:0/796752557 learned_addr learned my addr 172.21.15.148:0/796752557 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.961+0000 7f97d09fd640 10 monclient(hunting): authenticate will time out at 1434.586971s 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343387618 0 0) 0x7f97cc1a8e70 con 0x7f97cc09b170 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): my global_id is 104177 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97ac003520 con 0x7f97cc09b170 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878787929 0 0) 0x7f97cc1a6400 con 0x7f97cc09b5c0 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): my global_id is 6775 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97cc1a8e70 con 0x7f97cc09b5c0 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4045114280 0 0) 0x7f97ac003520 con 0x7f97cc09b170 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97cc1a6400 con 0x7f97cc09b170 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97bc001a00 con 0x7f97cc09b170 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): dump: 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.977 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 58277218 0 0) 0x7f97cc1a6400 con 0x7f97cc09b170 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.171:6789/0 conn(0x7f97cc09b5c0 legacy=0x7f97cc1cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.148:6789/0 conn(0x7f97cc0a0e90 legacy=0x7f97cc1cff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: _finish_hunting 0 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 monclient: found mon.c 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc1a6ba0 con 0x7f97cc09b170 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: _finish_auth 0 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: _check_auth_tickets 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 5 monclient: authenticate success, global_id 104177 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 10 monclient: _renew_subs 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc1a8b90 con 0x7f97cc09b170 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 10 monclient: _renew_subs 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f97cc1c3f80 con 0x7f97cc09b170 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f97bc001ca0 con 0x7f97cc09b170 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97bc003a60 con 0x7f97cc09b170 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97baffd640 10 monclient: dump: 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.962+0000 7f97c8d99640 4 set_mon_vals no callback set 2024-09-16T04:16:53.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97d09fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f97bc003c00 con 0x7f97cc09b170 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f97b00065c0 con 0x7f97b0002860 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f97bc0479f0 con 0x7f97cc09b170 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97cad9d640 10 monclient: get_auth_request con 0x7f97ac038be0 auth_method 0 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002525699 0 0) 0x7f97b00065c0 con 0x7f97b0002860 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 10 monclient(hunting): my global_id is 6778 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.963+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97cc1a6400 con 0x7f97b0002860 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2455411167 0 0) 0x7f97cc1a6400 con 0x7f97b0002860 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97b00065c0 con 0x7f97b0002860 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97c00047c0 con 0x7f97b0002860 2024-09-16T04:16:53.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 10 monclient: dump: 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2784582748 0 0) 0x7f97b00065c0 con 0x7f97b0002860 2024-09-16T04:16:53.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97b000a150 con 0x7f97b0002860 2024-09-16T04:16:53.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f97c0006b00 con 0x7f97b0002860 2024-09-16T04:16:53.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:16:53.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:16:53.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.171:6789/0 conn(0x7f97b0002860 legacy=0x7f97b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.964+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.171:6789/0 conn(0x7f97b0002860 legacy=0x7f97b0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:53.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.970+0000 7f97d09fd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.970+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f97b0009e20 con 0x7f97b00090e0 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 478406790 0 0) 0x7f97b0009e20 con 0x7f97b00090e0 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient(hunting): my global_id is 6781 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b00065c0 con 0x7f97b00090e0 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 914733739 0 0) 0x7f97b00065c0 con 0x7f97b00090e0 2024-09-16T04:16:53.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f97b0009e20 con 0x7f97b00090e0 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f97c0004e40 con 0x7f97b00090e0 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient: dump: 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:16:53.986 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3853725940 0 0) 0x7f97b0009e20 con 0x7f97b00090e0 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97b000c390 con 0x7f97b00090e0 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 270068008) 0x7f97c00061a0 con 0x7f97b00090e0 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.171:6789/0 conn(0x7f97b00090e0 legacy=0x7f97b0009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.971+0000 7f97baffd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.171:6789/0 conn(0x7f97b00090e0 legacy=0x7f97b0009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:16:53.987 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 46, 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 638009, 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:16:53.988 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:53.989 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:53.990 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:16:53.991 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:16:53.992 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:16:53.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:16:53.994 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:16:53.995 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:53.996 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:53.998 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:53.999 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:16:54.000 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.171:6812/7327273 conn(0x7f97ac038be0 legacy=0x7f97ac03b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 10 monclient: shutdown 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 >> v1:172.21.15.148:6790/0 conn(0x7f97cc09b170 legacy=0x7f97cc1c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97cad9d640 1 -- 172.21.15.148:0/796752557 reap_dead start 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97cad9d640 1 -- 172.21.15.148:0/796752557 reap_dead start 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 shutdown_connections 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 >> 172.21.15.148:0/796752557 conn(0x7f97cc0a5640 msgr2=0x7f97cc071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:16:54.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 shutdown_connections 2024-09-16T04:16:54.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:16:53.972+0000 7f97d09fd640 1 -- 172.21.15.148:0/796752557 wait complete. 2024-09-16T04:16:54.003 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:16:54.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:18:24.004 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:18:24.005 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:18:24.005 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:18:24.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:18:24.009 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:18:24.009 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T04:18:24.010 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T04:18:24.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:18:39.011 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T04:18:39.012 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T04:18:39.012 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:18:39.012 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.128+0000 7f14f5477640 10 monclient: get_monmap_and_config 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.128+0000 7f14f5477640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.128+0000 7f14f5477640 10 monclient: monmap: 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:39.129695+0000 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:39.129695+0000 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.141 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 1 Processor -- start 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 1 -- start start 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 10 monclient: init 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 10 monclient(hunting): picked mon.noname-c con 0x7f14f0133910 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.129+0000 7f14f5477640 10 monclient(hunting): picked mon.noname-a con 0x7f14f006c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 10 monclient(hunting): picked mon.noname-b con 0x7f14f013d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f009b0d0 con 0x7f14f006c430 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f009b340 con 0x7f14f0133910 2024-09-16T04:18:39.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f009b620 con 0x7f14f013d0e0 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 10 monclient(hunting): authenticate will time out at 1539.755067s 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14e7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f14f006c430 0x7f14f013b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40880/0 (socket says 172.21.15.148:40880) 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14e7fff640 1 -- 172.21.15.148:0/2859308689 learned_addr learned my addr 172.21.15.148:0/2859308689 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ef7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f14f013d0e0 0x7f14f013f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49152/0 (socket says 172.21.15.148:49152) 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129783435 0 0) 0x7f14f009b0d0 con 0x7f14f006c430 2024-09-16T04:18:39.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): my global_id is 114099 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14dc003520 con 0x7f14f006c430 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486356677 0 0) 0x7f14f009b620 con 0x7f14f013d0e0 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient(hunting): my global_id is 104180 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14f009b0d0 con 0x7f14f013d0e0 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2304617031 0 0) 0x7f14f009b0d0 con 0x7f14f013d0e0 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 >> v1:172.21.15.171:6789/0 conn(0x7f14f0133910 legacy=0x7f14f0133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 >> v1:172.21.15.148:6789/0 conn(0x7f14f006c430 legacy=0x7f14f013b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 monclient: found mon.noname-b 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f00633c0 con 0x7f14f013d0e0 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f14e8001990 con 0x7f14f013d0e0 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 5 monclient: authenticate success, global_id 104180 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:18:39.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: dump: 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14e8001bf0 con 0x7f14f013d0e0 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 1 -- 172.21.15.148:0/2859308689 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f14e8002000 con 0x7f14f013d0e0 2024-09-16T04:18:39.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14f5477640 4 set_mon_vals no callback set 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.130+0000 7f14ee7fc640 10 monclient: dump: 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.147 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 10 monclient: shutdown 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 1 -- 172.21.15.148:0/2859308689 >> v1:172.21.15.148:6790/0 conn(0x7f14f013d0e0 legacy=0x7f14f013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 1 -- 172.21.15.148:0/2859308689 shutdown_connections 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 1 -- 172.21.15.148:0/2859308689 >> 172.21.15.148:0/2859308689 conn(0x7f14f0062630 msgr2=0x7f14f0062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 1 -- 172.21.15.148:0/2859308689 shutdown_connections 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 1 -- 172.21.15.148:0/2859308689 wait complete. 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.131+0000 7f14f5477640 10 monclient: monmap: 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:39.148 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 1 Processor -- start 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 1 -- start start 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient: init 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient(hunting): picked mon.noname-b con 0x7f14f006c430 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient(hunting): picked mon.noname-c con 0x7f14f0133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient(hunting): picked mon.noname-a con 0x7f14f013d0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f0138250 con 0x7f14f013d0e0 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f013b370 con 0x7f14f006c430 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f00a4dc0 con 0x7f14f0133910 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14f5477640 10 monclient(hunting): authenticate will time out at 1539.757802s 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14e7fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f14f0133910 0x7f14f0137b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49164/0 (socket says 172.21.15.148:49164) 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.132+0000 7f14e7fff640 1 -- 172.21.15.148:0/4021857256 learned_addr learned my addr 172.21.15.148:0/4021857256 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 717474150 0 0) 0x7f14f00a4dc0 con 0x7f14f0133910 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): my global_id is 104183 2024-09-16T04:18:39.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14d0003520 con 0x7f14f0133910 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642610867 0 0) 0x7f14f0138250 con 0x7f14f013d0e0 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): my global_id is 114102 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14f00a4dc0 con 0x7f14f013d0e0 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057023677 0 0) 0x7f14f013b370 con 0x7f14f006c430 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): my global_id is 6787 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14f0138250 con 0x7f14f006c430 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 976128250 0 0) 0x7f14d0003520 con 0x7f14f0133910 2024-09-16T04:18:39.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14f013b370 con 0x7f14f0133910 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f14d8001360 con 0x7f14f0133910 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): dump: 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.152 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4063407214 0 0) 0x7f14f00a4dc0 con 0x7f14f013d0e0 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f14d0003520 con 0x7f14f013d0e0 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f14e8001f40 con 0x7f14f013d0e0 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient(hunting): dump: 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.153 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3349494560 0 0) 0x7f14f013b370 con 0x7f14f0133910 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 >> v1:172.21.15.171:6789/0 conn(0x7f14f006c430 legacy=0x7f14f0137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 >> v1:172.21.15.148:6789/0 conn(0x7f14f013d0e0 legacy=0x7f14f01d3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 monclient: found mon.c 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f00a23c0 con 0x7f14f0133910 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 5 monclient: authenticate success, global_id 104183 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 10 monclient: _renew_subs 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f14f013adb0 con 0x7f14f0133910 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 10 monclient: _renew_subs 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f14f0139960 con 0x7f14f0133910 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f14d8001650 con 0x7f14f0133910 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f14d8004c30 con 0x7f14f0133910 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14edffb640 4 set_mon_vals no callback set 2024-09-16T04:18:39.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.133+0000 7f14e77fe640 10 monclient: dump: 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f14d8010060 con 0x7f14f0133910 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14f5477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14f5477640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14f5477640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d4005080 con 0x7f14f0133910 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f14d8047a90 con 0x7f14f0133910 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.134+0000 7f14eeffd640 10 monclient: get_auth_request con 0x7f14d0039110 auth_method 0 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.139+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f14d8015c90 con 0x7f14f0133910 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.139+0000 7f14e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.139+0000 7f14e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.370+0000 7f14f5477640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:18:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.370+0000 7f14f5477640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:18:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.371+0000 7f14f5477640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.371+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f14d4005370 con 0x7f14f0133910 2024-09-16T04:18:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.371+0000 7f14e77fe640 1 -- 172.21.15.148:0/4021857256 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3470239608) 0x7f14d8015590 con 0x7f14f0133910 2024-09-16T04:18:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.371+0000 7f14e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:18:39.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.371+0000 7f14e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:39.374 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 >> v1:172.21.15.171:6812/7327273 conn(0x7f14d0039110 legacy=0x7f14d003b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 10 monclient: shutdown 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 >> v1:172.21.15.148:6790/0 conn(0x7f14f0133910 legacy=0x7f14f0137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 shutdown_connections 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 >> 172.21.15.148:0/4021857256 conn(0x7f14f0062630 msgr2=0x7f14f013d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 shutdown_connections 2024-09-16T04:18:39.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.373+0000 7f14f5477640 1 -- 172.21.15.148:0/4021857256 wait complete. 2024-09-16T04:18:39.386 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:18:39.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:18:39.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.483+0000 7f5f9b437640 10 monclient: get_monmap_and_config 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.483+0000 7f5f9b437640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.483+0000 7f5f9b437640 10 monclient: monmap: 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:39.484788+0000 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:39.484788+0000 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.486 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:39.487 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:39.487 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.484+0000 7f5f9b437640 1 Processor -- start 2024-09-16T04:18:39.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.484+0000 7f5f9b437640 1 -- start start 2024-09-16T04:18:39.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.484+0000 7f5f9b437640 10 monclient: init 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient(hunting): picked mon.noname-c con 0x7f5f940a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f9409e040 con 0x7f5f9409b5c0 2024-09-16T04:18:39.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f9409d660 con 0x7f5f940a0e90 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f9409c0d0 con 0x7f5f9409b170 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient(hunting): authenticate will time out at 1540.110119s 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f989d2640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5f9409b170 0x7f5f941383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49176/0 (socket says 172.21.15.148:49176) 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f989d2640 1 -- 172.21.15.148:0/3786818402 learned_addr learned my addr 172.21.15.148:0/3786818402 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f999d4640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5f9409b5c0 0x7f5f9413bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40896/0 (socket says 172.21.15.148:40896) 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988085151 0 0) 0x7f5f9409c0d0 con 0x7f5f9409b170 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): my global_id is 104186 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f7c003440 con 0x7f5f9409b170 2024-09-16T04:18:39.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2389197554 0 0) 0x7f5f84001490 con 0x7f5f9409b5c0 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): my global_id is 114105 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f9409c0d0 con 0x7f5f9409b5c0 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121951969 0 0) 0x7f5f8c001110 con 0x7f5f940a0e90 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient(hunting): my global_id is 6790 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f84001490 con 0x7f5f940a0e90 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4178026147 0 0) 0x7f5f7c003440 con 0x7f5f9409b170 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 >> v1:172.21.15.171:6789/0 conn(0x7f5f940a0e90 legacy=0x7f5f9409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 >> v1:172.21.15.148:6789/0 conn(0x7f5f9409b5c0 legacy=0x7f5f9413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 monclient: found mon.noname-b 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9409d380 con 0x7f5f9409b170 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f88001610 con 0x7f5f9409b170 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: dump: 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.498 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 5 monclient: authenticate success, global_id 104186 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f88001840 con 0x7f5f9409b170 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 1 -- 172.21.15.148:0/3786818402 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f88001c50 con 0x7f5f9409b170 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f9b437640 4 set_mon_vals no callback set 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.485+0000 7f5f93fff640 10 monclient: dump: 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.500 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 10 monclient: shutdown 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 1 -- 172.21.15.148:0/3786818402 >> v1:172.21.15.148:6790/0 conn(0x7f5f9409b170 legacy=0x7f5f941383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 1 -- 172.21.15.148:0/3786818402 shutdown_connections 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 1 -- 172.21.15.148:0/3786818402 >> 172.21.15.148:0/3786818402 conn(0x7f5f940a5640 msgr2=0x7f5f940a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 1 -- 172.21.15.148:0/3786818402 shutdown_connections 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 1 -- 172.21.15.148:0/3786818402 wait complete. 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.486+0000 7f5f9b437640 10 monclient: monmap: 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:39.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 1 Processor -- start 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 1 -- start start 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient: init 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient(hunting): picked mon.noname-b con 0x7f5f940a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f941c1cd0 con 0x7f5f9409b5c0 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f941a6460 con 0x7f5f940a0e90 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f941a8f20 con 0x7f5f9409b170 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f9b437640 10 monclient(hunting): authenticate will time out at 1540.112770s 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f991d3640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5f9409b170 0x7f5f941c15c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49184/0 (socket says 172.21.15.148:49184) 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f991d3640 1 -- 172.21.15.148:0/476523222 learned_addr learned my addr 172.21.15.148:0/476523222 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.487+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3491132024 0 0) 0x7f5f941c1cd0 con 0x7f5f9409b5c0 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): my global_id is 114108 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f80003520 con 0x7f5f9409b5c0 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147793916 0 0) 0x7f5f941a8f20 con 0x7f5f9409b170 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): my global_id is 104189 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f941c1cd0 con 0x7f5f9409b170 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587691000 0 0) 0x7f5f941a6460 con 0x7f5f940a0e90 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): my global_id is 6793 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f941a8f20 con 0x7f5f940a0e90 2024-09-16T04:18:39.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1565620037 0 0) 0x7f5f80003520 con 0x7f5f9409b5c0 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f941a6460 con 0x7f5f9409b5c0 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f88001b90 con 0x7f5f9409b5c0 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): dump: 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.505 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1503994482 0 0) 0x7f5f941c1cd0 con 0x7f5f9409b170 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f80003520 con 0x7f5f9409b170 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f8c001a00 con 0x7f5f9409b170 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient(hunting): dump: 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.506 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1707556516 0 0) 0x7f5f941a6460 con 0x7f5f9409b5c0 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.148:6790/0 conn(0x7f5f9409b170 legacy=0x7f5f941c15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.171:6789/0 conn(0x7f5f940a0e90 legacy=0x7f5f941d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 monclient: found mon.a 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f94072d10 con 0x7f5f9409b5c0 2024-09-16T04:18:39.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 5 monclient: authenticate success, global_id 114108 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 10 monclient: _renew_subs 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f940740a0 con 0x7f5f9409b5c0 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f88004c60 con 0x7f5f9409b5c0 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f88005030 con 0x7f5f9409b5c0 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f91ffb640 10 monclient: dump: 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 10 monclient: _renew_subs 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:18:39.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f941a8c40 con 0x7f5f9409b5c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.488+0000 7f5f937fe640 4 set_mon_vals no callback set 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5f88005290 con 0x7f5f9409b5c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f9b437640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f5f88047c00 con 0x7f5f9409b5c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f940008d0 con 0x7f5f9402d7c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f991d3640 10 monclient: get_auth_request con 0x7f5f80039090 auth_method 0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215545455 0 0) 0x7f5f940008d0 con 0x7f5f9402d7c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 10 monclient(hunting): my global_id is 114111 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f941a6460 con 0x7f5f9402d7c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.489+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 769656189 0 0) 0x7f5f941a6460 con 0x7f5f9402d7c0 2024-09-16T04:18:39.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f940008d0 con 0x7f5f9402d7c0 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f84001840 con 0x7f5f9402d7c0 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 10 monclient: dump: 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.511 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3376519699 0 0) 0x7f5f940008d0 con 0x7f5f9402d7c0 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f9402dba0 con 0x7f5f9402d7c0 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5f84001660 con 0x7f5f9402d7c0 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.148:6789/0 conn(0x7f5f9402d7c0 legacy=0x7f5f94068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.490+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.148:6789/0 conn(0x7f5f9402d7c0 legacy=0x7f5f94068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f9b437640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5f941a1670 con 0x7f5f941c4030 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 332293863 0 0) 0x7f5f941a1670 con 0x7f5f941c4030 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient(hunting): my global_id is 114114 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f940008d0 con 0x7f5f941c4030 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2119384120 0 0) 0x7f5f940008d0 con 0x7f5f941c4030 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f941a1670 con 0x7f5f941c4030 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5f84004ef0 con 0x7f5f941c4030 2024-09-16T04:18:39.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.496+0000 7f5f91ffb640 10 monclient: dump: 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1245242498 0 0) 0x7f5f941a1670 con 0x7f5f941c4030 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f941c4410 con 0x7f5f941c4030 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3978 (unknown 0 0 549989593) 0x7f5f84006360 con 0x7f5f941c4030 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.148:6789/0 conn(0x7f5f941c4030 legacy=0x7f5f941a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f91ffb640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.148:6789/0 conn(0x7f5f941c4030 legacy=0x7f5f941a0f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:18:39.516 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 52, 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 15399, 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.517 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.518 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:18:39.519 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:18:39.520 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.521 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.523 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:39.524 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.525 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.526 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.527 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:18:39.527 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.527 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.527 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.527 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:18:39.527 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:18:39.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.171:6812/7327273 conn(0x7f5f80039090 legacy=0x7f5f8003b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 10 monclient: shutdown 2024-09-16T04:18:39.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 >> v1:172.21.15.148:6789/0 conn(0x7f5f9409b5c0 legacy=0x7f5f941cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f999d4640 1 -- 172.21.15.148:0/476523222 reap_dead start 2024-09-16T04:18:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 shutdown_connections 2024-09-16T04:18:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 >> 172.21.15.148:0/476523222 conn(0x7f5f940a5640 msgr2=0x7f5f940a33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 shutdown_connections 2024-09-16T04:18:39.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.497+0000 7f5f9b437640 1 -- 172.21.15.148:0/476523222 wait complete. 2024-09-16T04:18:39.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:18:39.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:18:39.529 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:18:39.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.625+0000 7f52715cd640 10 monclient: get_monmap_and_config 2024-09-16T04:18:39.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.625+0000 7f52715cd640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.626+0000 7f52715cd640 10 monclient: monmap: 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:39.626913+0000 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:39.626913+0000 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:39.628 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.626+0000 7f52715cd640 1 Processor -- start 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.626+0000 7f52715cd640 1 -- start start 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.626+0000 7f52715cd640 10 monclient: init 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient(hunting): picked mon.noname-c con 0x7f526c09d700 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient(hunting): picked mon.noname-b con 0x7f526c09fb50 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient(hunting): picked mon.noname-a con 0x7f526c09dae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c0a0a40 con 0x7f526c09dae0 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c0a0d20 con 0x7f526c09d700 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c09f140 con 0x7f526c09fb50 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient(hunting): authenticate will time out at 1540.252189s 2024-09-16T04:18:39.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f526a7fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f526c09fb50 0x7f526c09ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49192/0 (socket says 172.21.15.148:49192) 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f526a7fc640 1 -- 172.21.15.148:0/2399662764 learned_addr learned my addr 172.21.15.148:0/2399662764 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f526b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f526c09dae0 0x7f526c13d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40940/0 (socket says 172.21.15.148:40940) 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808403539 0 0) 0x7f526c09f140 con 0x7f526c09fb50 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): my global_id is 104192 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5254003520 con 0x7f526c09fb50 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 723107483 0 0) 0x7f526c0a0a40 con 0x7f526c09dae0 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): my global_id is 114117 2024-09-16T04:18:39.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f526c09f140 con 0x7f526c09dae0 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3644425535 0 0) 0x7f526c0a0d20 con 0x7f526c09d700 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient(hunting): my global_id is 6796 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f526c0a0a40 con 0x7f526c09d700 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2204426417 0 0) 0x7f526c09f140 con 0x7f526c09dae0 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 >> v1:172.21.15.148:6790/0 conn(0x7f526c09fb50 legacy=0x7f526c09ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 >> v1:172.21.15.171:6789/0 conn(0x7f526c09d700 legacy=0x7f526c09e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 monclient: found mon.noname-a 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:18:39.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c0a3290 con 0x7f526c09dae0 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f525c0015f0 con 0x7f526c09dae0 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 5 monclient: authenticate success, global_id 114117 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: dump: 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.640 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f525c001860 con 0x7f526c09dae0 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 1 -- 172.21.15.148:0/2399662764 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f525c001c70 con 0x7f526c09dae0 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f5269ffb640 10 monclient: dump: 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.641 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.627+0000 7f52715cd640 4 set_mon_vals no callback set 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 10 monclient: shutdown 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 1 -- 172.21.15.148:0/2399662764 >> v1:172.21.15.148:6789/0 conn(0x7f526c09dae0 legacy=0x7f526c13d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 1 -- 172.21.15.148:0/2399662764 shutdown_connections 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 1 -- 172.21.15.148:0/2399662764 >> 172.21.15.148:0/2399662764 conn(0x7f526c06bb90 msgr2=0x7f526c06bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 1 -- 172.21.15.148:0/2399662764 shutdown_connections 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.628+0000 7f52715cd640 1 -- 172.21.15.148:0/2399662764 wait complete. 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient: monmap: 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:39.643 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 1 Processor -- start 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 1 -- start start 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient: init 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient(hunting): picked mon.noname-b con 0x7f526c09d700 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient(hunting): picked mon.noname-a con 0x7f526c09dae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 10 monclient(hunting): picked mon.noname-c con 0x7f526c09fb50 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c1c5d40 con 0x7f526c09dae0 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.629+0000 7f52715cd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c09e110 con 0x7f526c09d700 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526c06e970 con 0x7f526c09fb50 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 10 monclient(hunting): authenticate will time out at 1540.255016s 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f526a7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f526c09dae0 0x7f526c1d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40954/0 (socket says 172.21.15.148:40954) 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f526a7fc640 1 -- 172.21.15.148:0/801375337 learned_addr learned my addr 172.21.15.148:0/801375337 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566512242 0 0) 0x7f526c06e970 con 0x7f526c09fb50 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): my global_id is 104195 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5258003520 con 0x7f526c09fb50 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375652844 0 0) 0x7f526c1c5d40 con 0x7f526c09dae0 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): my global_id is 114120 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f526c06e970 con 0x7f526c09dae0 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894387578 0 0) 0x7f526c09e110 con 0x7f526c09d700 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): my global_id is 6799 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f526c1c5d40 con 0x7f526c09d700 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 485683859 0 0) 0x7f5258003520 con 0x7f526c09fb50 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f526c09e110 con 0x7f526c09fb50 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f525c0014f0 con 0x7f526c09fb50 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:39.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): dump: 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4162597379 0 0) 0x7f526c06e970 con 0x7f526c09dae0 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5258003520 con 0x7f526c09dae0 2024-09-16T04:18:39.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f52600014e0 con 0x7f526c09dae0 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient(hunting): dump: 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2911617418 0 0) 0x7f526c09e110 con 0x7f526c09fb50 2024-09-16T04:18:39.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.171:6789/0 conn(0x7f526c09d700 legacy=0x7f526c1c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.148:6789/0 conn(0x7f526c09dae0 legacy=0x7f526c1d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 monclient: found mon.c 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c1ac6f0 con 0x7f526c09fb50 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 5 monclient: authenticate success, global_id 104195 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 10 monclient: _renew_subs 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f526c1cc4a0 con 0x7f526c09fb50 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 10 monclient: _renew_subs 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f526c1ab200 con 0x7f526c09fb50 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f525c004b30 con 0x7f526c09fb50 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f525c004ce0 con 0x7f526c09fb50 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52697fa640 4 set_mon_vals no callback set 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.630+0000 7f52537fe640 10 monclient: dump: 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.650 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.631+0000 7f52715cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.631+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f524c0065c0 con 0x7f524c002860 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.631+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f525c010110 con 0x7f526c09fb50 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.631+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f525c0478a0 con 0x7f526c09fb50 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4095399226 0 0) 0x7f524c0065c0 con 0x7f524c002860 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient(hunting): my global_id is 104198 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f526c09e110 con 0x7f524c002860 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f526a7fc640 10 monclient: get_auth_request con 0x7f5258038de0 auth_method 0 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 205818433 0 0) 0x7f526c09e110 con 0x7f524c002860 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f524c0065c0 con 0x7f524c002860 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5264003790 con 0x7f524c002860 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient: dump: 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.652 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1603581702 0 0) 0x7f524c0065c0 con 0x7f524c002860 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f524c00a150 con 0x7f524c002860 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f52640059b0 con 0x7f524c002860 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.148:6790/0 conn(0x7f524c002860 legacy=0x7f524c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.632+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.148:6790/0 conn(0x7f524c002860 legacy=0x7f524c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.638+0000 7f52715cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.638+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f524c009e20 con 0x7f524c0090e0 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572688197 0 0) 0x7f524c009e20 con 0x7f524c0090e0 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient(hunting): my global_id is 104201 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f524c0065c0 con 0x7f524c0090e0 2024-09-16T04:18:39.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2552638361 0 0) 0x7f524c0065c0 con 0x7f524c0090e0 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f524c009e20 con 0x7f524c0090e0 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5264004fc0 con 0x7f524c0090e0 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient: dump: 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.656 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1789418174 0 0) 0x7f524c009e20 con 0x7f524c0090e0 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f524c00c390 con 0x7f524c0090e0 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 2657716391) 0x7f5264004e20 con 0x7f524c0090e0 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.148:6790/0 conn(0x7f524c0090e0 legacy=0x7f524c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52537fe640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.148:6790/0 conn(0x7f524c0090e0 legacy=0x7f524c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:18:39.657 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 52, 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 129075, 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:18:39.658 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:18:39.659 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.660 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:18:39.661 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.662 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.663 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.664 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.665 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.666 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:18:39.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.668 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:18:39.669 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:18:39.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.171:6812/7327273 conn(0x7f5258038de0 legacy=0x7f525803b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52715cd640 10 monclient: shutdown 2024-09-16T04:18:39.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52715cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 >> v1:172.21.15.148:6790/0 conn(0x7f526c09fb50 legacy=0x7f526c1d4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f526b7fe640 1 -- 172.21.15.148:0/801375337 reap_dead start 2024-09-16T04:18:39.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f526b7fe640 1 -- 172.21.15.148:0/801375337 reap_dead start 2024-09-16T04:18:39.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 shutdown_connections 2024-09-16T04:18:39.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.639+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 >> 172.21.15.148:0/801375337 conn(0x7f526c06bb90 msgr2=0x7f526c0a4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.640+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 shutdown_connections 2024-09-16T04:18:39.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.640+0000 7f52715cd640 1 -- 172.21.15.148:0/801375337 wait complete. 2024-09-16T04:18:39.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:18:39.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:18:39.671 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:18:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.768+0000 7f413d66e640 10 monclient: get_monmap_and_config 2024-09-16T04:18:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.768+0000 7f413d66e640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.768+0000 7f413d66e640 10 monclient: monmap: 2024-09-16T04:18:39.770 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:39.769795+0000 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:39.769795+0000 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:39.771 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.769+0000 7f413d66e640 1 Processor -- start 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.769+0000 7f413d66e640 1 -- start start 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.769+0000 7f413d66e640 10 monclient: init 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient(hunting): picked mon.noname-c con 0x7f41380a2680 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient(hunting): picked mon.noname-b con 0x7f41380a4920 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient(hunting): picked mon.noname-a con 0x7f4138136c50 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41380a08b0 con 0x7f4138136c50 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41380a0b90 con 0x7f41380a2680 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41380a4da0 con 0x7f41380a4920 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient(hunting): authenticate will time out at 1540.395145s 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f412e7fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f41380a4920 0x7f413813ba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49236/0 (socket says 172.21.15.148:49236) 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f412e7fc640 1 -- 172.21.15.148:0/112888578 learned_addr learned my addr 172.21.15.148:0/112888578 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 285890982 0 0) 0x7f41380a4da0 con 0x7f41380a4920 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): my global_id is 104204 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4124003520 con 0x7f41380a4920 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482879940 0 0) 0x7f41380a08b0 con 0x7f4138136c50 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): my global_id is 114123 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41380a4da0 con 0x7f4138136c50 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705459573 0 0) 0x7f41380a0b90 con 0x7f41380a2680 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient(hunting): my global_id is 6802 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41380a08b0 con 0x7f41380a2680 2024-09-16T04:18:39.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2822727122 0 0) 0x7f4124003520 con 0x7f41380a4920 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 >> v1:172.21.15.171:6789/0 conn(0x7f41380a2680 legacy=0x7f41380a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 >> v1:172.21.15.148:6789/0 conn(0x7f4138136c50 legacy=0x7f413813f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 monclient: found mon.noname-b 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413806dfd0 con 0x7f41380a4920 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 5 monclient: authenticate success, global_id 104204 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4120001610 con 0x7f41380a4920 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: dump: 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.784 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4120001930 con 0x7f41380a4920 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 1 -- 172.21.15.148:0/112888578 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4120001d40 con 0x7f41380a4920 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f413d66e640 4 set_mon_vals no callback set 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.770+0000 7f41367fc640 10 monclient: dump: 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.786 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 10 monclient: shutdown 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 1 -- 172.21.15.148:0/112888578 >> v1:172.21.15.148:6790/0 conn(0x7f41380a4920 legacy=0x7f413813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 1 -- 172.21.15.148:0/112888578 shutdown_connections 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 1 -- 172.21.15.148:0/112888578 >> 172.21.15.148:0/112888578 conn(0x7f41380a97b0 msgr2=0x7f41380a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 1 -- 172.21.15.148:0/112888578 shutdown_connections 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 1 -- 172.21.15.148:0/112888578 wait complete. 2024-09-16T04:18:39.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.771+0000 7f413d66e640 10 monclient: monmap: 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 1 Processor -- start 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 1 -- start start 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient: init 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient(hunting): picked mon.noname-b con 0x7f41380a2680 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient(hunting): picked mon.noname-c con 0x7f41380a4920 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient(hunting): picked mon.noname-a con 0x7f4138136c50 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4138135930 con 0x7f4138136c50 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4138132b30 con 0x7f41380a2680 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4138134480 con 0x7f41380a4920 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f413d66e640 10 monclient(hunting): authenticate will time out at 1540.397904s 2024-09-16T04:18:39.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f412e7fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f41380a4920 0x7f4138132420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49248/0 (socket says 172.21.15.148:49248) 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.772+0000 7f412e7fc640 1 -- 172.21.15.148:0/1757065135 learned_addr learned my addr 172.21.15.148:0/1757065135 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2350404815 0 0) 0x7f4138134480 con 0x7f41380a4920 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): my global_id is 104207 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4118003520 con 0x7f41380a4920 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089611209 0 0) 0x7f4138135930 con 0x7f4138136c50 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): my global_id is 114126 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4138134480 con 0x7f4138136c50 2024-09-16T04:18:39.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 929148610 0 0) 0x7f4138132b30 con 0x7f41380a2680 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): my global_id is 6805 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4138135930 con 0x7f41380a2680 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3444908319 0 0) 0x7f4118003520 con 0x7f41380a4920 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4138132b30 con 0x7f41380a4920 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41200015d0 con 0x7f41380a4920 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): dump: 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.791 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3324329761 0 0) 0x7f4138134480 con 0x7f4138136c50 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4118003520 con 0x7f4138136c50 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41300015d0 con 0x7f4138136c50 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): dump: 2024-09-16T04:18:39.792 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 289440035 0 0) 0x7f4138135930 con 0x7f41380a2680 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4138134480 con 0x7f41380a2680 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f41280018b0 con 0x7f41380a2680 2024-09-16T04:18:39.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient(hunting): dump: 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1999760128 0 0) 0x7f4138132b30 con 0x7f41380a4920 2024-09-16T04:18:39.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.171:6789/0 conn(0x7f41380a2680 legacy=0x7f4138131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.148:6789/0 conn(0x7f4138136c50 legacy=0x7f41381d3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 monclient: found mon.c 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4138135650 con 0x7f41380a4920 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 5 monclient: authenticate success, global_id 104207 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 10 monclient: _renew_subs 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4138134780 con 0x7f41380a4920 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 10 monclient: _renew_subs 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f41381d53f0 con 0x7f41380a4920 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4120001930 con 0x7f41380a4920 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4120004cc0 con 0x7f41380a4920 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f4135ffb640 4 set_mon_vals no callback set 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.773+0000 7f412ffff640 10 monclient: dump: 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.796 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f413d66e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f411c0065c0 con 0x7f411c002860 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f412000ff50 con 0x7f41380a4920 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f4120047650 con 0x7f41380a4920 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655763451 0 0) 0x7f411c0065c0 con 0x7f411c002860 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f41377fe640 10 monclient: get_auth_request con 0x7f41180390b0 auth_method 0 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 10 monclient(hunting): my global_id is 6808 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.774+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4138132b30 con 0x7f411c002860 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3063246347 0 0) 0x7f4138132b30 con 0x7f411c002860 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f411c0065c0 con 0x7f411c002860 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4128003790 con 0x7f411c002860 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 10 monclient: dump: 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.798 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3875986694 0 0) 0x7f411c0065c0 con 0x7f411c002860 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f411c00a150 con 0x7f411c002860 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f41280037f0 con 0x7f411c002860 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.171:6789/0 conn(0x7f411c002860 legacy=0x7f411c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.775+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.171:6789/0 conn(0x7f411c002860 legacy=0x7f411c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:39.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.781+0000 7f413d66e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:18:39.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.781+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f411c009e40 con 0x7f411c009100 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724888735 0 0) 0x7f411c009e40 con 0x7f411c009100 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient(hunting): my global_id is 6811 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f411c0065c0 con 0x7f411c009100 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 258017627 0 0) 0x7f411c0065c0 con 0x7f411c009100 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f411c009e40 con 0x7f411c009100 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4128003e30 con 0x7f411c009100 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 10 monclient: dump: 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4194139538 0 0) 0x7f411c009e40 con 0x7f411c009100 2024-09-16T04:18:39.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.782+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f411c00c3d0 con 0x7f411c009100 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 2124699226) 0x7f41280040b0 con 0x7f411c009100 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f412ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f412ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.171:6789/0 conn(0x7f411c009100 legacy=0x7f411c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f412ffff640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.171:6789/0 conn(0x7f411c009100 legacy=0x7f411c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 52, 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 743821, 2024-09-16T04:18:39.803 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.804 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.805 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:18:39.806 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:39.807 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:18:39.808 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:18:39.809 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.810 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:39.811 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:39.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.813 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:39.814 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:39.815 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:18:39.815 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:18:39.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.171:6812/7327273 conn(0x7f41180390b0 legacy=0x7f411803b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 10 monclient: shutdown 2024-09-16T04:18:39.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 >> v1:172.21.15.148:6790/0 conn(0x7f41380a4920 legacy=0x7f4138132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f41377fe640 1 -- 172.21.15.148:0/1757065135 reap_dead start 2024-09-16T04:18:39.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f41377fe640 1 -- 172.21.15.148:0/1757065135 reap_dead start 2024-09-16T04:18:39.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 shutdown_connections 2024-09-16T04:18:39.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 >> 172.21.15.148:0/1757065135 conn(0x7f41380a97b0 msgr2=0x7f413812f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 shutdown_connections 2024-09-16T04:18:39.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.783+0000 7f413d66e640 1 -- 172.21.15.148:0/1757065135 wait complete. 2024-09-16T04:18:39.817 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:18:39.817 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:18:39.817 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.912+0000 7f92a7fc4640 10 monclient: get_monmap_and_config 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.912+0000 7f92a7fc4640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.912+0000 7f92a7fc4640 10 monclient: monmap: 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:39.913313+0000 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:39.913313+0000 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.914 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:39.915 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:39.915 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.912+0000 7f92a7fc4640 1 Processor -- start 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 1 -- start start 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient: init 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f92a009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f92a009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f92a009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a00a1010 con 0x7f92a009bb70 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a009f030 con 0x7f92a009c660 2024-09-16T04:18:39.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a009dc60 con 0x7f92a009cc30 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a7fc4640 10 monclient(hunting): authenticate will time out at 1540.538691s 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a6561640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f92a009cc30 0x7f92a013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49264/0 (socket says 172.21.15.148:49264) 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a5d60640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f92a009bb70 0x7f92a009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40988/0 (socket says 172.21.15.148:40988) 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.913+0000 7f92a6561640 1 -- 172.21.15.148:0/3354521734 learned_addr learned my addr 172.21.15.148:0/3354521734 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346316529 0 0) 0x7f92a009dc60 con 0x7f92a009cc30 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): my global_id is 104210 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9288003520 con 0x7f92a009cc30 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818058882 0 0) 0x7f92a00a1010 con 0x7f92a009bb70 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient(hunting): my global_id is 114129 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92a009dc60 con 0x7f92a009bb70 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1782114515 0 0) 0x7f9288003520 con 0x7f92a009cc30 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 >> v1:172.21.15.171:6789/0 conn(0x7f92a009c660 legacy=0x7f92a0138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 >> v1:172.21.15.148:6789/0 conn(0x7f92a009bb70 legacy=0x7f92a009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 monclient: found mon.noname-b 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a009df40 con 0x7f92a009cc30 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 5 monclient: authenticate success, global_id 104210 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9290001990 con 0x7f92a009cc30 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: dump: 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.926 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9290001c00 con 0x7f92a009cc30 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 1 -- 172.21.15.148:0/3354521734 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9290001fd0 con 0x7f92a009cc30 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 4 set_mon_vals no callback set 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a4d5e640 10 monclient: dump: 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.927 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 10 monclient: shutdown 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 1 -- 172.21.15.148:0/3354521734 >> v1:172.21.15.148:6790/0 conn(0x7f92a009cc30 legacy=0x7f92a013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 1 -- 172.21.15.148:0/3354521734 shutdown_connections 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.914+0000 7f92a7fc4640 1 -- 172.21.15.148:0/3354521734 >> 172.21.15.148:0/3354521734 conn(0x7f92a00a5640 msgr2=0x7f92a00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 1 -- 172.21.15.148:0/3354521734 shutdown_connections 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 1 -- 172.21.15.148:0/3354521734 wait complete. 2024-09-16T04:18:39.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 10 monclient: build_initial_monmap 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 10 monclient: monmap: 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:39.929 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 1 Processor -- start 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 1 -- start start 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.915+0000 7f92a7fc4640 10 monclient: init 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient(hunting): picked mon.noname-c con 0x7f92a009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient(hunting): picked mon.noname-a con 0x7f92a009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient(hunting): picked mon.noname-b con 0x7f92a009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a0072080 con 0x7f92a009c660 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a00740a0 con 0x7f92a009cc30 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a01251a0 con 0x7f92a009bb70 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:39.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a7fc4640 10 monclient(hunting): authenticate will time out at 1540.541506s 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a5d60640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f92a009bb70 0x7f92a01229f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49268/0 (socket says 172.21.15.148:49268) 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a555f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f92a009c660 0x7f92a0123100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41000/0 (socket says 172.21.15.148:41000) 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f92a5d60640 1 -- 172.21.15.148:0/2256655362 learned_addr learned my addr 172.21.15.148:0/2256655362 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968930493 0 0) 0x7f92a01251a0 con 0x7f92a009bb70 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): my global_id is 104213 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f928c003520 con 0x7f92a009bb70 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026971391 0 0) 0x7f92a009f030 con 0x7f92a009c660 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): my global_id is 114132 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92a01251a0 con 0x7f92a009c660 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914310067 0 0) 0x7f92a009df40 con 0x7f92a009cc30 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 10 monclient(hunting): my global_id is 6817 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92a009f030 con 0x7f92a009cc30 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.916+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3450911867 0 0) 0x7f92a01251a0 con 0x7f92a009c660 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92a009df40 con 0x7f92a009c660 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9294001830 con 0x7f92a009c660 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:18:39.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): dump: 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 241394310 0 0) 0x7f928c003520 con 0x7f92a009bb70 2024-09-16T04:18:39.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92a01251a0 con 0x7f92a009bb70 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f92980017c0 con 0x7f92a009bb70 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): dump: 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2259453246 0 0) 0x7f92a009f030 con 0x7f92a009cc30 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f928c003520 con 0x7f92a009cc30 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9290001db0 con 0x7f92a009cc30 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient(hunting): dump: 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.935 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3218262776 0 0) 0x7f92a01251a0 con 0x7f92a009bb70 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 >> v1:172.21.15.171:6789/0 conn(0x7f92a009cc30 legacy=0x7f92a01d37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 >> v1:172.21.15.148:6789/0 conn(0x7f92a009c660 legacy=0x7f92a0123100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 monclient: found mon.c 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a00a4040 con 0x7f92a009bb70 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:39.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 5 monclient: authenticate success, global_id 104213 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 10 monclient: _renew_subs 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92a0126150 con 0x7f92a009bb70 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 10 monclient: _renew_subs 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f92a0124d00 con 0x7f92a009bb70 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9298001cf0 con 0x7f92a009bb70 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9298004980 con 0x7f92a009bb70 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929ffff640 4 set_mon_vals no callback set 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.917+0000 7f929e7fc640 10 monclient: dump: 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:39.937 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f929800fc10 con 0x7f92a009bb70 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f92a7fc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f92a7fc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f92a7fc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9280005080 con 0x7f92a009bb70 2024-09-16T04:18:39.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f9298047600 con 0x7f92a009bb70 2024-09-16T04:18:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.918+0000 7f92a555f640 10 monclient: get_auth_request con 0x7f928c0392b0 auth_method 0 2024-09-16T04:18:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.922+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9298015860 con 0x7f92a009bb70 2024-09-16T04:18:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.922+0000 7f929e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:39.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:39.922+0000 7f929e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:40.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.151+0000 7f92a7fc4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:18:40.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.151+0000 7f92a7fc4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:18:40.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.151+0000 7f92a7fc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:40.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.151+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9280005b80 con 0x7f92a009bb70 2024-09-16T04:18:40.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.153+0000 7f929e7fc640 1 -- 172.21.15.148:0/2256655362 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f9298015160 con 0x7f92a009bb70 2024-09-16T04:18:40.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.153+0000 7f929e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:18:40.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.153+0000 7f929e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:40.158 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.153+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.158 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.153+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=10,logm=56} crc {auth=3910466030,config=692619927,health=2640735855,kv=2270513034,logm=3848436664}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.153+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=3512590965,config=692619927,health=2815006003,kv=2270513034,logm=1435999433}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.153+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.153+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=10,logm=56} crc {auth=3910466030,config=692619927,health=2640735855,kv=2270513034,logm=3848436664}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.153+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=10,kv=10,logm=56} crc {auth=3536367042,config=692619927,health=3094638551,kv=2270513034,logm=4037613414}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.154+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.154+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067924552}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.154+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1024373538}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.154+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.154+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067924552}) 2024-09-16T04:18:40.159 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.154+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3750804416}) 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 >> v1:172.21.15.171:6812/7327273 conn(0x7f928c0392b0 legacy=0x7f928c03b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 10 monclient: shutdown 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 >> v1:172.21.15.148:6790/0 conn(0x7f92a009bb70 legacy=0x7f92a01229f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 shutdown_connections 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 >> 172.21.15.148:0/2256655362 conn(0x7f92a00a5640 msgr2=0x7f92a0133880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:40.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 shutdown_connections 2024-09-16T04:18:40.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:40.155+0000 7f92a7fc4640 1 -- 172.21.15.148:0/2256655362 wait complete. 2024-09-16T04:18:40.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.155+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.155+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248649620}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.155+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251494132}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.155+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.155+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248649620}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.155+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275722817}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.156+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.156+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490001260}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.156+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1407727902}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.156+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.156+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490001260}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.156+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113339168}) 2024-09-16T04:18:40.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.157+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.157+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744001796}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.157+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1291043420}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.157+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.157+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744001796}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.157+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2910606433}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.158+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.158+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729947996}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.158+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518439322}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.158+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.158+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729947996}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.158+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447049479}) 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.159+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.159+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632787213}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.159+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2730640740}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.159+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.159+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632787213}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.159+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198250890}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444402250}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987694581}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444402250}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837165085}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168407920}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3185355060}) 2024-09-16T04:18:40.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168407920}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.160+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3956911544}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.161+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.161+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666646272}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.161+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2688709309}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.161+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.161+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666646272}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.161+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201714349}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.162+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.162+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111155935}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.162+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979216768}) 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.162+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.162+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111155935}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.162+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684469659}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759990602}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3107310983}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759990602}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168729205}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137144425}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257658384}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137144425}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.163+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505133364}) 2024-09-16T04:18:40.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.164+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.164+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914325585}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.164+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022158589}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.164+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.164+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914325585}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.164+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842979382}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652794702}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1006834731}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652794702}) 2024-09-16T04:18:40.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3628218109}) 2024-09-16T04:18:40.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570106984}) 2024-09-16T04:18:40.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811713225}) 2024-09-16T04:18:40.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.165+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570106984}) 2024-09-16T04:18:40.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517036205}) 2024-09-16T04:18:40.174 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382244184}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4135005610}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382244184}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.166+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1050609290}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.167+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.167+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017289455}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.167+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=185341196}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.167+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.167+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017289455}) 2024-09-16T04:18:40.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.167+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9837602}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.168+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.168+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460054122}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.168+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=330929668}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.168+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.168+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460054122}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.168+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808967375}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.169+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.169+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090686138}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.169+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878068479}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.169+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.169+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090686138}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.169+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143504147}) 2024-09-16T04:18:40.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.170+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.170+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246413318}) 2024-09-16T04:18:40.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.170+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3953294457}) 2024-09-16T04:18:40.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.170+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.170+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246413318}) 2024-09-16T04:18:40.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.170+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1622108676}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.177+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.177+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606507552}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.177+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890174564}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.177+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.177+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606507552}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.177+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653076531}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132857926}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3215293857}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132857926}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273385750}) 2024-09-16T04:18:40.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697938603}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2625182145}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697938603}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.178+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569874463}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.179+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.179+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693589375}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.179+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=587995449}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.179+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.179+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693589375}) 2024-09-16T04:18:40.182 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.179+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=275324433}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722768528}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149319155}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722768528}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976713415}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043855841}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826075739}) 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043855841}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.180+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111703558}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3132444480}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358787613}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3132444480}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94024794}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020961573}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500670708}) 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020961573}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.181+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=683689280}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.182+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.182+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437122275}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.182+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3236869698}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.182+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.182+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437122275}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.182+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326282972}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117994421}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076970175}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117994421}) 2024-09-16T04:18:40.187 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134747692}) 2024-09-16T04:18:40.192 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.192 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986629822}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=145394262}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986629822}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.183+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275451881}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.184+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.184+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513804302}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.184+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1110068428}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.184+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.184+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513804302}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.184+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651184306}) 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957699186}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375423}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957699186}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711953313}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125573050}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440842996}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125573050}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.185+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4228139899}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.186+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.186+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367822046}) 2024-09-16T04:18:40.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.186+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434153355}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.186+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.186+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367822046}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.186+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959567382}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029613895}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304421273}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029613895}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1632903666}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363819100}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839191356}) 2024-09-16T04:18:40.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363819100}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.187+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218062795}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.188+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.188+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661326852}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.188+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260780949}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.188+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.188+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1661326852}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.188+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499819374}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724355092}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253893943}) 2024-09-16T04:18:40.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724355092}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846062774}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397802031}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683943157}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397802031}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.189+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729296906}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.190+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.190+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177552206}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.190+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010830487}) 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.190+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.190+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177552206}) 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.190+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=349733713}) 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640992146}) 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294442409}) 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640992146}) 2024-09-16T04:18:40.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1048244590}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269715135}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3662823378}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269715135}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.191+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582665183}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=808636070}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290146611}) 2024-09-16T04:18:40.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=808636070}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823617879}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.192+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226971301}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965471080}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226971301}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522783545}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=1593029052,mdsmap=1097804592,mgr=1797096635}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1564888442,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=1118702704}) 2024-09-16T04:18:40.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=1593029052,mdsmap=1097804592,mgr=1797096635}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.193+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=90,mdsmap=3,mgr=6,mgr_command_descs=1} crc {logm=3340382228,mdsmap=1097804592,mgr=1086676294,mgr_command_descs=3961477881}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.195+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.195+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=2586702516,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2310594831,mon_config_key=1612079775,monmap=2633035915,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=316000033}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.195+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=2586702516,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2619444612,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=2496656991}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.195+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.195+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=2586702516,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2310594831,mon_config_key=1612079775,monmap=2633035915,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=316000033}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.195+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=67} crc {mgr=2586702516,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2310594831,mon_config_key=3527728415,monmap=48254013,osd_metadata=1015068370,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=3854956859}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.196+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.196+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1604884509}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.196+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=3720868659}) 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.196+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:18:40.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.196+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1604884509}) 2024-09-16T04:18:40.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:18:40.196+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=3056285537}) 2024-09-16T04:18:41.176 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:18:41.176 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:18:41.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.286+0000 7fe6ce6f8640 10 monclient: get_monmap_and_config 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.286+0000 7fe6ce6f8640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.286+0000 7fe6ce6f8640 10 monclient: monmap: 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:41.287246+0000 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:41.287246+0000 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.288 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:41.289 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:41.289 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 1 Processor -- start 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 1 -- start start 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient: init 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c80a1010 con 0x7fe6c809cc30 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c809f030 con 0x7fe6c809bb70 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c809dc60 con 0x7fe6c809c660 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ce6f8640 10 monclient(hunting): authenticate will time out at 1541.912664s 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ccc95640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe6c809cc30 0x7fe6c813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41008/0 (socket says 172.21.15.148:41008) 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6c77fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe6c809c660 0x7fe6c8138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49284/0 (socket says 172.21.15.148:49284) 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.287+0000 7fe6ccc95640 1 -- 172.21.15.148:0/2580019418 learned_addr learned my addr 172.21.15.148:0/2580019418 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859497021 0 0) 0x7fe6c80a1010 con 0x7fe6c809cc30 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): my global_id is 114135 2024-09-16T04:18:41.298 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b0003520 con 0x7fe6c809cc30 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2418881329 0 0) 0x7fe6c809dc60 con 0x7fe6c809c660 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): my global_id is 104216 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c80a1010 con 0x7fe6c809c660 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 737432915 0 0) 0x7fe6c809f030 con 0x7fe6c809bb70 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient(hunting): my global_id is 6820 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c809dc60 con 0x7fe6c809bb70 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3004864209 0 0) 0x7fe6b0003520 con 0x7fe6c809cc30 2024-09-16T04:18:41.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 >> v1:172.21.15.148:6790/0 conn(0x7fe6c809c660 legacy=0x7fe6c8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 >> v1:172.21.15.171:6789/0 conn(0x7fe6c809bb70 legacy=0x7fe6c809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c809df40 con 0x7fe6c809cc30 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe6c0001990 con 0x7fe6c809cc30 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 5 monclient: authenticate success, global_id 114135 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:41.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: dump: 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:41.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6c0001bc0 con 0x7fe6c809cc30 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 1 -- 172.21.15.148:0/2580019418 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe6c0001fd0 con 0x7fe6c809cc30 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 4 set_mon_vals no callback set 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6c6ffd640 10 monclient: dump: 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.302 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 10 monclient: shutdown 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.288+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/2580019418 >> v1:172.21.15.148:6789/0 conn(0x7fe6c809cc30 legacy=0x7fe6c813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/2580019418 shutdown_connections 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/2580019418 >> 172.21.15.148:0/2580019418 conn(0x7fe6c80a5640 msgr2=0x7fe6c80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/2580019418 shutdown_connections 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/2580019418 wait complete. 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 10 monclient: monmap: 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.303 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.289+0000 7fe6ce6f8640 1 Processor -- start 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 1 -- start start 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient: init 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:18:41.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c81338e0 con 0x7fe6c809cc30 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c806edc0 con 0x7fe6c809c660 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c806fc40 con 0x7fe6c809bb70 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6ce6f8640 10 monclient(hunting): authenticate will time out at 1541.915744s 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6c7fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe6c809bb70 0x7fe6c806dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49290/0 (socket says 172.21.15.148:49290) 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6c7fff640 1 -- 172.21.15.148:0/158269925 learned_addr learned my addr 172.21.15.148:0/158269925 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369288055 0 0) 0x7fe6c806fc40 con 0x7fe6c809bb70 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.290+0000 7fe6c4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): my global_id is 104219 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b4003520 con 0x7fe6c809bb70 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674477825 0 0) 0x7fe6c81338e0 con 0x7fe6c809cc30 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): my global_id is 114138 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c806fc40 con 0x7fe6c809cc30 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771709597 0 0) 0x7fe6c806edc0 con 0x7fe6c809c660 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): my global_id is 6823 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c81338e0 con 0x7fe6c809c660 2024-09-16T04:18:41.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 533744857 0 0) 0x7fe6b4003520 con 0x7fe6c809bb70 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6c806edc0 con 0x7fe6c809bb70 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe6b8001970 con 0x7fe6c809bb70 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): dump: 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.307 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4194519906 0 0) 0x7fe6c806fc40 con 0x7fe6c809cc30 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6b4003520 con 0x7fe6c809cc30 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe6c0001f10 con 0x7fe6c809cc30 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): dump: 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.308 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 947524353 0 0) 0x7fe6c81338e0 con 0x7fe6c809c660 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6c806fc40 con 0x7fe6c809c660 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe6bc001250 con 0x7fe6c809c660 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient(hunting): dump: 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.310 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3629093036 0 0) 0x7fe6c806edc0 con 0x7fe6c809bb70 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 >> v1:172.21.15.171:6789/0 conn(0x7fe6c809c660 legacy=0x7fe6c806e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 >> v1:172.21.15.148:6789/0 conn(0x7fe6c809cc30 legacy=0x7fe6c81d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 monclient: found mon.c 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c8133600 con 0x7fe6c809bb70 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 5 monclient: authenticate success, global_id 104219 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 10 monclient: _renew_subs 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c8132490 con 0x7fe6c809bb70 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 10 monclient: _renew_subs 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6c8134070 con 0x7fe6c809bb70 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6b8001cf0 con 0x7fe6c809bb70 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe6b8003a60 con 0x7fe6c809bb70 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c4ff9640 10 monclient: dump: 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.312 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.291+0000 7fe6c67fc640 4 set_mon_vals no callback set 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.292+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe6b8003a60 con 0x7fe6c809bb70 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.292+0000 7fe6ce6f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.292+0000 7fe6ce6f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.292+0000 7fe6ce6f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.292+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a8005080 con 0x7fe6c809bb70 2024-09-16T04:18:41.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.292+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fe6b80478e0 con 0x7fe6c809bb70 2024-09-16T04:18:41.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.293+0000 7fe6c77fe640 10 monclient: get_auth_request con 0x7fe6b40392a0 auth_method 0 2024-09-16T04:18:41.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.297+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe6b8015ac0 con 0x7fe6c809bb70 2024-09-16T04:18:41.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.297+0000 7fe6c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.297+0000 7fe6c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:41.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6ce6f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:18:41.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6ce6f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:18:41.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6ce6f8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe6a8005370 con 0x7fe6c809bb70 2024-09-16T04:18:41.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6c4ff9640 1 -- 172.21.15.148:0/158269925 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3642890985) 0x7fe6b8019770 con 0x7fe6c809bb70 2024-09-16T04:18:41.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:18:41.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.528+0000 7fe6c4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:41.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 >> v1:172.21.15.171:6812/7327273 conn(0x7fe6b40392a0 legacy=0x7fe6b403b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 10 monclient: shutdown 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 >> v1:172.21.15.148:6790/0 conn(0x7fe6c809bb70 legacy=0x7fe6c806dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 shutdown_connections 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 >> 172.21.15.148:0/158269925 conn(0x7fe6c80a5640 msgr2=0x7fe6c809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 shutdown_connections 2024-09-16T04:18:41.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.531+0000 7fe6ce6f8640 1 -- 172.21.15.148:0/158269925 wait complete. 2024-09-16T04:18:41.543 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:18:41.543 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:18:41.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:18:41.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.640+0000 7f5b5e152640 10 monclient: get_monmap_and_config 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.640+0000 7f5b5e152640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.640+0000 7f5b5e152640 10 monclient: monmap: 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:41.641447+0000 2024-09-16T04:18:41.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:41.641447+0000 2024-09-16T04:18:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:41.643 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:41.643 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 1 Processor -- start 2024-09-16T04:18:41.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 1 -- start start 2024-09-16T04:18:41.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient: init 2024-09-16T04:18:41.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient(hunting): picked mon.noname-b con 0x7f5b5809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient(hunting): picked mon.noname-a con 0x7f5b5809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient(hunting): picked mon.noname-c con 0x7f5b5809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580a1010 con 0x7f5b5809c660 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b5809f030 con 0x7f5b5809cc30 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b5809dc60 con 0x7f5b5809bb70 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b5e152640 10 monclient(hunting): authenticate will time out at 1542.266766s 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b577fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5b5809bb70 0x7f5b5809bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49292/0 (socket says 172.21.15.148:49292) 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b56ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5b5809c660 0x7f5b58138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41018/0 (socket says 172.21.15.148:41018) 2024-09-16T04:18:41.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.641+0000 7f5b577fe640 1 -- 172.21.15.148:0/1452234204 learned_addr learned my addr 172.21.15.148:0/1452234204 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 6063322 0 0) 0x7f5b580a1010 con 0x7f5b5809c660 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): my global_id is 114141 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b40003520 con 0x7f5b5809c660 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3981454967 0 0) 0x7f5b5809dc60 con 0x7f5b5809bb70 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): my global_id is 104222 2024-09-16T04:18:41.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b580a1010 con 0x7f5b5809bb70 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2633758773 0 0) 0x7f5b5809f030 con 0x7f5b5809cc30 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient(hunting): my global_id is 6826 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b5809dc60 con 0x7f5b5809cc30 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3923712385 0 0) 0x7f5b580a1010 con 0x7f5b5809bb70 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 >> v1:172.21.15.171:6789/0 conn(0x7f5b5809cc30 legacy=0x7f5b5813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 >> v1:172.21.15.148:6789/0 conn(0x7f5b5809c660 legacy=0x7f5b58138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 monclient: found mon.noname-b 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5809df40 con 0x7f5b5809bb70 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 5 monclient: authenticate success, global_id 104222 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b48001520 con 0x7f5b5809bb70 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: dump: 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.655 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b48001790 con 0x7f5b5809bb70 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 1 -- 172.21.15.148:0/1452234204 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b48001ba0 con 0x7f5b5809bb70 2024-09-16T04:18:41.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 4 set_mon_vals no callback set 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b567fc640 10 monclient: dump: 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.657 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 10 monclient: shutdown 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.642+0000 7f5b5e152640 1 -- 172.21.15.148:0/1452234204 >> v1:172.21.15.148:6790/0 conn(0x7f5b5809bb70 legacy=0x7f5b5809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 1 -- 172.21.15.148:0/1452234204 shutdown_connections 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 1 -- 172.21.15.148:0/1452234204 >> 172.21.15.148:0/1452234204 conn(0x7f5b580a5640 msgr2=0x7f5b580a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 1 -- 172.21.15.148:0/1452234204 shutdown_connections 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 1 -- 172.21.15.148:0/1452234204 wait complete. 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 10 monclient: monmap: 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:41.658 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.643+0000 7f5b5e152640 1 Processor -- start 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 1 -- start start 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient: init 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient(hunting): picked mon.noname-c con 0x7f5b5809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient(hunting): picked mon.noname-a con 0x7f5b5809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient(hunting): picked mon.noname-b con 0x7f5b5809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580a3f90 con 0x7f5b5809c660 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580740a0 con 0x7f5b5809cc30 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b58124960 con 0x7f5b5809bb70 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b5e152640 10 monclient(hunting): authenticate will time out at 1542.269574s 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b577fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5b5809bb70 0x7f5b581cbb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49298/0 (socket says 172.21.15.148:49298) 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b577fe640 1 -- 172.21.15.148:0/605564152 learned_addr learned my addr 172.21.15.148:0/605564152 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914153313 0 0) 0x7f5b58124960 con 0x7f5b5809bb70 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): my global_id is 104225 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b44003520 con 0x7f5b5809bb70 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686392650 0 0) 0x7f5b580a3f90 con 0x7f5b5809c660 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 10 monclient(hunting): my global_id is 114144 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.644+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b58124960 con 0x7f5b5809c660 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369509334 0 0) 0x7f5b580740a0 con 0x7f5b5809cc30 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): my global_id is 6829 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b580a3f90 con 0x7f5b5809cc30 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 683928791 0 0) 0x7f5b44003520 con 0x7f5b5809bb70 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b580740a0 con 0x7f5b5809bb70 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b480047e0 con 0x7f5b5809bb70 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): dump: 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.662 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1952236402 0 0) 0x7f5b58124960 con 0x7f5b5809c660 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b44003520 con 0x7f5b5809c660 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b4c001190 con 0x7f5b5809c660 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient(hunting): dump: 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.663 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4022342019 0 0) 0x7f5b580740a0 con 0x7f5b5809bb70 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.171:6789/0 conn(0x7f5b5809cc30 legacy=0x7f5b581d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.148:6789/0 conn(0x7f5b5809c660 legacy=0x7f5b581d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 monclient: found mon.c 2024-09-16T04:18:41.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b58124c40 con 0x7f5b5809bb70 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 5 monclient: authenticate success, global_id 104225 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 10 monclient: _renew_subs 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5b58123230 con 0x7f5b5809bb70 2024-09-16T04:18:41.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 10 monclient: _renew_subs 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5b58121cc0 con 0x7f5b5809bb70 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5b48004c30 con 0x7f5b5809bb70 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b48001790 con 0x7f5b5809bb70 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b55ffb640 4 set_mon_vals no callback set 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.645+0000 7f5b3ffff640 10 monclient: dump: 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.666 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5b48010070 con 0x7f5b5809bb70 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b5e152640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b380065c0 con 0x7f5b38002860 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f5b48047a50 con 0x7f5b5809bb70 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3763904129 0 0) 0x7f5b380065c0 con 0x7f5b38002860 2024-09-16T04:18:41.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient(hunting): my global_id is 114147 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b580740a0 con 0x7f5b38002860 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b57fff640 10 monclient: get_auth_request con 0x7f5b44039210 auth_method 0 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3597672523 0 0) 0x7f5b580740a0 con 0x7f5b38002860 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b380065c0 con 0x7f5b38002860 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b4c004980 con 0x7f5b38002860 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 10 monclient: dump: 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.668 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 96256555 0 0) 0x7f5b380065c0 con 0x7f5b38002860 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.646+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b3800a150 con 0x7f5b38002860 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.647+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5b4c005ea0 con 0x7f5b38002860 2024-09-16T04:18:41.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.647+0000 7f5b3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.647+0000 7f5b3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:41.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.647+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.148:6789/0 conn(0x7f5b38002860 legacy=0x7f5b38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.647+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.148:6789/0 conn(0x7f5b38002860 legacy=0x7f5b38004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:41.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b5e152640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:18:41.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5b38009e40 con 0x7f5b38009100 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347825008 0 0) 0x7f5b38009e40 con 0x7f5b38009100 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient(hunting): my global_id is 114150 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b380065c0 con 0x7f5b38009100 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4082144058 0 0) 0x7f5b380065c0 con 0x7f5b38009100 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5b38009e40 con 0x7f5b38009100 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5b4c006520 con 0x7f5b38009100 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient: dump: 2024-09-16T04:18:41.672 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3832689164 0 0) 0x7f5b38009e40 con 0x7f5b38009100 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b3800c3d0 con 0x7f5b38009100 2024-09-16T04:18:41.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3978 (unknown 0 0 3171404627) 0x7f5b4c006780 con 0x7f5b38009100 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.148:6789/0 conn(0x7f5b38009100 legacy=0x7f5b38009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.653+0000 7f5b3ffff640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.148:6789/0 conn(0x7f5b38009100 legacy=0x7f5b38009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:18:41.674 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 52, 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 17555, 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.675 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:18:41.676 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.677 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:18:41.678 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.679 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.680 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.681 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.682 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.683 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.684 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.685 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.685 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.686 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:41.687 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.687 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.687 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:18:41.687 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:18:41.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.171:6812/7327273 conn(0x7f5b44039210 legacy=0x7f5b4403b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 10 monclient: shutdown 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 >> v1:172.21.15.148:6790/0 conn(0x7f5b5809bb70 legacy=0x7f5b581cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b57fff640 1 -- 172.21.15.148:0/605564152 reap_dead start 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b57fff640 1 -- 172.21.15.148:0/605564152 reap_dead start 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 shutdown_connections 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 >> 172.21.15.148:0/605564152 conn(0x7f5b580a5640 msgr2=0x7f5b581b9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 shutdown_connections 2024-09-16T04:18:41.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.654+0000 7f5b5e152640 1 -- 172.21.15.148:0/605564152 wait complete. 2024-09-16T04:18:41.689 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:18:41.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:18:41.690 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:18:41.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.787+0000 7fb3abe53640 10 monclient: get_monmap_and_config 2024-09-16T04:18:41.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.787+0000 7fb3abe53640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.787+0000 7fb3abe53640 10 monclient: monmap: 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:41.787941+0000 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:41.787941+0000 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:41.789 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.787+0000 7fb3abe53640 1 Processor -- start 2024-09-16T04:18:41.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.787+0000 7fb3abe53640 1 -- start start 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.787+0000 7fb3abe53640 10 monclient: init 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a4062630 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a4134ff0 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a413b050 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a40a1720 con 0x7fb3a4134ff0 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a4134610 con 0x7fb3a4062630 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a40a1960 con 0x7fb3a413b050 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3abe53640 10 monclient(hunting): authenticate will time out at 1542.413413s 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3aa3f0640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fb3a413b050 0x7fb3a413d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49302/0 (socket says 172.21.15.148:49302) 2024-09-16T04:18:41.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3aa3f0640 1 -- 172.21.15.148:0/525680983 learned_addr learned my addr 172.21.15.148:0/525680983 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a93ee640 1 --1- 172.21.15.148:0/525680983 >> v1:172.21.15.148:6789/0 conn(0x7fb3a4134ff0 0x7fb3a4139930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41044/0 (socket says 172.21.15.148:41044) 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233082768 0 0) 0x7fb3a40a1720 con 0x7fb3a4134ff0 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): my global_id is 114153 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb390003520 con 0x7fb3a4134ff0 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235068295 0 0) 0x7fb3a40a1960 con 0x7fb3a413b050 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 10 monclient(hunting): my global_id is 104228 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a40a1720 con 0x7fb3a413b050 2024-09-16T04:18:41.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3284216672 0 0) 0x7fb390003520 con 0x7fb3a4134ff0 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.788+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 >> v1:172.21.15.148:6790/0 conn(0x7fb3a413b050 legacy=0x7fb3a413d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 >> v1:172.21.15.171:6789/0 conn(0x7fb3a4062630 legacy=0x7fb3a41348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 1 monclient: found mon.noname-a 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a4099690 con 0x7fb3a4134ff0 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb394001610 con 0x7fb3a4134ff0 2024-09-16T04:18:41.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: dump: 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.801 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 5 monclient: authenticate success, global_id 114153 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb394001800 con 0x7fb3a4134ff0 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 1 -- 172.21.15.148:0/525680983 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb394001c10 con 0x7fb3a4134ff0 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3a8bed640 10 monclient: dump: 2024-09-16T04:18:41.802 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 4 set_mon_vals no callback set 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 10 monclient: shutdown 2024-09-16T04:18:41.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 1 -- 172.21.15.148:0/525680983 >> v1:172.21.15.148:6789/0 conn(0x7fb3a4134ff0 legacy=0x7fb3a4139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 1 -- 172.21.15.148:0/525680983 shutdown_connections 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 1 -- 172.21.15.148:0/525680983 >> 172.21.15.148:0/525680983 conn(0x7fb3a40a5860 msgr2=0x7fb3a40a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 1 -- 172.21.15.148:0/525680983 shutdown_connections 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.789+0000 7fb3abe53640 1 -- 172.21.15.148:0/525680983 wait complete. 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.790+0000 7fb3abe53640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.790+0000 7fb3abe53640 10 monclient: monmap: 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.804 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.790+0000 7fb3abe53640 1 Processor -- start 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.790+0000 7fb3abe53640 1 -- start start 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.790+0000 7fb3abe53640 10 monclient: init 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a4062630 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a4134ff0 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a413b050 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a4137b00 con 0x7fb3a4062630 2024-09-16T04:18:41.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a4137de0 con 0x7fb3a4134ff0 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a40a1010 con 0x7fb3a413b050 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient(hunting): authenticate will time out at 1542.416167s 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3a9bef640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb3a4062630 0x7fb3a40986d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41048/0 (socket says 172.21.15.148:41048) 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3a9bef640 1 -- 172.21.15.148:0/4194126355 learned_addr learned my addr 172.21.15.148:0/4194126355 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3aa3f0640 1 --1- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6790/0 conn(0x7fb3a413b050 0x7fb3a41d2150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49312/0 (socket says 172.21.15.148:49312) 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936798585 0 0) 0x7fb3a40a1010 con 0x7fb3a413b050 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): my global_id is 104231 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb388003520 con 0x7fb3a413b050 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591241070 0 0) 0x7fb3a4137b00 con 0x7fb3a4062630 2024-09-16T04:18:41.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): my global_id is 114156 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a40a1010 con 0x7fb3a4062630 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 324602644 0 0) 0x7fb388003520 con 0x7fb3a413b050 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb3a4137b00 con 0x7fb3a413b050 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb39c001a30 con 0x7fb3a413b050 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): dump: 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.807 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3120799756 0 0) 0x7fb3a40a1010 con 0x7fb3a4062630 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb388003520 con 0x7fb3a4062630 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525707961 0 0) 0x7fb3a4137de0 con 0x7fb3a4134ff0 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): my global_id is 6835 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a40a1010 con 0x7fb3a4134ff0 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3a0001b80 con 0x7fb3a4062630 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient(hunting): dump: 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.809 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1111958306 0 0) 0x7fb3a4137b00 con 0x7fb3a413b050 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.171:6789/0 conn(0x7fb3a4134ff0 legacy=0x7fb3a41cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6789/0 conn(0x7fb3a4062630 legacy=0x7fb3a40986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 monclient: found mon.c 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a409c630 con 0x7fb3a413b050 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb39a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 5 monclient: authenticate success, global_id 104231 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient: _renew_subs 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a41380c0 con 0x7fb3a413b050 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient: _renew_subs 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.791+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb3a41374f0 con 0x7fb3a413b050 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb39c001630 con 0x7fb3a413b050 2024-09-16T04:18:41.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb39c004a90 con 0x7fb3a413b050 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39bfff640 4 set_mon_vals no callback set 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 10 monclient: dump: 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.812 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb3abe53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb39c00fec0 con 0x7fb3a413b050 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.792+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb38c0065c0 con 0x7fb38c002860 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fb39c047800 con 0x7fb3a413b050 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653748650 0 0) 0x7fb38c0065c0 con 0x7fb38c002860 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient(hunting): my global_id is 104234 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3a4137b00 con 0x7fb38c002860 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb3a93ee640 10 monclient: get_auth_request con 0x7fb388039180 auth_method 0 2024-09-16T04:18:41.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1975587353 0 0) 0x7fb3a4137b00 con 0x7fb38c002860 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb38c0065c0 con 0x7fb38c002860 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3a0003970 con 0x7fb38c002860 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient: dump: 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.814 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4123219500 0 0) 0x7fb38c0065c0 con 0x7fb38c002860 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb38c00a150 con 0x7fb38c002860 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb3a0005b20 con 0x7fb38c002860 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6790/0 conn(0x7fb38c002860 legacy=0x7fb38c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.793+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6790/0 conn(0x7fb38c002860 legacy=0x7fb38c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:41.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.799+0000 7fb3abe53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.799+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb38c009e40 con 0x7fb38c009100 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729455429 0 0) 0x7fb38c009e40 con 0x7fb38c009100 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient(hunting): my global_id is 104237 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb38c0065c0 con 0x7fb38c009100 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1177340347 0 0) 0x7fb38c0065c0 con 0x7fb38c009100 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb38c009e40 con 0x7fb38c009100 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb3a0003fe0 con 0x7fb38c009100 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient: dump: 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2884851085 0 0) 0x7fb38c009e40 con 0x7fb38c009100 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb38c00c3d0 con 0x7fb38c009100 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 4016909773) 0x7fb3a0004250 con 0x7fb38c009100 2024-09-16T04:18:41.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6790/0 conn(0x7fb38c009100 legacy=0x7fb38c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb39a7fc640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6790/0 conn(0x7fb38c009100 legacy=0x7fb38c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 52, 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 131236, 2024-09-16T04:18:41.819 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.820 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.821 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:18:41.822 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:18:41.823 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.824 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.825 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.826 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:41.827 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.828 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.829 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.830 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:18:41.831 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.171:6812/7327273 conn(0x7fb388039180 legacy=0x7fb38803b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb3abe53640 10 monclient: shutdown 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb3abe53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 >> v1:172.21.15.148:6790/0 conn(0x7fb3a413b050 legacy=0x7fb3a41d2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb3aa3f0640 1 -- 172.21.15.148:0/4194126355 reap_dead start 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.800+0000 7fb3aa3f0640 1 -- 172.21.15.148:0/4194126355 reap_dead start 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.801+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 shutdown_connections 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.801+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 >> 172.21.15.148:0/4194126355 conn(0x7fb3a40a5860 msgr2=0x7fb3a4092130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.801+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 shutdown_connections 2024-09-16T04:18:41.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.801+0000 7fb3abe53640 1 -- 172.21.15.148:0/4194126355 wait complete. 2024-09-16T04:18:41.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:18:41.833 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:18:41.833 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.929+0000 7fbbbe52c640 10 monclient: get_monmap_and_config 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.929+0000 7fbbbe52c640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.929+0000 7fbbbe52c640 10 monclient: monmap: 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:41.930400+0000 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:41.930400+0000 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.931 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.932 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.932 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:41.932 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:41.932 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 1 Processor -- start 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 1 -- start start 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient: init 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb80a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb80a0e00 con 0x7fbbb813d060 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb80a10e0 con 0x7fbbb80a2410 2024-09-16T04:18:41.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb80a4970 con 0x7fbbb80a1b10 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbbe52c640 10 monclient(hunting): authenticate will time out at 1542.555751s 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbb77fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fbbb80a1b10 0x7fbbb813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49348/0 (socket says 172.21.15.148:49348) 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.930+0000 7fbbb77fe640 1 -- 172.21.15.148:0/2051006328 learned_addr learned my addr 172.21.15.148:0/2051006328 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214071845 0 0) 0x7fbbb80a0e00 con 0x7fbbb813d060 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): my global_id is 114159 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba0003520 con 0x7fbbb813d060 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2762473890 0 0) 0x7fbbb80a4970 con 0x7fbbb80a1b10 2024-09-16T04:18:41.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): my global_id is 104240 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb80a0e00 con 0x7fbbb80a1b10 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905418323 0 0) 0x7fbbb80a10e0 con 0x7fbbb80a2410 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient(hunting): my global_id is 6838 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb80a4970 con 0x7fbbb80a2410 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1731122942 0 0) 0x7fbba0003520 con 0x7fbbb813d060 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 >> v1:172.21.15.148:6790/0 conn(0x7fbbb80a1b10 legacy=0x7fbbb813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 >> v1:172.21.15.171:6789/0 conn(0x7fbbb80a2410 legacy=0x7fbbb80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb806dfd0 con 0x7fbbb813d060 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbb0001990 con 0x7fbbb813d060 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 5 monclient: authenticate success, global_id 114159 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:41.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: dump: 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbb0001c30 con 0x7fbbb813d060 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 10 monclient: get_monmap_and_config success 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 1 -- 172.21.15.148:0/2051006328 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbb0002040 con 0x7fbbb813d060 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 4 set_mon_vals no callback set 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbb6ffd640 10 monclient: dump: 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.946 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 10 monclient: shutdown 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.931+0000 7fbbbe52c640 1 -- 172.21.15.148:0/2051006328 >> v1:172.21.15.148:6789/0 conn(0x7fbbb813d060 legacy=0x7fbbb813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 1 -- 172.21.15.148:0/2051006328 shutdown_connections 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 1 -- 172.21.15.148:0/2051006328 >> 172.21.15.148:0/2051006328 conn(0x7fbbb80a97b0 msgr2=0x7fbbb80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 1 -- 172.21.15.148:0/2051006328 shutdown_connections 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 1 -- 172.21.15.148:0/2051006328 wait complete. 2024-09-16T04:18:41.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 10 monclient: build_initial_monmap 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 10 monclient: monmap: 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.932+0000 7fbbbe52c640 1 Processor -- start 2024-09-16T04:18:41.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 1 -- start start 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient: init 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb80a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb80a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb8137b30 con 0x7fbbb80a1b10 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb809efa0 con 0x7fbbb80a2410 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb8137e10 con 0x7fbbb813d060 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbbe52c640 10 monclient(hunting): authenticate will time out at 1542.558537s 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb7fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fbbb80a1b10 0x7fbbb81328c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41058/0 (socket says 172.21.15.148:41058) 2024-09-16T04:18:41.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb7fff640 1 -- 172.21.15.148:0/1677788946 learned_addr learned my addr 172.21.15.148:0/1677788946 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3256434746 0 0) 0x7fbbb8137b30 con 0x7fbbb80a1b10 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): my global_id is 114162 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba4003520 con 0x7fbbb80a1b10 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246040972 0 0) 0x7fbbb8137e10 con 0x7fbbb813d060 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): my global_id is 104243 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb8137b30 con 0x7fbbb813d060 2024-09-16T04:18:41.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126071668 0 0) 0x7fbbb809efa0 con 0x7fbbb80a2410 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 10 monclient(hunting): my global_id is 6841 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.933+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb8137e10 con 0x7fbbb80a2410 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3135587178 0 0) 0x7fbbb8137b30 con 0x7fbbb813d060 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbb809efa0 con 0x7fbbb813d060 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbb0001890 con 0x7fbbb813d060 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): dump: 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.951 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1890398422 0 0) 0x7fbba4003520 con 0x7fbbb80a1b10 2024-09-16T04:18:41.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbb8137b30 con 0x7fbbb80a1b10 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbba8001930 con 0x7fbbb80a1b10 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): dump: 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.953 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1759660260 0 0) 0x7fbbb8137e10 con 0x7fbbb80a2410 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbba4003520 con 0x7fbbb80a2410 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbac001a10 con 0x7fbbb80a2410 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient(hunting): dump: 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.954 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1240437143 0 0) 0x7fbbb809efa0 con 0x7fbbb813d060 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.171:6789/0 conn(0x7fbbb80a2410 legacy=0x7fbbb81d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.148:6789/0 conn(0x7fbbb80a1b10 legacy=0x7fbbb81328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 monclient: found mon.c 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb813cc20 con 0x7fbbb813d060 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:18:41.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 5 monclient: authenticate success, global_id 104243 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 10 monclient: _renew_subs 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb80a0130 con 0x7fbbb813d060 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 10 monclient: _renew_subs 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbb809df60 con 0x7fbbb813d060 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbbb0003ce0 con 0x7fbbb813d060 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbb0004030 con 0x7fbbb813d060 2024-09-16T04:18:41.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb67fc640 4 set_mon_vals no callback set 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.934+0000 7fbbb4ff9640 10 monclient: dump: 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.957 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fbbb000f2c0 con 0x7fbbb813d060 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbbe52c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbb980065c0 con 0x7fbb98002860 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fbbb00467d0 con 0x7fbbb813d060 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb7fff640 10 monclient: get_auth_request con 0x7fbba4039180 auth_method 0 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2801734230 0 0) 0x7fbb980065c0 con 0x7fbb98002860 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 10 monclient(hunting): my global_id is 6844 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb809efa0 con 0x7fbb98002860 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.935+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3625830833 0 0) 0x7fbbb809efa0 con 0x7fbb98002860 2024-09-16T04:18:41.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb980065c0 con 0x7fbb98002860 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbac004980 con 0x7fbb98002860 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 10 monclient: dump: 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.959 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3026307730 0 0) 0x7fbb980065c0 con 0x7fbb98002860 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb9800a150 con 0x7fbb98002860 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbbac004a00 con 0x7fbb98002860 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.171:6789/0 conn(0x7fbb98002860 legacy=0x7fbb98004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.936+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.171:6789/0 conn(0x7fbb98002860 legacy=0x7fbb98004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.942+0000 7fbbbe52c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.942+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbb98009e40 con 0x7fbb98009100 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986711746 0 0) 0x7fbb98009e40 con 0x7fbb98009100 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient(hunting): my global_id is 6847 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb980065c0 con 0x7fbb98009100 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3763220042 0 0) 0x7fbb980065c0 con 0x7fbb98009100 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb98009e40 con 0x7fbb98009100 2024-09-16T04:18:41.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fbbac005000 con 0x7fbb98009100 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient: dump: 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:18:41.963 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 982734269 0 0) 0x7fbb98009e40 con 0x7fbb98009100 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb9800c3d0 con 0x7fbb98009100 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 2345552736) 0x7fbbac007370 con 0x7fbb98009100 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.171:6789/0 conn(0x7fbb98009100 legacy=0x7fbb98009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.943+0000 7fbbb4ff9640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.171:6789/0 conn(0x7fbb98009100 legacy=0x7fbb98009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:18:41.964 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 52, 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 745981, 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:18:41.965 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:18:41.966 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.967 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:18:41.968 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:18:41.969 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.970 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.971 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:18:41.972 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:18:41.973 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.974 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:18:41.975 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:18:41.976 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:18:41.977 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:18:41.977 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:18:41.977 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:18:41.977 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:18:41.977 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:18:41.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.171:6812/7327273 conn(0x7fbba4039180 legacy=0x7fbba403b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 10 monclient: shutdown 2024-09-16T04:18:41.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:18:41.978 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 >> v1:172.21.15.148:6790/0 conn(0x7fbbb813d060 legacy=0x7fbbb81d4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:18:41.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbcac9640 1 -- 172.21.15.148:0/1677788946 reap_dead start 2024-09-16T04:18:41.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbcac9640 1 -- 172.21.15.148:0/1677788946 reap_dead start 2024-09-16T04:18:41.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 shutdown_connections 2024-09-16T04:18:41.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 >> 172.21.15.148:0/1677788946 conn(0x7fbbb80a97b0 msgr2=0x7fbbb80a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:18:41.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 shutdown_connections 2024-09-16T04:18:41.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:41.944+0000 7fbbbe52c640 1 -- 172.21.15.148:0/1677788946 wait complete. 2024-09-16T04:18:41.979 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:18:41.980 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:18:41.980 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T04:18:41.980 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:18:41.980 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:18:41.980 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:18:41.980 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:18:41.991 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:18:41.991 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:18:41.992 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:18:41.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:18:56.992 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:18:56.993 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:18:56.993 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:18:56.993 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:18:57.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.099+0000 7f774e952640 10 monclient: get_monmap_and_config 2024-09-16T04:18:57.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.099+0000 7f774e952640 10 monclient: build_initial_monmap 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.099+0000 7f774e952640 10 monclient: monmap: 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:18:57.100103+0000 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:18:57.100103+0000 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:18:57.101 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.099+0000 7f774e952640 1 Processor -- start 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 1 -- start start 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient: init 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient: _reopen_session rank -1 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient(hunting): picked mon.noname-b con 0x7f77480a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient(hunting): picked mon.noname-a con 0x7f77480a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient(hunting): picked mon.noname-c con 0x7f774813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77480a0e00 con 0x7f77480a1b10 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77480a10e0 con 0x7f774813d060 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77480a4970 con 0x7f77480a2410 2024-09-16T04:18:57.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient(hunting): _renew_subs 2024-09-16T04:18:57.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f774e952640 10 monclient(hunting): authenticate will time out at 1557.725550s 2024-09-16T04:18:57.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f7747fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f77480a27f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:18:57.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f77477fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f77480a1b10 0x7f774813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34374/0 (socket says 172.21.15.148:34374) 2024-09-16T04:18:57.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.100+0000 7f77477fe640 1 -- 172.21.15.148:0/499309782 learned_addr learned my addr 172.21.15.148:0/499309782 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:18:57.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.301+0000 7f7747fff640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f77480a27f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:18:57.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:57.702+0000 7f7747fff640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f77480a27f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:18:58.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:18:58.503+0000 7f7747fff640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f77480a27f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): tick 2024-09-16T04:19:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:19:00.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f77480a27f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.171:6789/0 conn(0x7f774813d060 legacy=0x7f774813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6789/0 conn(0x7f77480a1b10 legacy=0x7f774813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f77280036c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f77480a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f772800ab40 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 -- 172.21.15.148:0/499309782 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774806dfd0 con 0x7f772800ab40 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 -- 172.21.15.148:0/499309782 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77480a4970 con 0x7f77280036c0 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 1 -- 172.21.15.148:0/499309782 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7728010870 con 0x7f77480a2410 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f7745ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:00.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.100+0000 7f77477fe640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f7728009340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:00.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.301+0000 7f77477fe640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f7728009340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:00.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:00.702+0000 7f77477fe640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f7728009340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:01.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:01.503+0000 7f77477fe640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f7728009340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:02.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.006+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722613193 0 0) 0x7f774806dfd0 con 0x7f772800ab40 2024-09-16T04:19:02.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.006+0000 7f7746ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.006+0000 7f7746ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.006+0000 7f7746ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.006+0000 7f7746ffd640 10 monclient(hunting): my global_id is 114165 2024-09-16T04:19:02.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.006+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7734003520 con 0x7f772800ab40 2024-09-16T04:19:02.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1284867238 0 0) 0x7f7734003520 con 0x7f772800ab40 2024-09-16T04:19:02.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6790/0 conn(0x7f77480a2410 legacy=0x7f7728009340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:19:02.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.171:6789/0 conn(0x7f77280036c0 legacy=0x7f7728005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:19:02.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 1 monclient: found mon.noname-a 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7728011aa0 con 0x7f772800ab40 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: _finish_auth 0 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7740001890 con 0x7f772800ab40 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f7746ffd640 10 monclient: dump: 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.011 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f774e952640 5 monclient: authenticate success, global_id 114165 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.007+0000 7f774e952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7740001af0 con 0x7f772800ab40 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f7746ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:02.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f7746ffd640 1 -- 172.21.15.148:0/499309782 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7740001ec0 con 0x7f772800ab40 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f7746ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f7746ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f7746ffd640 10 monclient: dump: 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.013 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f774e952640 10 monclient: get_monmap_and_config success 2024-09-16T04:19:02.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.008+0000 7f774e952640 4 set_mon_vals no callback set 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 10 monclient: shutdown 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 1 -- 172.21.15.148:0/499309782 >> v1:172.21.15.148:6789/0 conn(0x7f772800ab40 legacy=0x7f772800cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 1 -- 172.21.15.148:0/499309782 shutdown_connections 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 1 -- 172.21.15.148:0/499309782 >> 172.21.15.148:0/499309782 conn(0x7f77480a97b0 msgr2=0x7f77480a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 1 -- 172.21.15.148:0/499309782 shutdown_connections 2024-09-16T04:19:02.018 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.009+0000 7f774e952640 1 -- 172.21.15.148:0/499309782 wait complete. 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.010+0000 7f774e952640 10 monclient: build_initial_monmap 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.010+0000 7f774e952640 10 monclient: monmap: 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.011+0000 7f774e952640 1 Processor -- start 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.011+0000 7f774e952640 1 -- start start 2024-09-16T04:19:02.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.011+0000 7f774e952640 10 monclient: init 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient(hunting): picked mon.noname-c con 0x7f77280036c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient(hunting): picked mon.noname-a con 0x7f772800ab40 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient(hunting): picked mon.noname-b con 0x7f77480a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7748124dc0 con 0x7f772800ab40 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7748123bf0 con 0x7f77480a2410 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f7747fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f77280036c0 legacy=0x7f77481399e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77480a8290 con 0x7f77280036c0 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f774e952640 10 monclient(hunting): authenticate will time out at 1562.637599s 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f77477fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f772800ab40 0x7f77481d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34388/0 (socket says 172.21.15.148:34388) 2024-09-16T04:19:02.020 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.012+0000 7f77477fe640 1 -- 172.21.15.148:0/412383780 learned_addr learned my addr 172.21.15.148:0/412383780 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2212661035 0 0) 0x7f7748124dc0 con 0x7f772800ab40 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): my global_id is 114168 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f772800ff90 con 0x7f772800ab40 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1663004219 0 0) 0x7f772800ff90 con 0x7f772800ab40 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7748124dc0 con 0x7f772800ab40 2024-09-16T04:19:02.021 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f77300018b0 con 0x7f772800ab40 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): dump: 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.022 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.013+0000 7f7744ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.015+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 109862558 0 0) 0x7f7748124dc0 con 0x7f772800ab40 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 >> v1:172.21.15.148:6790/0 conn(0x7f77280036c0 legacy=0x7f77481399e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 >> v1:172.21.15.171:6789/0 conn(0x7f77480a2410 legacy=0x7f77481d3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 1 monclient: found mon.a 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7748125380 con 0x7f772800ab40 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: _finish_auth 0 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 5 monclient: authenticate success, global_id 114168 2024-09-16T04:19:02.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 10 monclient: _renew_subs 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7748123970 con 0x7f772800ab40 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 10 monclient: _renew_subs 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f774813a0f0 con 0x7f772800ab40 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7730001c40 con 0x7f772800ab40 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7730004c30 con 0x7f772800ab40 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f77467fc640 4 set_mon_vals no callback set 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.016+0000 7f7744ff9640 10 monclient: dump: 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.025 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.017+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7730004e50 con 0x7f772800ab40 2024-09-16T04:19:02.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.017+0000 7f774e952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.017+0000 7f774e952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.017+0000 7f774e952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.017+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772c005080 con 0x7f772800ab40 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.017+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f7730047b30 con 0x7f772800ab40 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.018+0000 7f7747fff640 10 monclient: get_auth_request con 0x7f7728039b00 auth_method 0 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.022+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7730015d90 con 0x7f772800ab40 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.022+0000 7f7744ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.022+0000 7f7744ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f774e952640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f774e952640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f774e952640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f772c005370 con 0x7f772800ab40 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f7744ff9640 1 -- 172.21.15.148:0/412383780 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1193274541) 0x7f7730015690 con 0x7f772800ab40 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f7744ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:19:02.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.253+0000 7f7744ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:19:02.256 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":54,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:19:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 >> v1:172.21.15.171:6812/7327273 conn(0x7f7728039b00 legacy=0x7f772803bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 10 monclient: shutdown 2024-09-16T04:19:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 >> v1:172.21.15.148:6789/0 conn(0x7f772800ab40 legacy=0x7f77481d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 shutdown_connections 2024-09-16T04:19:02.257 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 >> 172.21.15.148:0/412383780 conn(0x7f77480a97b0 msgr2=0x7f77480a75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:02.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 shutdown_connections 2024-09-16T04:19:02.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.255+0000 7f774e952640 1 -- 172.21.15.148:0/412383780 wait complete. 2024-09-16T04:19:02.268 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:19:02.268 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:19:02.268 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:19:02.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.365+0000 7f162a4dd640 10 monclient: get_monmap_and_config 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.365+0000 7f162a4dd640 10 monclient: build_initial_monmap 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.365+0000 7f162a4dd640 10 monclient: monmap: 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:19:02.366126+0000 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:19:02.366126+0000 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:19:02.367 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.365+0000 7f162a4dd640 1 Processor -- start 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.365+0000 7f162a4dd640 1 -- start start 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.365+0000 7f162a4dd640 10 monclient: init 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient(hunting): picked mon.noname-c con 0x7f16240a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient(hunting): picked mon.noname-a con 0x7f16240a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:02.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient(hunting): picked mon.noname-b con 0x7f162413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16240a0e00 con 0x7f16240a1b10 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16240a10e0 con 0x7f16240a2410 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16240a4970 con 0x7f162413d060 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f162a4dd640 10 monclient(hunting): authenticate will time out at 1562.991486s 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1628a7a640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f162413d060 legacy=0x7f162413f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f16237fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f16240a1b10 0x7f162413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60318/0 (socket says 172.21.15.148:60318) 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f16237fe640 1 -- 172.21.15.148:0/2648096122 learned_addr learned my addr 172.21.15.148:0/2648096122 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574889315 0 0) 0x7f16240a0e00 con 0x7f16240a1b10 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 10 monclient(hunting): my global_id is 114171 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1610003520 con 0x7f16240a1b10 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2365766509 0 0) 0x7f1610003520 con 0x7f16240a1b10 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 >> v1:172.21.15.148:6790/0 conn(0x7f162413d060 legacy=0x7f162413f450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 >> v1:172.21.15.171:6789/0 conn(0x7f16240a2410 legacy=0x7f16240a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 1 monclient: found mon.noname-a 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.366+0000 7f1622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162406dfd0 con 0x7f16240a1b10 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: _finish_auth 0 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:02.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f160c001610 con 0x7f16240a1b10 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 5 monclient: authenticate success, global_id 114171 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: dump: 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.379 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f160c001970 con 0x7f16240a1b10 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 10 monclient: get_monmap_and_config success 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 1 -- 172.21.15.148:0/2648096122 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f160c0048a0 con 0x7f16240a1b10 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 4 set_mon_vals no callback set 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f1622ffd640 10 monclient: dump: 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.380 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 10 monclient: shutdown 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 1 -- 172.21.15.148:0/2648096122 >> v1:172.21.15.148:6789/0 conn(0x7f16240a1b10 legacy=0x7f162413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 1 -- 172.21.15.148:0/2648096122 shutdown_connections 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 1 -- 172.21.15.148:0/2648096122 >> 172.21.15.148:0/2648096122 conn(0x7f16240a97b0 msgr2=0x7f16240a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:02.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 1 -- 172.21.15.148:0/2648096122 shutdown_connections 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.367+0000 7f162a4dd640 1 -- 172.21.15.148:0/2648096122 wait complete. 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.368+0000 7f162a4dd640 10 monclient: build_initial_monmap 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.368+0000 7f162a4dd640 10 monclient: monmap: 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:19:02.382 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.368+0000 7f162a4dd640 1 Processor -- start 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.368+0000 7f162a4dd640 1 -- start start 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.368+0000 7f162a4dd640 10 monclient: init 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient(hunting): picked mon.noname-c con 0x7f16240a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient(hunting): picked mon.noname-a con 0x7f16240a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient(hunting): picked mon.noname-b con 0x7f162413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f162413aa10 con 0x7f16240a2410 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f162413c760 con 0x7f162413d060 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f162413a030 con 0x7f16240a1b10 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:02.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1623fff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f16240a1b10 legacy=0x7f1624136ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient(hunting): authenticate will time out at 1562.994168s 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f16237fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f16240a2410 0x7f16241371d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60334/0 (socket says 172.21.15.148:60334) 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f16237fe640 1 -- 172.21.15.148:0/2481566547 learned_addr learned my addr 172.21.15.148:0/2481566547 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 153913679 0 0) 0x7f162406dfd0 con 0x7f16240a2410 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): my global_id is 114174 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1604003520 con 0x7f16240a2410 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2382392808 0 0) 0x7f161c003d30 con 0x7f162413d060 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): my global_id is 6856 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f162406dfd0 con 0x7f162413d060 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 617252556 0 0) 0x7f1604003520 con 0x7f16240a2410 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f161c003d30 con 0x7f16240a2410 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f160c001430 con 0x7f16240a2410 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): dump: 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.385 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2528586736 0 0) 0x7f162406dfd0 con 0x7f162413d060 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1604003520 con 0x7f162413d060 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1174591504 0 0) 0x7f161c003d30 con 0x7f16240a2410 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.148:6790/0 conn(0x7f16240a1b10 legacy=0x7f1624136ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.171:6789/0 conn(0x7f162413d060 legacy=0x7f16241d3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:19:02.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 monclient: found mon.a 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16241378e0 con 0x7f16240a2410 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: _finish_auth 0 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 5 monclient: authenticate success, global_id 114174 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient: _renew_subs 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f162413cc70 con 0x7f16240a2410 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient: _renew_subs 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16241388a0 con 0x7f16240a2410 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f160c001970 con 0x7f16240a2410 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f160c004e60 con 0x7f16240a2410 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f16227fc640 4 set_mon_vals no callback set 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.369+0000 7f1620ff9640 10 monclient: dump: 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.388 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f160c010290 con 0x7f16240a2410 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f162a4dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f16080065c0 con 0x7f1608002860 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f160c047af0 con 0x7f16240a2410 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262286225 0 0) 0x7f16080065c0 con 0x7f1608002860 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 10 monclient(hunting): my global_id is 114177 2024-09-16T04:19:02.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f161c003d30 con 0x7f1608002860 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.370+0000 7f1628a7a640 10 monclient: get_auth_request con 0x7f1604038e50 auth_method 0 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3308549883 0 0) 0x7f161c003d30 con 0x7f1608002860 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16080065c0 con 0x7f1608002860 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f16140016f0 con 0x7f1608002860 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 10 monclient: dump: 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.390 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2352692766 0 0) 0x7f16080065c0 con 0x7f1608002860 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f160800a150 con 0x7f1608002860 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1614001b20 con 0x7f1608002860 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.148:6789/0 conn(0x7f1608002860 legacy=0x7f1608004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.371+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.148:6789/0 conn(0x7f1608002860 legacy=0x7f1608004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f162a4dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1608009e40 con 0x7f1608009100 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574409753 0 0) 0x7f1608009e40 con 0x7f1608009100 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f1620ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f1620ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f1620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f1620ff9640 10 monclient(hunting): my global_id is 114180 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.377+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16080065c0 con 0x7f1608009100 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 20894368 0 0) 0x7f16080065c0 con 0x7f1608009100 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1608009e40 con 0x7f1608009100 2024-09-16T04:19:02.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1614003790 con 0x7f1608009100 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 10 monclient: dump: 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.394 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 668926334 0 0) 0x7f1608009e40 con 0x7f1608009100 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f160800c3d0 con 0x7f1608009100 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4134 (unknown 0 0 1786637610) 0x7f1614003790 con 0x7f1608009100 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.148:6789/0 conn(0x7f1608009100 legacy=0x7f1608009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1620ff9640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.148:6789/0 conn(0x7f1608009100 legacy=0x7f1608009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:19:02.395 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 54, 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 38281, 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:19:02.396 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:19:02.397 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:19:02.398 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:19:02.399 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.400 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.401 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:19:02.402 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.403 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.404 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.405 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.406 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.407 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:19:02.407 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.407 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.407 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:19:02.407 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:19:02.408 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.171:6812/7327273 conn(0x7f1604038e50 legacy=0x7f160403b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 10 monclient: shutdown 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 >> v1:172.21.15.148:6789/0 conn(0x7f16240a2410 legacy=0x7f16241371d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1628a7a640 1 -- 172.21.15.148:0/2481566547 reap_dead start 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f1628a7a640 1 -- 172.21.15.148:0/2481566547 reap_dead start 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 shutdown_connections 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 >> 172.21.15.148:0/2481566547 conn(0x7f16240a97b0 msgr2=0x7f1624135780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:02.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 shutdown_connections 2024-09-16T04:19:02.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.378+0000 7f162a4dd640 1 -- 172.21.15.148:0/2481566547 wait complete. 2024-09-16T04:19:02.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:19:02.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:19:02.410 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:19:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.506+0000 7fa9a176d640 10 monclient: get_monmap_and_config 2024-09-16T04:19:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.506+0000 7fa9a176d640 10 monclient: build_initial_monmap 2024-09-16T04:19:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.507+0000 7fa9a176d640 10 monclient: monmap: 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:19:02.507926+0000 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:19:02.507926+0000 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:19:02.509 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.507+0000 7fa9a176d640 1 Processor -- start 2024-09-16T04:19:02.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.507+0000 7fa9a176d640 1 -- start start 2024-09-16T04:19:02.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.507+0000 7fa9a176d640 10 monclient: init 2024-09-16T04:19:02.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:02.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient(hunting): picked mon.noname-c con 0x7fa99c09bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient(hunting): picked mon.noname-b con 0x7fa99c09c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient(hunting): picked mon.noname-a con 0x7fa99c09cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c0a1010 con 0x7fa99c09cc30 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c09f030 con 0x7fa99c09bb70 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c09dc60 con 0x7fa99c09c660 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient(hunting): authenticate will time out at 1563.133206s 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa99a7fc640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa99c09c660 legacy=0x7fa99c138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa99b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa99c09cc30 0x7fa99c13c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60360/0 (socket says 172.21.15.148:60360) 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa99b7fe640 1 -- 172.21.15.148:0/399703351 learned_addr learned my addr 172.21.15.148:0/399703351 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779992304 0 0) 0x7fa99c0a1010 con 0x7fa99c09cc30 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient(hunting): my global_id is 114183 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa984003520 con 0x7fa99c09cc30 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2989551125 0 0) 0x7fa984003520 con 0x7fa99c09cc30 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 >> v1:172.21.15.148:6790/0 conn(0x7fa99c09c660 legacy=0x7fa99c138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 >> v1:172.21.15.171:6789/0 conn(0x7fa99c09bb70 legacy=0x7fa99c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 monclient: found mon.noname-a 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c09df40 con 0x7fa99c09cc30 2024-09-16T04:19:02.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: _finish_auth 0 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa994001990 con 0x7fa99c09cc30 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 5 monclient: authenticate success, global_id 114183 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: dump: 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.522 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa994001c80 con 0x7fa99c09cc30 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 10 monclient: get_monmap_and_config success 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 1 -- 172.21.15.148:0/399703351 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa994002090 con 0x7fa99c09cc30 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa9a176d640 4 set_mon_vals no callback set 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.508+0000 7fa999ffb640 10 monclient: dump: 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 10 monclient: shutdown 2024-09-16T04:19:02.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 1 -- 172.21.15.148:0/399703351 >> v1:172.21.15.148:6789/0 conn(0x7fa99c09cc30 legacy=0x7fa99c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 1 -- 172.21.15.148:0/399703351 shutdown_connections 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 1 -- 172.21.15.148:0/399703351 >> 172.21.15.148:0/399703351 conn(0x7fa99c0a5640 msgr2=0x7fa99c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 1 -- 172.21.15.148:0/399703351 shutdown_connections 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 1 -- 172.21.15.148:0/399703351 wait complete. 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 10 monclient: build_initial_monmap 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.509+0000 7fa9a176d640 10 monclient: monmap: 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:02.525 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 1 Processor -- start 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 1 -- start start 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient: init 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient(hunting): picked mon.noname-c con 0x7fa99c09bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient(hunting): picked mon.noname-b con 0x7fa99c09c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient(hunting): picked mon.noname-a con 0x7fa99c09cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:02.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c0740a0 con 0x7fa99c09cc30 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c132440 con 0x7fa99c09c660 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa99affd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fa99c09bb70 legacy=0x7fa99c11d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99c120040 con 0x7fa99c09bb70 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa9a176d640 10 monclient(hunting): authenticate will time out at 1563.135903s 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa99b7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa99c09cc30 0x7fa99c11e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:60364/0 (socket says 172.21.15.148:60364) 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.510+0000 7fa99b7fe640 1 -- 172.21.15.148:0/648006347 learned_addr learned my addr 172.21.15.148:0/648006347 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2063869549 0 0) 0x7fa99c0740a0 con 0x7fa99c09cc30 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): my global_id is 114186 2024-09-16T04:19:02.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa988003520 con 0x7fa99c09cc30 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264388325 0 0) 0x7fa99c132440 con 0x7fa99c09c660 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): my global_id is 6862 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa99c0740a0 con 0x7fa99c09c660 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 722892950 0 0) 0x7fa988003520 con 0x7fa99c09cc30 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa99c132440 con 0x7fa99c09cc30 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9940017b0 con 0x7fa99c09cc30 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:19:02.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): dump: 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3074367658 0 0) 0x7fa99c0740a0 con 0x7fa99c09c660 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa988003520 con 0x7fa99c09c660 2024-09-16T04:19:02.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9900040c0 con 0x7fa99c09c660 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient(hunting): dump: 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2682488515 0 0) 0x7fa99c132440 con 0x7fa99c09cc30 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.148:6790/0 conn(0x7fa99c09bb70 legacy=0x7fa99c11d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.171:6789/0 conn(0x7fa99c09c660 legacy=0x7fa99c11def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: _finish_hunting 0 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 monclient: found mon.a 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c121370 con 0x7fa99c09cc30 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: _finish_auth 0 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: _check_auth_tickets 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 5 monclient: authenticate success, global_id 114186 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 10 monclient: _renew_subs 2024-09-16T04:19:02.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c121650 con 0x7fa99c09cc30 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 10 monclient: _renew_subs 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa99c1d0970 con 0x7fa99c09cc30 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa994001c80 con 0x7fa99c09cc30 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9940051e0 con 0x7fa99c09cc30 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9997fa640 4 set_mon_vals no callback set 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.511+0000 7fa9837fe640 10 monclient: dump: 2024-09-16T04:19:02.532 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa994010610 con 0x7fa99c09cc30 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9a176d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa97c0065c0 con 0x7fa97c002860 2024-09-16T04:19:02.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fa994047db0 con 0x7fa99c09cc30 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2599772399 0 0) 0x7fa97c0065c0 con 0x7fa97c002860 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa99a7fc640 10 monclient: get_auth_request con 0x7fa988038ec0 auth_method 0 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 10 monclient(hunting): my global_id is 6865 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.512+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa99c132440 con 0x7fa97c002860 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3750981658 0 0) 0x7fa99c132440 con 0x7fa97c002860 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa97c0065c0 con 0x7fa97c002860 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa98c0016f0 con 0x7fa97c002860 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 10 monclient: dump: 2024-09-16T04:19:02.534 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4081375990 0 0) 0x7fa97c0065c0 con 0x7fa97c002860 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa97c00a150 con 0x7fa97c002860 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa98c001ae0 con 0x7fa97c002860 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:02.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:19:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.171:6789/0 conn(0x7fa97c002860 legacy=0x7fa97c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.513+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.171:6789/0 conn(0x7fa97c002860 legacy=0x7fa97c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:19:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.519+0000 7fa9a176d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:19:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.519+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa97c009e40 con 0x7fa97c009100 2024-09-16T04:19:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909870764 0 0) 0x7fa97c009e40 con 0x7fa97c009100 2024-09-16T04:19:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient(hunting): _negotiate 2024-09-16T04:19:02.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient(hunting): my global_id is 6868 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa97c0065c0 con 0x7fa97c009100 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2368419982 0 0) 0x7fa97c0065c0 con 0x7fa97c009100 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa97c009e40 con 0x7fa97c009100 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa98c003a80 con 0x7fa97c009100 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient: dump: 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.538 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2121177793 0 0) 0x7fa97c009e40 con 0x7fa97c009100 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa97c00c3d0 con 0x7fa97c009100 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4300 (unknown 0 0 4015652543) 0x7fa98c001d30 con 0x7fa97c009100 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.171:6789/0 conn(0x7fa97c009100 legacy=0x7fa97c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.520+0000 7fa9837fe640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.171:6789/0 conn(0x7fa97c009100 legacy=0x7fa97c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 54, 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 766559, 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:19:02.540 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:19:02.541 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:19:02.542 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:19:02.543 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:19:02.544 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:19:02.545 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:19:02.546 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:19:02.547 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:19:02.548 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:19:02.549 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.550 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:19:02.551 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:19:02.552 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:19:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.171:6812/7327273 conn(0x7fa988038ec0 legacy=0x7fa98803b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 10 monclient: shutdown 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 >> v1:172.21.15.148:6789/0 conn(0x7fa99c09cc30 legacy=0x7fa99c11e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa99b7fe640 1 -- 172.21.15.148:0/648006347 reap_dead start 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa99b7fe640 1 -- 172.21.15.148:0/648006347 reap_dead start 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 shutdown_connections 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 >> 172.21.15.148:0/648006347 conn(0x7fa99c0a5640 msgr2=0x7fa99c0a31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 shutdown_connections 2024-09-16T04:19:02.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:19:02.521+0000 7fa9a176d640 1 -- 172.21.15.148:0/648006347 wait complete. 2024-09-16T04:19:02.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:19:02.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:19:46.071 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=110 2024-09-16T04:19:46.072 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T04:19:46.072 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:19:46.072 INFO:tasks.workunit.client.0.smithi171.stderr:+ export CEPH_ARGS= 2024-09-16T04:19:46.073 INFO:tasks.workunit.client.0.smithi171.stderr:+ CEPH_ARGS= 2024-09-16T04:19:46.073 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph auth del client.foo 2024-09-16T04:19:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.147+0000 7fb9f3ca5640 10 monclient: get_monmap_and_config 2024-09-16T04:19:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.147+0000 7fb9f3ca5640 10 monclient: build_initial_monmap 2024-09-16T04:19:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.147+0000 7fb9f3ca5640 10 monclient: monmap: 2024-09-16T04:19:46.149 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:19:46.150 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:46.150 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:19:46.148858+0000 2024-09-16T04:19:46.150 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:19:46.148858+0000 2024-09-16T04:19:46.150 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:46.150 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:19:46.150 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:46.151 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:19:46.151 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:19:46.151 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.147+0000 7fb9f3ca5640 1 Processor -- start 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.147+0000 7fb9f3ca5640 1 -- start start 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.147+0000 7fb9f3ca5640 10 monclient: init 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ec09ba20 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:46.160 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ec072e40 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:46.163 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ec09c510 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:46.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a4900 con 0x7fb9ec09ba20 2024-09-16T04:19:46.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a4e40 con 0x7fb9ec09c510 2024-09-16T04:19:46.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a5010 con 0x7fb9ec072e40 2024-09-16T04:19:46.164 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f3ca5640 10 monclient(hunting): authenticate will time out at 1561.654535s 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f1240640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb9ec072e40 legacy=0x7fb9ec13bf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f2242640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb9ec09c510 0x7fb9ec13f640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:48958/0 (socket says 172.21.15.171:48958) 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f2242640 1 -- 172.21.15.171:0/3360122314 learned_addr learned my addr 172.21.15.171:0/3360122314 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544489301 0 0) 0x7fb9ec0a4e40 con 0x7fb9ec09c510 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f0a3f640 10 monclient(hunting): _negotiate 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f0a3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f0a3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f0a3f640 10 monclient(hunting): my global_id is 6871 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.148+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d8003520 con 0x7fb9ec09c510 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639421206 0 0) 0x7fb9ec0a4900 con 0x7fb9ec09ba20 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient(hunting): _negotiate 2024-09-16T04:19:46.165 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient(hunting): my global_id is 114189 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9ec0a4e40 con 0x7fb9ec09ba20 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1064275274 0 0) 0x7fb9d8003520 con 0x7fb9ec09c510 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 >> v1:172.21.15.148:6790/0 conn(0x7fb9ec072e40 legacy=0x7fb9ec13bf10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 >> v1:172.21.15.148:6789/0 conn(0x7fb9ec09ba20 legacy=0x7fb9ec09be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: _finish_hunting 0 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 monclient: found mon.noname-c 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.166 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec0a51e0 con 0x7fb9ec09c510 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: _finish_auth 0 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: _check_auth_tickets 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 5 monclient: authenticate success, global_id 6871 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb9e4001990 con 0x7fb9ec09c510 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: dump: 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.167 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:19:46.168 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9e4001c40 con 0x7fb9ec09c510 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 10 monclient: get_monmap_and_config success 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 1 -- 172.21.15.171:0/3360122314 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb9e4002010 con 0x7fb9ec09c510 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 4 set_mon_vals no callback set 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f0a3f640 10 monclient: dump: 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.169 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 10 monclient: shutdown 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/3360122314 >> v1:172.21.15.171:6789/0 conn(0x7fb9ec09c510 legacy=0x7fb9ec13f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/3360122314 shutdown_connections 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/3360122314 >> 172.21.15.171:0/3360122314 conn(0x7fb9ec0a94e0 msgr2=0x7fb9ec0a98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/3360122314 shutdown_connections 2024-09-16T04:19:46.170 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.149+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/3360122314 wait complete. 2024-09-16T04:19:46.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.150+0000 7fb9f3ca5640 10 monclient: build_initial_monmap 2024-09-16T04:19:46.171 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.150+0000 7fb9f3ca5640 10 monclient: monmap: 2024-09-16T04:19:46.171 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:19:46.171 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:19:46.171 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:19:46.171 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.150+0000 7fb9f3ca5640 1 Processor -- start 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.150+0000 7fb9f3ca5640 1 -- start start 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.150+0000 7fb9f3ca5640 10 monclient: init 2024-09-16T04:19:46.172 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient: _reopen_session rank -1 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ec072e40 addr v1:172.21.15.148:6790/0 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ec09ba20 addr v1:172.21.15.171:6789/0 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ec09c510 addr v1:172.21.15.148:6789/0 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec133720 con 0x7fb9ec09c510 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec137a90 con 0x7fb9ec09ba20 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f1a41640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fb9ec072e40 legacy=0x7fb9ec137380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec134240 con 0x7fb9ec072e40 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient(hunting): _renew_subs 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f3ca5640 10 monclient(hunting): authenticate will time out at 1561.657403s 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f1240640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fb9ec09ba20 0x7fb9ec1d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:48966/0 (socket says 172.21.15.171:48966) 2024-09-16T04:19:46.173 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9f1240640 1 -- 172.21.15.171:0/1277764491 learned_addr learned my addr 172.21.15.171:0/1277764491 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106391603 0 0) 0x7fb9ec137a90 con 0x7fb9ec09ba20 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): my global_id is 6874 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d0003520 con 0x7fb9ec09ba20 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3443704460 0 0) 0x7fb9d0003520 con 0x7fb9ec09ba20 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9ec137a90 con 0x7fb9ec09ba20 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb9dc004180 con 0x7fb9ec09ba20 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:19:46.174 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): dump: 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:46.175 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436291603 0 0) 0x7fb9ec133720 con 0x7fb9ec09c510 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 10 monclient(hunting): my global_id is 114192 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.151+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d0003520 con 0x7fb9ec09c510 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 364235124 0 0) 0x7fb9ec137a90 con 0x7fb9ec09ba20 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 >> v1:172.21.15.148:6790/0 conn(0x7fb9ec072e40 legacy=0x7fb9ec137380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 >> v1:172.21.15.148:6789/0 conn(0x7fb9ec09c510 legacy=0x7fb9ec1d3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 monclient: found mon.b 2024-09-16T04:19:46.176 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec0a0be0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: _finish_auth 0 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 5 monclient: authenticate success, global_id 6874 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 10 monclient: _renew_subs 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb9ec133af0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 10 monclient: _renew_subs 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb9ec0a04f0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.177 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9dc0044e0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fb9dc0048b0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e37fe640 4 set_mon_vals no callback set 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 10 monclient: dump: 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:19:46.178 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fb9dc004ad0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.152+0000 7fb9f3ca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.153+0000 7fb9f3ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.153+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d4005080 con 0x7fb9ec09ba20 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.153+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fb9dc047be0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.153+0000 7fb9f1a41640 10 monclient: get_auth_request con 0x7fb9d0038f70 auth_method 0 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.157+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fb9ec0a0be0 con 0x7fb9ec09ba20 2024-09-16T04:19:46.179 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.157+0000 7fb9e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:19:46.180 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.157+0000 7fb9e27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:19:46.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.390+0000 7fb9f3ca5640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T04:19:46.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.390+0000 7fb9f3ca5640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T04:19:46.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.390+0000 7fb9f3ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.393 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.390+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) -- 0x7fb9d4005370 con 0x7fb9ec09ba20 2024-09-16T04:19:46.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.395+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v19) ==== 80+0+0 (unknown 1312558543 0 0) 0x7fb9dc015e40 con 0x7fb9ec09ba20 2024-09-16T04:19:46.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.395+0000 7fb9e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T04:19:46.397 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.395+0000 7fb9e27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:19:46.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.559+0000 7fb9f3ca5640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T04:19:46.561 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.559+0000 7fb9f3ca5640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T04:19:46.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.559+0000 7fb9f3ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:19:46.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.559+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) -- 0x7fb9d4002210 con 0x7fb9ec09ba20 2024-09-16T04:19:46.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.560+0000 7fb9e27fc640 1 -- 172.21.15.171:0/1277764491 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v19) ==== 80+0+0 (unknown 1312558543 0 0) 0x7fb9dc015740 con 0x7fb9ec09ba20 2024-09-16T04:19:46.562 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.560+0000 7fb9e27fc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-16T04:19:46.563 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.560+0000 7fb9e27fc640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 >> v1:172.21.15.171:6812/7327273 conn(0x7fb9d0038f70 legacy=0x7fb9d003b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 10 monclient: shutdown 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 >> v1:172.21.15.171:6789/0 conn(0x7fb9ec09ba20 legacy=0x7fb9ec1d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 shutdown_connections 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 >> 172.21.15.171:0/1277764491 conn(0x7fb9ec0a94e0 msgr2=0x7fb9ec0a8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.562+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 shutdown_connections 2024-09-16T04:19:46.565 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:19:46.563+0000 7fb9f3ca5640 1 -- 172.21.15.171:0/1277764491 wait complete. 2024-09-16T04:19:46.574 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-16T04:19:46.574 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-16T04:19:46.574 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T04:19:46.575 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:19:46.575 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T04:19:46.575 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-16T04:20:32.556 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:20:32.556 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:20:32.556 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:20:32.557 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:20:32.561 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:20:32.561 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:20:32.562 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:20:32.562 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:20:47.563 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:20:47.563 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:20:47.564 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:20:47.564 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:20:47.587 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:20:47.587 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:20:47.587 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:47.587 INFO:tasks.workunit.client.0.smithi171.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-16T04:20:47.587 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-16T04:20:47.588 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-16T04:20:47.588 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=0 2024-09-16T04:20:47.588 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T04:20:47.588 INFO:tasks.workunit.client.0.smithi171.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T04:20:47.588 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T04:20:47.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient: get_monmap_and_config 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient: build_initial_monmap 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient: monmap: 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:47.684085+0000 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:47.684085+0000 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:47.685 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 1 Processor -- start 2024-09-16T04:20:47.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 1 -- start start 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient: init 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient(hunting): picked mon.noname-b con 0x7f860c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient(hunting): picked mon.noname-c con 0x7f860c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient(hunting): picked mon.noname-a con 0x7f860c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c0a0e00 con 0x7f860c13d060 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c0a10e0 con 0x7f860c0a1b10 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c0a4970 con 0x7f860c0a2410 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f86136f1640 10 monclient(hunting): authenticate will time out at 1668.309518s 2024-09-16T04:20:47.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f861148d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f860c0a2410 0x7f860c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60456/0 (socket says 172.21.15.148:60456) 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f861148d640 1 -- 172.21.15.148:0/3340628210 learned_addr learned my addr 172.21.15.148:0/3340628210 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430306655 0 0) 0x7f860c0a4970 con 0x7f860c0a2410 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): my global_id is 124118 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f8003520 con 0x7f860c0a2410 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2061717534 0 0) 0x7f860c0a0e00 con 0x7f860c13d060 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): my global_id is 114261 2024-09-16T04:20:47.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f860c0a4970 con 0x7f860c13d060 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771402519 0 0) 0x7f860c0a10e0 con 0x7f860c0a1b10 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 10 monclient(hunting): my global_id is 6934 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.683+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f860c0a0e00 con 0x7f860c0a1b10 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3900893125 0 0) 0x7f860c0a4970 con 0x7f860c13d060 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 >> v1:172.21.15.148:6790/0 conn(0x7f860c0a2410 legacy=0x7f860c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 >> v1:172.21.15.171:6789/0 conn(0x7f860c0a1b10 legacy=0x7f860c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: _finish_hunting 0 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 monclient: found mon.noname-a 2024-09-16T04:20:47.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c06dfd0 con 0x7f860c13d060 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: _finish_auth 0 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: _check_auth_tickets 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8604001970 con 0x7f860c13d060 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 5 monclient: authenticate success, global_id 114261 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: dump: 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:47.697 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8604001c50 con 0x7f860c13d060 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:47.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 4 set_mon_vals no callback set 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 1 -- 172.21.15.148:0/3340628210 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8604002060 con 0x7f860c13d060 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f8603fff640 10 monclient: dump: 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:47.699 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 10 monclient: shutdown 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 1 -- 172.21.15.148:0/3340628210 >> v1:172.21.15.148:6789/0 conn(0x7f860c13d060 legacy=0x7f860c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 1 -- 172.21.15.148:0/3340628210 shutdown_connections 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 1 -- 172.21.15.148:0/3340628210 >> 172.21.15.148:0/3340628210 conn(0x7f860c0a97b0 msgr2=0x7f860c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 1 -- 172.21.15.148:0/3340628210 shutdown_connections 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.684+0000 7f86136f1640 1 -- 172.21.15.148:0/3340628210 wait complete. 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.685+0000 7f86136f1640 10 monclient: build_initial_monmap 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.685+0000 7f86136f1640 10 monclient: monmap: 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:47.700 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.685+0000 7f86136f1640 1 Processor -- start 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.685+0000 7f86136f1640 1 -- start start 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.685+0000 7f86136f1640 10 monclient: init 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient(hunting): picked mon.noname-a con 0x7f860c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:47.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient(hunting): picked mon.noname-c con 0x7f860c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient(hunting): picked mon.noname-b con 0x7f860c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c13cb90 con 0x7f860c0a1b10 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c0a0130 con 0x7f860c13d060 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c137b30 con 0x7f860c0a2410 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient(hunting): authenticate will time out at 1668.312198s 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f861148d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f860c0a1b10 0x7f860c1328c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48294/0 (socket says 172.21.15.148:48294) 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f861148d640 1 -- 172.21.15.148:0/4093156541 learned_addr learned my addr 172.21.15.148:0/4093156541 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1688839196 0 0) 0x7f8608001910 con 0x7f860c0a1b10 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:47.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): my global_id is 114264 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f0003520 con 0x7f860c0a1b10 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100576685 0 0) 0x7f860c137b30 con 0x7f860c0a2410 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): my global_id is 124121 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8608001910 con 0x7f860c0a2410 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1359617922 0 0) 0x7f860c06dfd0 con 0x7f860c13d060 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:47.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): my global_id is 6937 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f860c137b30 con 0x7f860c13d060 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 279608299 0 0) 0x7f85f0003520 con 0x7f860c0a1b10 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f860c06dfd0 con 0x7f860c0a1b10 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8608001c40 con 0x7f860c0a1b10 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): dump: 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:47.704 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3047946951 0 0) 0x7f8608001910 con 0x7f860c0a2410 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85f0003520 con 0x7f860c0a2410 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f85fc001830 con 0x7f860c0a2410 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): dump: 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:47.705 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3679104239 0 0) 0x7f860c137b30 con 0x7f860c13d060 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8608001910 con 0x7f860c13d060 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8604001e00 con 0x7f860c13d060 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:47.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient(hunting): dump: 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2319570266 0 0) 0x7f860c06dfd0 con 0x7f860c0a1b10 2024-09-16T04:20:47.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 >> v1:172.21.15.148:6790/0 conn(0x7f860c0a2410 legacy=0x7f860c1d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 >> v1:172.21.15.171:6789/0 conn(0x7f860c13d060 legacy=0x7f860c1d4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 monclient: found mon.a 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c137e10 con 0x7f860c0a1b10 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient: _finish_auth 0 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f8601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 5 monclient: authenticate success, global_id 114264 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient: _renew_subs 2024-09-16T04:20:47.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.686+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f860c09f250 con 0x7f860c0a1b10 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8608004a90 con 0x7f860c0a1b10 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8608004de0 con 0x7f860c0a1b10 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 10 monclient: dump: 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:47.709 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f86037fe640 4 set_mon_vals no callback set 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f86136f1640 10 monclient: _renew_subs 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f86136f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f860c13c650 con 0x7f860c0a1b10 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.687+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8608010070 con 0x7f860c0a1b10 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.688+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f8608047aa0 con 0x7f860c0a1b10 2024-09-16T04:20:47.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.688+0000 7f86136f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.688+0000 7f86136f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.688+0000 7f86136f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.688+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c140520 con 0x7f860c0a1b10 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.688+0000 7f8610c8c640 10 monclient: get_auth_request con 0x7f85f00390d0 auth_method 0 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.692+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f860c13cb90 con 0x7f860c0a1b10 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.692+0000 7f8601ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:47.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.692+0000 7f8601ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:47.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.923+0000 7f86136f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:20:47.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.923+0000 7f86136f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:47.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.923+0000 7f86136f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:47.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.923+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f860c0a46a0 con 0x7f860c0a1b10 2024-09-16T04:20:47.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.924+0000 7f8601ffb640 1 -- 172.21.15.148:0/4093156541 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3720578745) 0x7f8608015ca0 con 0x7f860c0a1b10 2024-09-16T04:20:47.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.924+0000 7f8601ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:47.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.924+0000 7f8601ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:47.927 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 >> v1:172.21.15.171:6812/7327273 conn(0x7f85f00390d0 legacy=0x7f85f003b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 10 monclient: shutdown 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 >> v1:172.21.15.148:6789/0 conn(0x7f860c0a1b10 legacy=0x7f860c1328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 shutdown_connections 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 >> 172.21.15.148:0/4093156541 conn(0x7f860c0a97b0 msgr2=0x7f860c0a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 shutdown_connections 2024-09-16T04:20:47.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:47.926+0000 7f86136f1640 1 -- 172.21.15.148:0/4093156541 wait complete. 2024-09-16T04:20:47.939 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:20:47.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:20:47.939 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:20:48.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient: get_monmap_and_config 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient: monmap: 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:48.038148+0000 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:48.038148+0000 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:48.039 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 1 Processor -- start 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 1 -- start start 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient: init 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d40a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d40a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d40a0e00 con 0x7fc3d413d060 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d40a10e0 con 0x7fc3d40a2410 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d40a4970 con 0x7fc3d40a1b10 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d8c94640 10 monclient(hunting): authenticate will time out at 1668.663496s 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d1d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc3d40a1b10 0x7fc3d413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60462/0 (socket says 172.21.15.148:60462) 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d1d9b640 1 -- 172.21.15.148:0/2404769017 learned_addr learned my addr 172.21.15.148:0/2404769017 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d2d9d640 1 --1- 172.21.15.148:0/2404769017 >> v1:172.21.15.148:6789/0 conn(0x7fc3d413d060 0x7fc3d413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48300/0 (socket says 172.21.15.148:48300) 2024-09-16T04:20:48.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655345682 0 0) 0x7fc3d40a4970 con 0x7fc3d40a1b10 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): my global_id is 124124 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c0003520 con 0x7fc3d40a1b10 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716643922 0 0) 0x7fc3d40a0e00 con 0x7fc3d413d060 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): my global_id is 114267 2024-09-16T04:20:48.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d40a4970 con 0x7fc3d413d060 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953941622 0 0) 0x7fc3d40a10e0 con 0x7fc3d40a2410 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 10 monclient(hunting): my global_id is 6940 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.037+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d40a0e00 con 0x7fc3d40a2410 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1304919204 0 0) 0x7fc3d40a4970 con 0x7fc3d413d060 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 >> v1:172.21.15.148:6790/0 conn(0x7fc3d40a1b10 legacy=0x7fc3d413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 >> v1:172.21.15.171:6789/0 conn(0x7fc3d40a2410 legacy=0x7fc3d40a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 monclient: found mon.noname-a 2024-09-16T04:20:48.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d406dfd0 con 0x7fc3d413d060 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 5 monclient: authenticate success, global_id 114267 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3cc0018f0 con 0x7fc3d413d060 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: dump: 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.055 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3cc001bd0 con 0x7fc3d413d060 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 4 set_mon_vals no callback set 2024-09-16T04:20:48.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 1 -- 172.21.15.148:0/2404769017 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3cc001fe0 con 0x7fc3d413d060 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d159a640 10 monclient: dump: 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.057 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 10 monclient: shutdown 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 1 -- 172.21.15.148:0/2404769017 >> v1:172.21.15.148:6789/0 conn(0x7fc3d413d060 legacy=0x7fc3d413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 1 -- 172.21.15.148:0/2404769017 shutdown_connections 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 1 -- 172.21.15.148:0/2404769017 >> 172.21.15.148:0/2404769017 conn(0x7fc3d40a97b0 msgr2=0x7fc3d40a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 1 -- 172.21.15.148:0/2404769017 shutdown_connections 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.038+0000 7fc3d8c94640 1 -- 172.21.15.148:0/2404769017 wait complete. 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.039+0000 7fc3d8c94640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.039+0000 7fc3d8c94640 10 monclient: monmap: 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.039+0000 7fc3d8c94640 1 Processor -- start 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.039+0000 7fc3d8c94640 1 -- start start 2024-09-16T04:20:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.039+0000 7fc3d8c94640 10 monclient: init 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d40a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d40a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d4137c80 con 0x7fc3d40a1b10 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d4137f60 con 0x7fc3d40a2410 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d413cd60 con 0x7fc3d413d060 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d8c94640 10 monclient(hunting): authenticate will time out at 1668.666405s 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc3d40a1b10 0x7fc3d409d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48316/0 (socket says 172.21.15.148:48316) 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3d259c640 1 -- 172.21.15.148:0/727707298 learned_addr learned my addr 172.21.15.148:0/727707298 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859813913 0 0) 0x7fc3d413cd60 con 0x7fc3d413d060 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): my global_id is 124127 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b4003550 con 0x7fc3d413d060 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047144686 0 0) 0x7fc3d4137c80 con 0x7fc3d40a1b10 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): my global_id is 114270 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d413cd60 con 0x7fc3d40a1b10 2024-09-16T04:20:48.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1304140309 0 0) 0x7fc3d4137f60 con 0x7fc3d40a2410 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): my global_id is 6943 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d4137c80 con 0x7fc3d40a2410 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2539632213 0 0) 0x7fc3b4003550 con 0x7fc3d413d060 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3d4137f60 con 0x7fc3d413d060 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3cc001d80 con 0x7fc3d413d060 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): dump: 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.062 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2066571105 0 0) 0x7fc3d413cd60 con 0x7fc3d40a1b10 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3b4003550 con 0x7fc3d40a1b10 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c40019d0 con 0x7fc3d40a1b10 2024-09-16T04:20:48.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 10 monclient(hunting): dump: 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3464687270 0 0) 0x7fc3d4137c80 con 0x7fc3d40a2410 2024-09-16T04:20:48.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.040+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3d413cd60 con 0x7fc3d40a2410 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3bc001890 con 0x7fc3d40a2410 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient(hunting): dump: 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.065 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4038648033 0 0) 0x7fc3d4137f60 con 0x7fc3d413d060 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.171:6789/0 conn(0x7fc3d40a2410 legacy=0x7fc3d409d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3d40a1b10 legacy=0x7fc3d409d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 monclient: found mon.c 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d40a0130 con 0x7fc3d413d060 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 5 monclient: authenticate success, global_id 124127 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 10 monclient: _renew_subs 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d409e950 con 0x7fc3d413d060 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 10 monclient: _renew_subs 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3d413c710 con 0x7fc3d413d060 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc3cc001bd0 con 0x7fc3d413d060 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3cc004fb0 con 0x7fc3d413d060 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:48.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d0d99640 4 set_mon_vals no callback set 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3caffd640 10 monclient: dump: 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.041+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3b80065c0 con 0x7fc3b8002860 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc3cc004fb0 con 0x7fc3d413d060 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fc3cc0479e0 con 0x7fc3d413d060 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942151822 0 0) 0x7fc3b80065c0 con 0x7fc3b8002860 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): my global_id is 114273 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3d4137f60 con 0x7fc3b8002860 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3d1d9b640 10 monclient: get_auth_request con 0x7fc3b4039230 auth_method 0 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3d259c640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8002860 legacy=0x7fc3b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3d259c640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8002860 legacy=0x7fc3b8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:20:48.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3d259c640 1 --1- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8002860 0x7fc3b8004d00 :-1 s=OPENED pgs=563 cs=1 l=1).handle_message read tag failed 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3d259c640 1 --1- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8002860 0x7fc3b8004d00 :-1 s=OPENED pgs=563 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8002860 legacy=0x7fc3b8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8002860 legacy=0x7fc3b8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3b80065c0 con 0x7fc3b4042a00 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1512320187 0 0) 0x7fc3b80065c0 con 0x7fc3b4042a00 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient(hunting): my global_id is 114276 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b4003550 con 0x7fc3b4042a00 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4227829483 0 0) 0x7fc3b4003550 con 0x7fc3b4042a00 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3b80065c0 con 0x7fc3b4042a00 2024-09-16T04:20:48.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c4003b40 con 0x7fc3b4042a00 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.042+0000 7fc3caffd640 10 monclient: dump: 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3841365906 0 0) 0x7fc3b80065c0 con 0x7fc3b4042a00 2024-09-16T04:20:48.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3b40446a0 con 0x7fc3b4042a00 2024-09-16T04:20:48.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc3c4003eb0 con 0x7fc3b4042a00 2024-09-16T04:20:48.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:48.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b4042a00 legacy=0x7fc3b4042e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.043+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b4042a00 legacy=0x7fc3b4042e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3b8009da0 con 0x7fc3b8009140 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1544884493 0 0) 0x7fc3b8009da0 con 0x7fc3b8009140 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient(hunting): my global_id is 114279 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b80065c0 con 0x7fc3b8009140 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2680211274 0 0) 0x7fc3b80065c0 con 0x7fc3b8009140 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3b8009da0 con 0x7fc3b8009140 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc3c4004430 con 0x7fc3b8009140 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient: dump: 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3866216034 0 0) 0x7fc3b8009da0 con 0x7fc3b8009140 2024-09-16T04:20:48.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3b800c340 con 0x7fc3b8009140 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4146 (unknown 0 0 489545561) 0x7fc3c4007800 con 0x7fc3b8009140 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8009140 legacy=0x7fc3b80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3caffd640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6789/0 conn(0x7fc3b8009140 legacy=0x7fc3b80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d2d9d640 1 -- 172.21.15.148:0/727707298 reap_dead start 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:48.076 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 58, 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 143953, 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.077 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:20:48.078 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.079 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:20:48.080 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.081 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:48.082 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.083 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.084 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.085 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.086 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.087 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:20:48.088 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.171:6812/7327273 conn(0x7fc3b4039230 legacy=0x7fc3b403b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 10 monclient: shutdown 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 >> v1:172.21.15.148:6790/0 conn(0x7fc3d413d060 legacy=0x7fc3d41d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 shutdown_connections 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 >> 172.21.15.148:0/727707298 conn(0x7fc3d40a97b0 msgr2=0x7fc3d4135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 shutdown_connections 2024-09-16T04:20:48.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.050+0000 7fc3d8c94640 1 -- 172.21.15.148:0/727707298 wait complete. 2024-09-16T04:20:48.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:20:48.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:20:48.091 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=0 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=0 2024-09-16T04:20:48.094 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T04:20:48.095 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T04:20:48.095 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T04:20:48.190 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient: get_monmap_and_config 2024-09-16T04:20:48.190 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.190 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient: monmap: 2024-09-16T04:20:48.190 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.190 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:48.189426+0000 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:48.189426+0000 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 1 Processor -- start 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 1 -- start start 2024-09-16T04:20:48.191 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient: init 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient(hunting): picked mon.noname-a con 0x7f296009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient(hunting): picked mon.noname-c con 0x7f296009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient(hunting): picked mon.noname-b con 0x7f296009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29600a1010 con 0x7f296009bb70 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009f030 con 0x7f296009c660 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009dc60 con 0x7f296009cc30 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2967d8d640 10 monclient(hunting): authenticate will time out at 1668.814754s 2024-09-16T04:20:48.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2965b29640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f296009bb70 0x7f296009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48342/0 (socket says 172.21.15.148:48342) 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.188+0000 7f2965b29640 1 -- 172.21.15.148:0/3462441850 learned_addr learned my addr 172.21.15.148:0/3462441850 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2797336941 0 0) 0x7f29600a1010 con 0x7f296009bb70 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): my global_id is 114288 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2948003520 con 0x7f296009bb70 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659818631 0 0) 0x7f296009dc60 con 0x7f296009cc30 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): my global_id is 124136 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29600a1010 con 0x7f296009cc30 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 178466577 0 0) 0x7f296009f030 con 0x7f296009c660 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient(hunting): my global_id is 6952 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296009dc60 con 0x7f296009c660 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2400593405 0 0) 0x7f2948003520 con 0x7f296009bb70 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 >> v1:172.21.15.148:6790/0 conn(0x7f296009cc30 legacy=0x7f296013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 >> v1:172.21.15.171:6789/0 conn(0x7f296009c660 legacy=0x7f2960138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 monclient: found mon.noname-a 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296009df40 con 0x7f296009bb70 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2958001610 con 0x7f296009bb70 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2967d8d640 5 monclient: authenticate success, global_id 114288 2024-09-16T04:20:48.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2967d8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: dump: 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.206 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2967d8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29580018c0 con 0x7f296009bb70 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 1 -- 172.21.15.148:0/3462441850 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2958001c90 con 0x7f296009bb70 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2964b27640 10 monclient: dump: 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.207 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2967d8d640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.189+0000 7f2967d8d640 4 set_mon_vals no callback set 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 10 monclient: shutdown 2024-09-16T04:20:48.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 1 -- 172.21.15.148:0/3462441850 >> v1:172.21.15.148:6789/0 conn(0x7f296009bb70 legacy=0x7f296009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 1 -- 172.21.15.148:0/3462441850 shutdown_connections 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 1 -- 172.21.15.148:0/3462441850 >> 172.21.15.148:0/3462441850 conn(0x7f29600a5640 msgr2=0x7f29600a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 1 -- 172.21.15.148:0/3462441850 shutdown_connections 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 1 -- 172.21.15.148:0/3462441850 wait complete. 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 10 monclient: monmap: 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:48.209 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.190+0000 7f2967d8d640 1 Processor -- start 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 1 -- start start 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient: init 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:20:48.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient(hunting): picked mon.noname-c con 0x7f296009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient(hunting): picked mon.noname-b con 0x7f296009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient(hunting): picked mon.noname-a con 0x7f296009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296006e600 con 0x7f296009cc30 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296011c8f0 con 0x7f296009c660 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296011cbd0 con 0x7f296009bb70 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2967d8d640 10 monclient(hunting): authenticate will time out at 1668.817592s 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2965b29640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f296009bb70 0x7f296013b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60484/0 (socket says 172.21.15.148:60484) 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f2965b29640 1 -- 172.21.15.148:0/728337042 learned_addr learned my addr 172.21.15.148:0/728337042 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 103207699 0 0) 0x7f296011cbd0 con 0x7f296009bb70 2024-09-16T04:20:48.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): my global_id is 124139 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c003520 con 0x7f296009bb70 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278917277 0 0) 0x7f29600a1010 con 0x7f296009cc30 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): my global_id is 114291 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296011cbd0 con 0x7f296009cc30 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562548279 0 0) 0x7f296011c8f0 con 0x7f296009c660 2024-09-16T04:20:48.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 10 monclient(hunting): my global_id is 6955 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.191+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29600a1010 con 0x7f296009c660 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 964080935 0 0) 0x7f294c003520 con 0x7f296009bb70 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f296011c8f0 con 0x7f296009bb70 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3485232686 0 0) 0x7f29600a1010 con 0x7f296009c660 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f294c003520 con 0x7f296009c660 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3867108048 0 0) 0x7f296011cbd0 con 0x7f296009cc30 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29600a1010 con 0x7f296009cc30 2024-09-16T04:20:48.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2958001c10 con 0x7f296009bb70 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.214 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2954001830 con 0x7f296009c660 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.215 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2950001bb0 con 0x7f296009cc30 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient(hunting): dump: 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.216 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1665767217 0 0) 0x7f296011c8f0 con 0x7f296009bb70 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.171:6789/0 conn(0x7f296009c660 legacy=0x7f296013bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.148:6789/0 conn(0x7f296009cc30 legacy=0x7f29601d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 monclient: found mon.c 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296013c360 con 0x7f296009bb70 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 5 monclient: authenticate success, global_id 124139 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 10 monclient: _renew_subs 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f296009b8a0 con 0x7f296009bb70 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 10 monclient: _renew_subs 2024-09-16T04:20:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f296009ad00 con 0x7f296009bb70 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2958004ba0 con 0x7f296009bb70 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2958004ef0 con 0x7f296009bb70 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295e7fc640 10 monclient: dump: 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.192+0000 7f295ffff640 4 set_mon_vals no callback set 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f29580103e0 con 0x7f296009bb70 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f2967d8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f296011bfa0 con 0x7f296002d7c0 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f29580478b0 con 0x7f296009bb70 2024-09-16T04:20:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f2965328640 10 monclient: get_auth_request con 0x7f294c039140 auth_method 0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965583932 0 0) 0x7f296011bfa0 con 0x7f296002d7c0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient(hunting): my global_id is 124142 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296011c8f0 con 0x7f296002d7c0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1203645145 0 0) 0x7f296011c8f0 con 0x7f296002d7c0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f296011bfa0 con 0x7f296002d7c0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2950003d00 con 0x7f296002d7c0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 10 monclient: dump: 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3987658472 0 0) 0x7f296011bfa0 con 0x7f296002d7c0 2024-09-16T04:20:48.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.193+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f296002dba0 con 0x7f296002d7c0 2024-09-16T04:20:48.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.194+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2950005280 con 0x7f296002d7c0 2024-09-16T04:20:48.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.194+0000 7f295e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.194+0000 7f295e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:48.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.194+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.148:6790/0 conn(0x7f296002d7c0 legacy=0x7f2960068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.194+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.148:6790/0 conn(0x7f296002d7c0 legacy=0x7f2960068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f2967d8d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f296006bc20 con 0x7f2960135950 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016728191 0 0) 0x7f296006bc20 con 0x7f2960135950 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient(hunting): my global_id is 124145 2024-09-16T04:20:48.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296011bfa0 con 0x7f2960135950 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2994091313 0 0) 0x7f296011bfa0 con 0x7f2960135950 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f296006bc20 con 0x7f2960135950 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2950003f70 con 0x7f2960135950 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 10 monclient: dump: 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.226 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 658256026 0 0) 0x7f296006bc20 con 0x7f2960135950 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.200+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2960068890 con 0x7f2960135950 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 887759418) 0x7f2950005430 con 0x7f2960135950 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f295e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f295e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.148:6790/0 conn(0x7f2960135950 legacy=0x7f29601d7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f295e7fc640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.148:6790/0 conn(0x7f2960135950 legacy=0x7f29601d7250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 58, 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 15550, 2024-09-16T04:20:48.228 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:48.229 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.230 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.231 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:20:48.232 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:20:48.233 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:20:48.234 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:48.234 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:20:48.234 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.235 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.235 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.235 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.235 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.236 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:20:48.237 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:20:48.238 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:48.239 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.240 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.241 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:20:48.242 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:20:48.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.171:6812/7327273 conn(0x7f294c039140 legacy=0x7f294c03b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 10 monclient: shutdown 2024-09-16T04:20:48.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 >> v1:172.21.15.148:6790/0 conn(0x7f296009bb70 legacy=0x7f296013b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f296632a640 1 -- 172.21.15.148:0/728337042 reap_dead start 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f296632a640 1 -- 172.21.15.148:0/728337042 reap_dead start 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 shutdown_connections 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 >> 172.21.15.148:0/728337042 conn(0x7f29600a5640 msgr2=0x7f2960072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 shutdown_connections 2024-09-16T04:20:48.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.201+0000 7f2967d8d640 1 -- 172.21.15.148:0/728337042 wait complete. 2024-09-16T04:20:48.244 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:20:48.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:20:48.245 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:20:48.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.342+0000 7f6fefba6640 10 monclient: get_monmap_and_config 2024-09-16T04:20:48.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.342+0000 7f6fefba6640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.342+0000 7f6fefba6640 10 monclient: monmap: 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:48.343201+0000 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:48.343201+0000 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.345 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:48.346 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:48.346 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.342+0000 7f6fefba6640 1 Processor -- start 2024-09-16T04:20:48.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 1 -- start start 2024-09-16T04:20:48.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient: init 2024-09-16T04:20:48.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe80a1010 con 0x7f6fe809cc30 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe809f030 con 0x7f6fe809c660 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe809dc60 con 0x7f6fe809bb70 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fefba6640 10 monclient(hunting): authenticate will time out at 1668.968551s 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fee143640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6fe809cc30 0x7f6fe813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48358/0 (socket says 172.21.15.148:48358) 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fee143640 1 -- 172.21.15.148:0/3287071515 learned_addr learned my addr 172.21.15.148:0/3287071515 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fed942640 1 --1- 172.21.15.148:0/3287071515 >> v1:172.21.15.148:6790/0 conn(0x7f6fe809bb70 0x7f6fe809bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60516/0 (socket says 172.21.15.148:60516) 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728858219 0 0) 0x7f6fe80a1010 con 0x7f6fe809cc30 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): my global_id is 114294 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd4003520 con 0x7f6fe809cc30 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2444655137 0 0) 0x7f6fe809dc60 con 0x7f6fe809bb70 2024-09-16T04:20:48.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 10 monclient(hunting): my global_id is 124148 2024-09-16T04:20:48.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.343+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe80a1010 con 0x7f6fe809bb70 2024-09-16T04:20:48.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3115895717 0 0) 0x7f6fe809f030 con 0x7f6fe809c660 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient(hunting): my global_id is 6958 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe809dc60 con 0x7f6fe809c660 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1918375029 0 0) 0x7f6fd4003520 con 0x7f6fe809cc30 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 >> v1:172.21.15.148:6790/0 conn(0x7f6fe809bb70 legacy=0x7f6fe809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 >> v1:172.21.15.171:6789/0 conn(0x7f6fe809c660 legacy=0x7f6fe8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 monclient: found mon.noname-a 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe809df40 con 0x7f6fe809cc30 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fe00018c0 con 0x7f6fe809cc30 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 5 monclient: authenticate success, global_id 114294 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: dump: 2024-09-16T04:20:48.360 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.361 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fe0001af0 con 0x7f6fe809cc30 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 1 -- 172.21.15.148:0/3287071515 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fe0001f00 con 0x7f6fe809cc30 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 4 set_mon_vals no callback set 2024-09-16T04:20:48.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fec940640 10 monclient: dump: 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 10 monclient: shutdown 2024-09-16T04:20:48.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 1 -- 172.21.15.148:0/3287071515 >> v1:172.21.15.148:6789/0 conn(0x7f6fe809cc30 legacy=0x7f6fe813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 1 -- 172.21.15.148:0/3287071515 shutdown_connections 2024-09-16T04:20:48.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 1 -- 172.21.15.148:0/3287071515 >> 172.21.15.148:0/3287071515 conn(0x7f6fe80a5640 msgr2=0x7f6fe80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 1 -- 172.21.15.148:0/3287071515 shutdown_connections 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.344+0000 7f6fefba6640 1 -- 172.21.15.148:0/3287071515 wait complete. 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.345+0000 7f6fefba6640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.345+0000 7f6fefba6640 10 monclient: monmap: 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:48.365 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.345+0000 7f6fefba6640 1 Processor -- start 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.345+0000 7f6fefba6640 1 -- start start 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.345+0000 7f6fefba6640 10 monclient: init 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe813e290 con 0x7f6fe809c660 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe8141630 con 0x7f6fe809bb70 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe80740a0 con 0x7f6fe809cc30 2024-09-16T04:20:48.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fefba6640 10 monclient(hunting): authenticate will time out at 1668.971223s 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fed141640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f6fe809c660 0x7f6fe8140f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48360/0 (socket says 172.21.15.148:48360) 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fed141640 1 -- 172.21.15.148:0/1365926436 learned_addr learned my addr 172.21.15.148:0/1365926436 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4260933151 0 0) 0x7f6fe80740a0 con 0x7f6fe809cc30 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): my global_id is 124151 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fcc003550 con 0x7f6fe809cc30 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3163505440 0 0) 0x7f6fe813e290 con 0x7f6fe809c660 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): my global_id is 114297 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe80740a0 con 0x7f6fe809c660 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655343499 0 0) 0x7f6fe8141630 con 0x7f6fe809bb70 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): my global_id is 6961 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe813e290 con 0x7f6fe809bb70 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 687257834 0 0) 0x7f6fcc003550 con 0x7f6fe809cc30 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe8141630 con 0x7f6fe809cc30 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fe0001e40 con 0x7f6fe809cc30 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): dump: 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:48.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 753183564 0 0) 0x7f6fe80740a0 con 0x7f6fe809c660 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fcc003550 con 0x7f6fe809c660 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fe4001870 con 0x7f6fe809c660 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): dump: 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.370 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1870833244 0 0) 0x7f6fe813e290 con 0x7f6fe809bb70 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fe80740a0 con 0x7f6fe809bb70 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fdc001250 con 0x7f6fe809bb70 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient(hunting): dump: 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.371 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 736194148 0 0) 0x7f6fe8141630 con 0x7f6fe809cc30 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.171:6789/0 conn(0x7f6fe809bb70 legacy=0x7f6fe8140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.148:6789/0 conn(0x7f6fe809c660 legacy=0x7f6fe8140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.346+0000 7f6fda7fc640 1 monclient: found mon.c 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe81424b0 con 0x7f6fe809cc30 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 5 monclient: authenticate success, global_id 124151 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 10 monclient: _renew_subs 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe813dfb0 con 0x7f6fe809cc30 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 10 monclient: _renew_subs 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6fe813f9b0 con 0x7f6fe809cc30 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6fe0001af0 con 0x7f6fe809cc30 2024-09-16T04:20:48.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fe0004e10 con 0x7f6fe809cc30 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fdbfff640 4 set_mon_vals no callback set 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 10 monclient: dump: 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.374 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6fe0004e10 con 0x7f6fe809cc30 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.347+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fd00065c0 con 0x7f6fd0002860 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f6fe0047a10 con 0x7f6fe809cc30 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fed141640 10 monclient: get_auth_request con 0x7f6fcc039100 auth_method 0 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169764352 0 0) 0x7f6fd00065c0 con 0x7f6fd0002860 2024-09-16T04:20:48.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient(hunting): my global_id is 6964 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe8141630 con 0x7f6fd0002860 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2473841487 0 0) 0x7f6fe8141630 con 0x7f6fd0002860 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd00065c0 con 0x7f6fd0002860 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fdc004980 con 0x7f6fd0002860 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 10 monclient: dump: 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.376 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 704273045 0 0) 0x7f6fd00065c0 con 0x7f6fd0002860 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.348+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fd000a150 con 0x7f6fd0002860 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.349+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6fdc004b70 con 0x7f6fd0002860 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.349+0000 7f6fda7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.349+0000 7f6fda7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:48.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.349+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.171:6789/0 conn(0x7f6fd0002860 legacy=0x7f6fd0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.349+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.171:6789/0 conn(0x7f6fd0002860 legacy=0x7f6fd0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fefba6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6fd0009e40 con 0x7f6fd0009100 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1185757940 0 0) 0x7f6fd0009e40 con 0x7f6fd0009100 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient(hunting): my global_id is 6967 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd00065c0 con 0x7f6fd0009100 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2618724035 0 0) 0x7f6fd00065c0 con 0x7f6fd0009100 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fd0009e40 con 0x7f6fd0009100 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6fdc0051b0 con 0x7f6fd0009100 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 10 monclient: dump: 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4009481375 0 0) 0x7f6fd0009e40 con 0x7f6fd0009100 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.355+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fd000c3d0 con 0x7f6fd0009100 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 2949094785) 0x7f6fdc005390 con 0x7f6fd0009100 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fda7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fda7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.171:6789/0 conn(0x7f6fd0009100 legacy=0x7f6fd0009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fda7fc640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.171:6789/0 conn(0x7f6fd0009100 legacy=0x7f6fd0009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:48.382 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 58, 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 872394, 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.383 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.384 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.385 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:20:48.386 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:48.387 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:48.388 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.389 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.390 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.391 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:20:48.392 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.393 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:48.394 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:20:48.395 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:20:48.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.171:6812/7327273 conn(0x7f6fcc039100 legacy=0x7f6fcc03b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 10 monclient: shutdown 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 >> v1:172.21.15.148:6790/0 conn(0x7f6fe809cc30 legacy=0x7f6fe81d3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fee143640 1 -- 172.21.15.148:0/1365926436 reap_dead start 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fee143640 1 -- 172.21.15.148:0/1365926436 reap_dead start 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 shutdown_connections 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 >> 172.21.15.148:0/1365926436 conn(0x7f6fe80a5640 msgr2=0x7f6fe8072a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 shutdown_connections 2024-09-16T04:20:48.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.356+0000 7f6fefba6640 1 -- 172.21.15.148:0/1365926436 wait complete. 2024-09-16T04:20:48.398 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:20:48.398 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:20:48.399 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:20:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.494+0000 7f855b9d8640 10 monclient: get_monmap_and_config 2024-09-16T04:20:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.494+0000 7f855b9d8640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.494+0000 7f855b9d8640 10 monclient: monmap: 2024-09-16T04:20:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.496 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:48.495591+0000 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:48.495591+0000 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:48.497 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 1 Processor -- start 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 1 -- start start 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 10 monclient: init 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 10 monclient(hunting): picked mon.noname-a con 0x7f855409bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 10 monclient(hunting): picked mon.noname-c con 0x7f855409c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 10 monclient(hunting): picked mon.noname-b con 0x7f855409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85540a1010 con 0x7f855409bb70 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.495+0000 7f855b9d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855409f030 con 0x7f855409c660 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f855409dc60 con 0x7f855409cc30 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 10 monclient(hunting): authenticate will time out at 1669.121065s 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f8559774640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f855409bb70 0x7f855409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48368/0 (socket says 172.21.15.148:48368) 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f8559774640 1 -- 172.21.15.148:0/3406675354 learned_addr learned my addr 172.21.15.148:0/3406675354 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104472451 0 0) 0x7f85540a1010 con 0x7f855409bb70 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): my global_id is 114300 2024-09-16T04:20:48.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8540003520 con 0x7f855409bb70 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660345109 0 0) 0x7f855409dc60 con 0x7f855409cc30 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): my global_id is 124154 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85540a1010 con 0x7f855409cc30 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729832440 0 0) 0x7f855409f030 con 0x7f855409c660 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient(hunting): my global_id is 6970 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f855409dc60 con 0x7f855409c660 2024-09-16T04:20:48.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2772187986 0 0) 0x7f8540003520 con 0x7f855409bb70 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 >> v1:172.21.15.148:6790/0 conn(0x7f855409cc30 legacy=0x7f855413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 >> v1:172.21.15.171:6789/0 conn(0x7f855409c660 legacy=0x7f8554138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 monclient: found mon.noname-a 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855409df40 con 0x7f855409bb70 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 5 monclient: authenticate success, global_id 114300 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8550001610 con 0x7f855409bb70 2024-09-16T04:20:48.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: dump: 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.509 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f85500018c0 con 0x7f855409bb70 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 1 -- 172.21.15.148:0/3406675354 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8550001cd0 con 0x7f855409bb70 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f855b9d8640 4 set_mon_vals no callback set 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.496+0000 7f854bfff640 10 monclient: dump: 2024-09-16T04:20:48.510 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 10 monclient: shutdown 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 1 -- 172.21.15.148:0/3406675354 >> v1:172.21.15.148:6789/0 conn(0x7f855409bb70 legacy=0x7f855409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 1 -- 172.21.15.148:0/3406675354 shutdown_connections 2024-09-16T04:20:48.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 1 -- 172.21.15.148:0/3406675354 >> 172.21.15.148:0/3406675354 conn(0x7f85540a5640 msgr2=0x7f85540a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 1 -- 172.21.15.148:0/3406675354 shutdown_connections 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 1 -- 172.21.15.148:0/3406675354 wait complete. 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 10 monclient: build_initial_monmap 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.497+0000 7f855b9d8640 10 monclient: monmap: 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:48.512 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 1 Processor -- start 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 1 -- start start 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient: init 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient(hunting): picked mon.noname-c con 0x7f855409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient(hunting): picked mon.noname-a con 0x7f855409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient(hunting): picked mon.noname-b con 0x7f855409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8554120040 con 0x7f855409c660 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85540a3f00 con 0x7f855409cc30 2024-09-16T04:20:48.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85540740a0 con 0x7f855409bb70 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f855b9d8640 10 monclient(hunting): authenticate will time out at 1669.123809s 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f8559774640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f855409bb70 0x7f8554131670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60552/0 (socket says 172.21.15.148:60552) 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.498+0000 7f8559774640 1 -- 172.21.15.148:0/967715377 learned_addr learned my addr 172.21.15.148:0/967715377 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893796604 0 0) 0x7f85540740a0 con 0x7f855409bb70 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): my global_id is 124157 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8538003520 con 0x7f855409bb70 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 880349101 0 0) 0x7f8554120040 con 0x7f855409c660 2024-09-16T04:20:48.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): my global_id is 114303 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85540740a0 con 0x7f855409c660 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313788487 0 0) 0x7f85540a3f00 con 0x7f855409cc30 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): my global_id is 6973 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8554120040 con 0x7f855409cc30 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 581177643 0 0) 0x7f8538003520 con 0x7f855409bb70 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85540a3f00 con 0x7f855409bb70 2024-09-16T04:20:48.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2251367203 0 0) 0x7f85540740a0 con 0x7f855409c660 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8538003520 con 0x7f855409c660 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8550001c10 con 0x7f855409bb70 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): dump: 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.516 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8544001830 con 0x7f855409c660 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): dump: 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.517 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1249896159 0 0) 0x7f8554120040 con 0x7f855409cc30 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85540740a0 con 0x7f855409cc30 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f854c001d10 con 0x7f855409cc30 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient(hunting): dump: 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.518 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3047352599 0 0) 0x7f85540a3f00 con 0x7f855409bb70 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 >> v1:172.21.15.171:6789/0 conn(0x7f855409cc30 legacy=0x7f85541cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 >> v1:172.21.15.148:6789/0 conn(0x7f855409c660 legacy=0x7f85541cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 monclient: found mon.c 2024-09-16T04:20:48.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855411dd50 con 0x7f855409bb70 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: _finish_auth 0 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 5 monclient: authenticate success, global_id 124157 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 10 monclient: _renew_subs 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f855411fa80 con 0x7f855409bb70 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8550004cc0 con 0x7f855409bb70 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8550005010 con 0x7f855409bb70 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:48.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f8549ffb640 10 monclient: dump: 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f854b7fe640 4 set_mon_vals no callback set 2024-09-16T04:20:48.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 10 monclient: _renew_subs 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.499+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f855411e630 con 0x7f855409bb70 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f85500102a0 con 0x7f855409bb70 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f85500477f0 con 0x7f855409bb70 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f855b9d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f855b9d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f855b9d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f855406fd40 con 0x7f855409bb70 2024-09-16T04:20:48.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.500+0000 7f8558f73640 10 monclient: get_auth_request con 0x7f8538039120 auth_method 0 2024-09-16T04:20:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.505+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8550015b90 con 0x7f855409bb70 2024-09-16T04:20:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.505+0000 7f8549ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:48.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.505+0000 7f8549ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:48.582 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=0 2024-09-16T04:20:48.582 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 0 -ne 0 ]] 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T04:20:48.583 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-16T04:20:48.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.737+0000 7f855b9d8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:20:48.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.737+0000 7f855b9d8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:20:48.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.737+0000 7f855b9d8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:48.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.737+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f855406bc20 con 0x7f855409bb70 2024-09-16T04:20:48.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.738+0000 7f8549ffb640 1 -- 172.21.15.148:0/967715377 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f855001e040 con 0x7f855409bb70 2024-09-16T04:20:48.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.738+0000 7f8549ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:20:48.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.738+0000 7f8549ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:48.740 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.738+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.740 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.738+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3472430111,config=692619927,health=2313241988,kv=2270513034,logm=2139078019}) 2024-09-16T04:20:48.740 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.738+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=1,health=11,kv=10,logm=51} crc {auth=3475647933,config=1228700967,health=2313241988,kv=2270513034,logm=287357295}) 2024-09-16T04:20:48.740 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.738+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.741 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.738+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3472430111,config=692619927,health=2313241988,kv=2270513034,logm=2139078019}) 2024-09-16T04:20:48.741 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.738+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3697938807,config=692619927,health=1555756363,kv=2096439852,logm=3963561584}) 2024-09-16T04:20:48.741 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.739+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.741 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.739+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792864438}) 2024-09-16T04:20:48.741 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.739+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108073841}) 2024-09-16T04:20:48.742 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.739+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.742 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.739+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792864438}) 2024-09-16T04:20:48.742 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.739+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844672761}) 2024-09-16T04:20:48.742 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.740+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.742 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.740+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115275226}) 2024-09-16T04:20:48.742 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.740+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346558270}) 2024-09-16T04:20:48.743 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.740+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.743 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.740+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115275226}) 2024-09-16T04:20:48.743 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.740+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403912337}) 2024-09-16T04:20:48.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 >> v1:172.21.15.171:6812/7327273 conn(0x7f8538039120 legacy=0x7f853803b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 10 monclient: shutdown 2024-09-16T04:20:48.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:48.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 >> v1:172.21.15.148:6790/0 conn(0x7f855409bb70 legacy=0x7f8554131670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:48.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 shutdown_connections 2024-09-16T04:20:48.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 >> 172.21.15.148:0/967715377 conn(0x7f85540a5640 msgr2=0x7f85540a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:48.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 shutdown_connections 2024-09-16T04:20:48.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:48.741+0000 7f855b9d8640 1 -- 172.21.15.148:0/967715377 wait complete. 2024-09-16T04:20:48.747 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.741+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.747 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.741+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009194353}) 2024-09-16T04:20:48.747 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.741+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034877104}) 2024-09-16T04:20:48.747 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.741+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.747 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.741+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009194353}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.741+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3911772986}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.742+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.742+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506506602}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.742+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071153648}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.742+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.742+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506506602}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.742+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717818514}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624885498}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763392586}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624885498}) 2024-09-16T04:20:48.748 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179363226}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950237455}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396293617}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950237455}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.743+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1439485571}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.744+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.744+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1219575304}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.744+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780175848}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.744+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.744+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1219575304}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.744+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285762065}) 2024-09-16T04:20:48.749 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355778036}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1815739349}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355778036}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1486393205}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467682336}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989484068}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467682336}) 2024-09-16T04:20:48.750 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.745+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3393583715}) 2024-09-16T04:20:48.754 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.746+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.754 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.746+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998545546}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.746+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728902241}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.746+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.746+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998545546}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.746+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598479182}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075971469}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53179106}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075971469}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881782236}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286486066}) 2024-09-16T04:20:48.755 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024710784}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286486066}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.747+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3486097253}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.748+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.748+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595305454}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.748+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466395270}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.748+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.748+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595305454}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.748+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392811835}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303836618}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639500838}) 2024-09-16T04:20:48.756 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303836618}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3133121560}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761339956}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1732214652}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761339956}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.749+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466475794}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.750+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.750+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345447020}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.750+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523929036}) 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.750+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.757 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.750+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345447020}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.750+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260837983}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877961771}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339145241}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877961771}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523736137}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338757745}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748057064}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.751+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338757745}) 2024-09-16T04:20:48.758 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246461682}) 2024-09-16T04:20:48.759 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.759 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=465944121}) 2024-09-16T04:20:48.759 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113119036}) 2024-09-16T04:20:48.759 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.759 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=465944121}) 2024-09-16T04:20:48.759 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.752+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475635542}) 2024-09-16T04:20:48.760 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:20:48.766 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.753+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.766 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.753+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200225863}) 2024-09-16T04:20:48.766 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.753+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661059605}) 2024-09-16T04:20:48.766 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.753+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.753+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200225863}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.753+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098337989}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558633332}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323062122}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558633332}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2314010176}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394469313}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4150998175}) 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.767 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394469313}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.754+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766226369}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.755+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.755+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807949342}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.755+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2106916217}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.755+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.755+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807949342}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.755+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461151220}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1588810083}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350634182}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1588810083}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=40195117}) 2024-09-16T04:20:48.768 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878419105}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951799070}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878419105}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.756+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837803078}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.757+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.757+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29997446}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.757+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225866336}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.757+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.757+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29997446}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.757+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=101827182}) 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.769 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099805428}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4180763500}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099805428}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197869571}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949268285}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050005288}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949268285}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.758+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594111292}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156929056}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=431397087}) 2024-09-16T04:20:48.770 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156929056}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221420480}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819862922}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838465669}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819862922}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.759+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2166541754}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.760+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.760+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558222386}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.760+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=233939399}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.760+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.760+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558222386}) 2024-09-16T04:20:48.771 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.760+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226134555}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201271594}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155871010}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201271594}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324303042}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65569131}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996155456}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65569131}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.761+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365774380}) 2024-09-16T04:20:48.772 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.762+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.773 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.762+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106938682}) 2024-09-16T04:20:48.773 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.762+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2520537014}) 2024-09-16T04:20:48.773 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.762+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.762+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106938682}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.762+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728714429}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456476758}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1359949753}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456476758}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687117882}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914272592}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460684171}) 2024-09-16T04:20:48.774 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914272592}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.763+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659572345}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.764+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.764+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059881738}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.764+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3298062087}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.764+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.764+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059881738}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.764+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1373119258}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819325348}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047976229}) 2024-09-16T04:20:48.775 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.776 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819325348}) 2024-09-16T04:20:48.776 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122896518}) 2024-09-16T04:20:48.782 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.782 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228306285}) 2024-09-16T04:20:48.782 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2794165768}) 2024-09-16T04:20:48.783 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.783 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228306285}) 2024-09-16T04:20:48.783 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.765+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2488654137}) 2024-09-16T04:20:48.783 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.783 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447284165}) 2024-09-16T04:20:48.783 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123236057}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447284165}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=322187194}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218644966}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1003988938}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.766+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218644966}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2036086008}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385163188}) 2024-09-16T04:20:48.784 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619942811}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385163188}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.767+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544778139}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964119580}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412174749}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964119580}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035854254}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867910855}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3495334228}) 2024-09-16T04:20:48.785 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867910855}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.768+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2347374660}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.769+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.769+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290527191}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.769+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2712517819}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.769+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.769+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290527191}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.769+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679492129}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358446340}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2086444179}) 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.786 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358446340}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597624813}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231892973}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1065259030}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231892973}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.770+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1960754763}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.771+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.771+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000433590}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.771+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=412394052}) 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.771+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.787 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.771+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000433590}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.771+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866345604}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.772+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.772+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039574064}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.772+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706129105}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.772+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.772+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039574064}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.772+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3825280280}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.773+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.773+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=1} crc {logm=4189660518,mdsmap=2492720936,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2857002826,mon_config_key=2589496441,monmap=1290997623}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.773+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=4} crc {logm=2364113262,mdsmap=3140514009,mgr=3517470555,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2911909512,mon_config_key=2589496441,monmap=2633035914,osd_metadata=2495496720}) 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.773+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.788 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.773+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=1} crc {logm=4189660518,mdsmap=2492720936,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2857002826,mon_config_key=2589496441,monmap=1290997623}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.773+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9,monmap=3,osd_metadata=2} crc {logm=2927179404,mdsmap=3140514009,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3927595371,mon_config_key=2716198159,monmap=2633035914,osd_metadata=1235286560}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.774+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.774+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=825007398}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.774+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=1499490399}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.774+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.774+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=825007398}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.774+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=4007021953,osd_pg_creating=3462100867,osd_snap=841190635,osdmap=3328953329}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.775+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.775+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=1130724617}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.775+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=25} crc {osdmap=650649516}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.775+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.775+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=1130724617}) 2024-09-16T04:20:48.789 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:20:48.775+0000 7fd827406640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=28} crc {osdmap=1648193572}) 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:20:48.899 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T04:20:48.900 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T04:20:48.900 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-16T04:20:49.214 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:20:49.214 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:20:49.214 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:49.214 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-16T04:20:49.214 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-16T04:20:49.214 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T04:20:49.215 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:20:49.215 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T04:20:49.215 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-16T04:20:49.527 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:20:49.527 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:20:49.527 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:49.528 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-16T04:20:49.528 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-16T04:20:49.528 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:20:49.528 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T04:20:49.528 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T04:20:49.528 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-16T04:20:49.761 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:20:49.761 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:20:49.779 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=110 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T04:20:49.780 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T04:20:49.781 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-16T04:20:49.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.867+0000 7f61008f1640 10 monclient: get_monmap_and_config 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.867+0000 7f61008f1640 10 monclient: build_initial_monmap 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.867+0000 7f61008f1640 10 monclient: monmap: 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:49.868270+0000 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:49.868270+0000 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:49.869 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.867+0000 7f61008f1640 1 Processor -- start 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.867+0000 7f61008f1640 1 -- start start 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.867+0000 7f61008f1640 10 monclient: init 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient(hunting): picked mon.noname-a con 0x7f60fc133910 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient(hunting): picked mon.noname-b con 0x7f60fc06c430 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient(hunting): picked mon.noname-c con 0x7f60fc13d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:49.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60fc09b0d0 con 0x7f60fc133910 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60fc09b340 con 0x7f60fc13d0e0 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60fc09b620 con 0x7f60fc06c430 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f61008f1640 10 monclient(hunting): authenticate will time out at 1670.493657s 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f60f9d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f60fc06c430 0x7f60fc13b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60564/0 (socket says 172.21.15.148:60564) 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f60fa59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f60fc133910 0x7f60fc133cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48386/0 (socket says 172.21.15.148:48386) 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.868+0000 7f60f9d9b640 1 -- 172.21.15.148:0/2480968809 learned_addr learned my addr 172.21.15.148:0/2480968809 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292229623 0 0) 0x7f60fc09b0d0 con 0x7f60fc133910 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:49.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): my global_id is 114336 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60e4003520 con 0x7f60fc133910 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3079774007 0 0) 0x7f60fc09b620 con 0x7f60fc06c430 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): my global_id is 124190 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60fc09b0d0 con 0x7f60fc06c430 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922301949 0 0) 0x7f60fc09b340 con 0x7f60fc13d0e0 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:49.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient(hunting): my global_id is 7006 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60fc09b620 con 0x7f60fc13d0e0 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1689933172 0 0) 0x7f60fc09b0d0 con 0x7f60fc06c430 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 >> v1:172.21.15.171:6789/0 conn(0x7f60fc13d0e0 legacy=0x7f60fc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 >> v1:172.21.15.148:6789/0 conn(0x7f60fc133910 legacy=0x7f60fc133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: _finish_hunting 0 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 monclient: found mon.noname-b 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60fc0633c0 con 0x7f60fc06c430 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: _finish_auth 0 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: _check_auth_tickets 2024-09-16T04:20:49.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60f00015f0 con 0x7f60fc06c430 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 5 monclient: authenticate success, global_id 124190 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: dump: 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.882 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60f0001950 con 0x7f60fc06c430 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 4 set_mon_vals no callback set 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 1 -- 172.21.15.148:0/2480968809 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60f00048a0 con 0x7f60fc06c430 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f60f959a640 10 monclient: dump: 2024-09-16T04:20:49.883 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 10 monclient: shutdown 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.869+0000 7f61008f1640 1 -- 172.21.15.148:0/2480968809 >> v1:172.21.15.148:6790/0 conn(0x7f60fc06c430 legacy=0x7f60fc13b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.870+0000 7f61008f1640 1 -- 172.21.15.148:0/2480968809 shutdown_connections 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.870+0000 7f61008f1640 1 -- 172.21.15.148:0/2480968809 >> 172.21.15.148:0/2480968809 conn(0x7f60fc062630 msgr2=0x7f60fc062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.870+0000 7f61008f1640 1 -- 172.21.15.148:0/2480968809 shutdown_connections 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.870+0000 7f61008f1640 1 -- 172.21.15.148:0/2480968809 wait complete. 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.870+0000 7f61008f1640 10 monclient: build_initial_monmap 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.870+0000 7f61008f1640 10 monclient: monmap: 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:49.885 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 1 Processor -- start 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 1 -- start start 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient: init 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient(hunting): picked mon.noname-b con 0x7f60fc06c430 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient(hunting): picked mon.noname-c con 0x7f60fc133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:49.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient(hunting): picked mon.noname-a con 0x7f60fc13d0e0 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60fc132820 con 0x7f60fc13d0e0 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60fc1cbf40 con 0x7f60fc06c430 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60fc06e000 con 0x7f60fc133910 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f61008f1640 10 monclient(hunting): authenticate will time out at 1670.496809s 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f60f9d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f60fc133910 0x7f60fc1cb830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60576/0 (socket says 172.21.15.148:60576) 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f60f9d9b640 1 -- 172.21.15.148:0/38543874 learned_addr learned my addr 172.21.15.148:0/38543874 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.871+0000 7f60fad9d640 1 --1- 172.21.15.148:0/38543874 >> v1:172.21.15.148:6789/0 conn(0x7f60fc13d0e0 0x7f60fc1d36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48400/0 (socket says 172.21.15.148:48400) 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228604744 0 0) 0x7f60fc132820 con 0x7f60fc13d0e0 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:49.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): my global_id is 114339 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60dc003520 con 0x7f60fc13d0e0 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3227527495 0 0) 0x7f60fc06e000 con 0x7f60fc133910 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): my global_id is 124193 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60fc132820 con 0x7f60fc133910 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 622524813 0 0) 0x7f60fc1cbf40 con 0x7f60fc06c430 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): my global_id is 7009 2024-09-16T04:20:49.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60fc06e000 con 0x7f60fc06c430 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2524464187 0 0) 0x7f60fc06e000 con 0x7f60fc06c430 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60fc1cbf40 con 0x7f60fc06c430 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60ec001110 con 0x7f60fc06c430 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): dump: 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.889 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1181605661 0 0) 0x7f60fc132820 con 0x7f60fc133910 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60fc06e000 con 0x7f60fc133910 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3351931130 0 0) 0x7f60dc003520 con 0x7f60fc13d0e0 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60fc132820 con 0x7f60fc13d0e0 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60f0001b00 con 0x7f60fc133910 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): dump: 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:49.890 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60f4001bb0 con 0x7f60fc13d0e0 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient(hunting): dump: 2024-09-16T04:20:49.891 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4046346031 0 0) 0x7f60fc1cbf40 con 0x7f60fc06c430 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 >> v1:172.21.15.148:6790/0 conn(0x7f60fc133910 legacy=0x7f60fc1cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 >> v1:172.21.15.148:6789/0 conn(0x7f60fc13d0e0 legacy=0x7f60fc1d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:49.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 monclient: found mon.b 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60fc130530 con 0x7f60fc06c430 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: _finish_auth 0 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 5 monclient: authenticate success, global_id 7009 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 10 monclient: _renew_subs 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60fc132260 con 0x7f60fc06c430 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 10 monclient: _renew_subs 2024-09-16T04:20:49.893 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f60fc131220 con 0x7f60fc06c430 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60ec0047e0 con 0x7f60fc06c430 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f60ec004b30 con 0x7f60fc06c430 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60f8d99640 4 set_mon_vals no callback set 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.872+0000 7f60eaffd640 10 monclient: dump: 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:49.894 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.873+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f60ec00fdc0 con 0x7f60fc06c430 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.873+0000 7f61008f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.873+0000 7f61008f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.873+0000 7f61008f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.873+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e0005080 con 0x7f60fc06c430 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.873+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f60ec047770 con 0x7f60fc06c430 2024-09-16T04:20:49.895 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.874+0000 7f60f9d9b640 10 monclient: get_auth_request con 0x7f60dc039340 auth_method 0 2024-09-16T04:20:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.878+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f60ec015970 con 0x7f60fc06c430 2024-09-16T04:20:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.878+0000 7f60eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:49.896 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:49.878+0000 7f60eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.109+0000 7f61008f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.109+0000 7f61008f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.109+0000 7f61008f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.109+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60e0005370 con 0x7f60fc06c430 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.110+0000 7f60eaffd640 1 -- 172.21.15.148:0/38543874 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3392551720) 0x7f60ec015270 con 0x7f60fc06c430 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.110+0000 7f60eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:50.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.110+0000 7f60eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:50.114 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:20:50.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 >> v1:172.21.15.171:6812/7327273 conn(0x7f60dc039340 legacy=0x7f60dc03b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 10 monclient: shutdown 2024-09-16T04:20:50.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 >> v1:172.21.15.171:6789/0 conn(0x7f60fc06c430 legacy=0x7f60fc1cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 shutdown_connections 2024-09-16T04:20:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 >> 172.21.15.148:0/38543874 conn(0x7f60fc062630 msgr2=0x7f60fc09fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 shutdown_connections 2024-09-16T04:20:50.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.112+0000 7f61008f1640 1 -- 172.21.15.148:0/38543874 wait complete. 2024-09-16T04:20:50.124 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:20:50.125 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:20:50.125 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:20:50.125 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:20:50.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.222+0000 7f714359e640 10 monclient: get_monmap_and_config 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.222+0000 7f714359e640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.222+0000 7f714359e640 10 monclient: monmap: 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:50.223086+0000 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:50.223086+0000 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:50.224 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.222+0000 7f714359e640 1 Processor -- start 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.222+0000 7f714359e640 1 -- start start 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.222+0000 7f714359e640 10 monclient: init 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient(hunting): picked mon.noname-a con 0x7f7144134660 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient(hunting): picked mon.noname-c con 0x7f714409b730 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient(hunting): picked mon.noname-b con 0x7f714413ac50 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714409e970 con 0x7f7144134660 2024-09-16T04:20:50.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71440a22c0 con 0x7f714409b730 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714409b390 con 0x7f714413ac50 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714359e640 10 monclient(hunting): authenticate will time out at 1670.848555s 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714259c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7144134660 0x7f7144134a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48414/0 (socket says 172.21.15.148:48414) 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714259c640 1 -- 172.21.15.148:0/2196399157 learned_addr learned my addr 172.21.15.148:0/2196399157 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370986790 0 0) 0x7f714409b390 con 0x7f714413ac50 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): my global_id is 124196 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7130003520 con 0x7f714413ac50 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2937497499 0 0) 0x7f714409e970 con 0x7f7144134660 2024-09-16T04:20:50.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 10 monclient(hunting): my global_id is 114342 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.223+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714409b390 con 0x7f7144134660 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624040251 0 0) 0x7f71440a22c0 con 0x7f714409b730 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient(hunting): my global_id is 7012 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714409e970 con 0x7f714409b730 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2787221408 0 0) 0x7f7130003520 con 0x7f714413ac50 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 >> v1:172.21.15.171:6789/0 conn(0x7f714409b730 legacy=0x7f7144139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 >> v1:172.21.15.148:6789/0 conn(0x7f7144134660 legacy=0x7f7144134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 monclient: found mon.noname-b 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714409e040 con 0x7f714413ac50 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f713c001990 con 0x7f714413ac50 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 5 monclient: authenticate success, global_id 124196 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: dump: 2024-09-16T04:20:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.237 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f713c001cb0 con 0x7f714413ac50 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 1 -- 172.21.15.148:0/2196399157 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f713c002080 con 0x7f714413ac50 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 4 set_mon_vals no callback set 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714159a640 10 monclient: dump: 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 10 monclient: shutdown 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 1 -- 172.21.15.148:0/2196399157 >> v1:172.21.15.148:6790/0 conn(0x7f714413ac50 legacy=0x7f714413d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 1 -- 172.21.15.148:0/2196399157 shutdown_connections 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 1 -- 172.21.15.148:0/2196399157 >> 172.21.15.148:0/2196399157 conn(0x7f71440a1530 msgr2=0x7f71440a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 1 -- 172.21.15.148:0/2196399157 shutdown_connections 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.224+0000 7f714359e640 1 -- 172.21.15.148:0/2196399157 wait complete. 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.225+0000 7f714359e640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.225+0000 7f714359e640 10 monclient: monmap: 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.240 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.225+0000 7f714359e640 1 Processor -- start 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.225+0000 7f714359e640 1 -- start start 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.225+0000 7f714359e640 10 monclient: init 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient(hunting): picked mon.noname-b con 0x7f714409b730 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient(hunting): picked mon.noname-a con 0x7f7144134660 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient(hunting): picked mon.noname-c con 0x7f714413ac50 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7144137520 con 0x7f7144134660 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7144073ed0 con 0x7f714409b730 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7144137100 con 0x7f714413ac50 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f714359e640 10 monclient(hunting): authenticate will time out at 1670.851475s 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f7141d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7144134660 0x7f7144071060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48430/0 (socket says 172.21.15.148:48430) 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f7142d9d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f714413ac50 0x7f7144071770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60590/0 (socket says 172.21.15.148:60590) 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f7141d9b640 1 -- 172.21.15.148:0/2920158016 learned_addr learned my addr 172.21.15.148:0/2920158016 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012677173 0 0) 0x7f7144137100 con 0x7f714413ac50 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): my global_id is 124199 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7124003520 con 0x7f714413ac50 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314935478 0 0) 0x7f714409e970 con 0x7f7144134660 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): my global_id is 114345 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7144137100 con 0x7f7144134660 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458389116 0 0) 0x7f7144073ed0 con 0x7f714409b730 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 10 monclient(hunting): my global_id is 7015 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f714409e970 con 0x7f714409b730 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2901055072 0 0) 0x7f7124003520 con 0x7f714413ac50 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7144073ed0 con 0x7f714413ac50 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1858227416 0 0) 0x7f7144137100 con 0x7f7144134660 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.226+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7124003520 con 0x7f7144134660 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f713c0017b0 con 0x7f714413ac50 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): dump: 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.244 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f712c001850 con 0x7f7144134660 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient(hunting): dump: 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.245 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 328818479 0 0) 0x7f7144073ed0 con 0x7f714413ac50 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.171:6789/0 conn(0x7f714409b730 legacy=0x7f7144070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.148:6789/0 conn(0x7f7144134660 legacy=0x7f7144071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 monclient: found mon.c 2024-09-16T04:20:50.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71441356f0 con 0x7f714413ac50 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 5 monclient: authenticate success, global_id 124199 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 10 monclient: _renew_subs 2024-09-16T04:20:50.247 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7144136e20 con 0x7f714413ac50 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 10 monclient: _renew_subs 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f71441d27d0 con 0x7f714413ac50 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f713c003e40 con 0x7f714413ac50 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f713c004190 con 0x7f714413ac50 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.248 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f7140d99640 4 set_mon_vals no callback set 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f713affd640 10 monclient: dump: 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.249 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.227+0000 7f714359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71280065c0 con 0x7f7128002860 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f713c0043b0 con 0x7f714413ac50 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f713c047ae0 con 0x7f714413ac50 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 603613449 0 0) 0x7f71280065c0 con 0x7f7128002860 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient(hunting): my global_id is 114348 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7144073ed0 con 0x7f7128002860 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f7141d9b640 10 monclient: get_auth_request con 0x7f7124038e00 auth_method 0 2024-09-16T04:20:50.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 725738561 0 0) 0x7f7144073ed0 con 0x7f7128002860 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71280065c0 con 0x7f7128002860 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7134004980 con 0x7f7128002860 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 10 monclient: dump: 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.251 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1329189798 0 0) 0x7f71280065c0 con 0x7f7128002860 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.228+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f712800a150 con 0x7f7128002860 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.229+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7134006be0 con 0x7f7128002860 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.229+0000 7f713affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.229+0000 7f713affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.229+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.148:6789/0 conn(0x7f7128002860 legacy=0x7f7128004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.252 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.229+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.148:6789/0 conn(0x7f7128002860 legacy=0x7f7128004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.234+0000 7f714359e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7128009e40 con 0x7f7128009100 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8621890 0 0) 0x7f7128009e40 con 0x7f7128009100 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient(hunting): my global_id is 114351 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71280065c0 con 0x7f7128009100 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 334661515 0 0) 0x7f71280065c0 con 0x7f7128009100 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7128009e40 con 0x7f7128009100 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7134006300 con 0x7f7128009100 2024-09-16T04:20:50.254 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient: dump: 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.255 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4094135587 0 0) 0x7f7128009e40 con 0x7f7128009100 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f712800c3d0 con 0x7f7128009100 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4146 (unknown 0 0 2228351185) 0x7f7134006160 con 0x7f7128009100 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.148:6789/0 conn(0x7f7128009100 legacy=0x7f7128009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.235+0000 7f713affd640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.148:6789/0 conn(0x7f7128009100 legacy=0x7f7128009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:50.256 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 58, 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 146138, 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.257 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:20:50.258 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.259 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:20:50.260 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.261 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:20:50.262 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.263 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.264 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.265 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.266 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.267 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:20:50.268 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:20:50.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.171:6812/7327273 conn(0x7f7124038e00 legacy=0x7f712403b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 10 monclient: shutdown 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 >> v1:172.21.15.148:6790/0 conn(0x7f714413ac50 legacy=0x7f7144071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f7142d9d640 1 -- 172.21.15.148:0/2920158016 reap_dead start 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 shutdown_connections 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 >> 172.21.15.148:0/2920158016 conn(0x7f71440a1530 msgr2=0x7f71440a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 shutdown_connections 2024-09-16T04:20:50.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.236+0000 7f714359e640 1 -- 172.21.15.148:0/2920158016 wait complete. 2024-09-16T04:20:50.270 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:20:50.270 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:20:50.271 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:20:50.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.368+0000 7f124c7d6640 10 monclient: get_monmap_and_config 2024-09-16T04:20:50.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.368+0000 7f124c7d6640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.368+0000 7f124c7d6640 10 monclient: monmap: 2024-09-16T04:20:50.370 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:50.369413+0000 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:50.369413+0000 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 1 Processor -- start 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 1 -- start start 2024-09-16T04:20:50.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient: init 2024-09-16T04:20:50.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1244134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f124409e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f124413ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1244135360 con 0x7f124409e240 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1244134440 con 0x7f1244134870 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124409e650 con 0x7f124413ae80 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124c7d6640 10 monclient(hunting): authenticate will time out at 1670.994740s 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124ad73640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f124413ae80 0x7f124413d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60604/0 (socket says 172.21.15.148:60604) 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f124ad73640 1 -- 172.21.15.148:0/2285328528 learned_addr learned my addr 172.21.15.148:0/2285328528 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.369+0000 7f1249d71640 1 --1- 172.21.15.148:0/2285328528 >> v1:172.21.15.148:6789/0 conn(0x7f124409e240 0x7f1244139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48460/0 (socket says 172.21.15.148:48460) 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289756946 0 0) 0x7f1244135360 con 0x7f124409e240 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): my global_id is 114354 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1230003520 con 0x7f124409e240 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516181494 0 0) 0x7f124409e650 con 0x7f124413ae80 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): my global_id is 124202 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1244135360 con 0x7f124413ae80 2024-09-16T04:20:50.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2882040666 0 0) 0x7f1244134440 con 0x7f1244134870 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient(hunting): my global_id is 7018 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f124409e650 con 0x7f1244134870 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3134675026 0 0) 0x7f1244135360 con 0x7f124413ae80 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 >> v1:172.21.15.171:6789/0 conn(0x7f1244134870 legacy=0x7f1244134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 >> v1:172.21.15.148:6789/0 conn(0x7f124409e240 legacy=0x7f1244139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 monclient: found mon.noname-b 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1244073230 con 0x7f124413ae80 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f123c0018f0 con 0x7f124413ae80 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f124c7d6640 5 monclient: authenticate success, global_id 124202 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f124c7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: dump: 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.384 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f124c7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f123c001b50 con 0x7f124413ae80 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 1 -- 172.21.15.148:0/2285328528 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f123c001f20 con 0x7f124413ae80 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f124c7d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f124c7d6640 4 set_mon_vals no callback set 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.370+0000 7f1249570640 10 monclient: dump: 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.386 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 10 monclient: shutdown 2024-09-16T04:20:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 1 -- 172.21.15.148:0/2285328528 >> v1:172.21.15.148:6790/0 conn(0x7f124413ae80 legacy=0x7f124413d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 1 -- 172.21.15.148:0/2285328528 shutdown_connections 2024-09-16T04:20:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 1 -- 172.21.15.148:0/2285328528 >> 172.21.15.148:0/2285328528 conn(0x7f12440a5640 msgr2=0x7f12440a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 1 -- 172.21.15.148:0/2285328528 shutdown_connections 2024-09-16T04:20:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 1 -- 172.21.15.148:0/2285328528 wait complete. 2024-09-16T04:20:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.371+0000 7f124c7d6640 10 monclient: monmap: 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 1 Processor -- start 2024-09-16T04:20:50.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 1 -- start start 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient: init 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f124409e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1244134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f124413ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124411d6a0 con 0x7f124409e240 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124411ebc0 con 0x7f1244134870 2024-09-16T04:20:50.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12440a3da0 con 0x7f124413ae80 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124c7d6640 10 monclient(hunting): authenticate will time out at 1670.997567s 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124a572640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f124409e240 0x7f1244133170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48472/0 (socket says 172.21.15.148:48472) 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124a572640 1 -- 172.21.15.148:0/3677369379 learned_addr learned my addr 172.21.15.148:0/3677369379 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f124ad73640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f124413ae80 0x7f12441d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60608/0 (socket says 172.21.15.148:60608) 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160384433 0 0) 0x7f12440a3da0 con 0x7f124413ae80 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): my global_id is 124205 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1228003520 con 0x7f124413ae80 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 496228099 0 0) 0x7f124411d6a0 con 0x7f124409e240 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): my global_id is 114357 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12440a3da0 con 0x7f124409e240 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235109411 0 0) 0x7f124411ebc0 con 0x7f1244134870 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 10 monclient(hunting): my global_id is 7021 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.372+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f124411d6a0 con 0x7f1244134870 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1983594216 0 0) 0x7f124411d6a0 con 0x7f1244134870 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f124411ebc0 con 0x7f1244134870 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1234001a30 con 0x7f1244134870 2024-09-16T04:20:50.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): dump: 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2237959385 0 0) 0x7f1228003520 con 0x7f124413ae80 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f124411d6a0 con 0x7f124413ae80 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2459739220 0 0) 0x7f12440a3da0 con 0x7f124409e240 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1228003520 con 0x7f124409e240 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f123c0017f0 con 0x7f124413ae80 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): dump: 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.394 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f12400018f0 con 0x7f124409e240 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient(hunting): dump: 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.395 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3546161388 0 0) 0x7f124411ebc0 con 0x7f1244134870 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.148:6790/0 conn(0x7f124413ae80 legacy=0x7f12441d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.148:6789/0 conn(0x7f124409e240 legacy=0x7f1244133170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 monclient: found mon.b 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:50.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124411e8e0 con 0x7f1244134870 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 5 monclient: authenticate success, global_id 7021 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 10 monclient: _renew_subs 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f124411d3e0 con 0x7f1244134870 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 10 monclient: _renew_subs 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f124411c430 con 0x7f1244134870 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1234001c90 con 0x7f1244134870 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1234004c30 con 0x7f1244134870 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f1248d6f640 4 set_mon_vals no callback set 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.373+0000 7f123affd640 10 monclient: dump: 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.398 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f124c7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f122c0065c0 con 0x7f122c002860 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f123400fec0 con 0x7f1244134870 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f1234047700 con 0x7f1244134870 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151972086 0 0) 0x7f122c0065c0 con 0x7f122c002860 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient(hunting): my global_id is 124208 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f124411ebc0 con 0x7f122c002860 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f124ad73640 10 monclient: get_auth_request con 0x7f1228039320 auth_method 0 2024-09-16T04:20:50.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 336777584 0 0) 0x7f124411ebc0 con 0x7f122c002860 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f122c0065c0 con 0x7f122c002860 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1240003790 con 0x7f122c002860 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 10 monclient: dump: 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.400 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3118759271 0 0) 0x7f122c0065c0 con 0x7f122c002860 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.374+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f122c00a150 con 0x7f122c002860 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.375+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f12400057c0 con 0x7f122c002860 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.375+0000 7f123affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.375+0000 7f123affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.375+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.148:6790/0 conn(0x7f122c002860 legacy=0x7f122c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.375+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.148:6790/0 conn(0x7f122c002860 legacy=0x7f122c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:50.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f124c7d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f122c009e20 con 0x7f122c0090e0 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 156443940 0 0) 0x7f122c009e20 con 0x7f122c0090e0 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient(hunting): my global_id is 124211 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f122c0065c0 con 0x7f122c0090e0 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1194140686 0 0) 0x7f122c0065c0 con 0x7f122c0090e0 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f122c009e20 con 0x7f122c0090e0 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1240003cf0 con 0x7f122c0090e0 2024-09-16T04:20:50.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient: dump: 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.405 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1281382532 0 0) 0x7f122c009e20 con 0x7f122c0090e0 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f122c00c390 con 0x7f122c0090e0 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 3249412911) 0x7f1240005a10 con 0x7f122c0090e0 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.148:6790/0 conn(0x7f122c0090e0 legacy=0x7f122c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.381+0000 7f123affd640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.148:6790/0 conn(0x7f122c0090e0 legacy=0x7f122c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:20:50.406 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 58, 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 17730, 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:20:50.407 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.408 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.409 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:50.410 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.411 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:20:50.412 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:20:50.413 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:20:50.414 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.415 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:20:50.416 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.417 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:20:50.418 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:20:50.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.171:6812/7327273 conn(0x7f1228039320 legacy=0x7f122803b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 10 monclient: shutdown 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 >> v1:172.21.15.171:6789/0 conn(0x7f1244134870 legacy=0x7f12441ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124ad73640 1 -- 172.21.15.148:0/3677369379 reap_dead start 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124ad73640 1 -- 172.21.15.148:0/3677369379 reap_dead start 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 shutdown_connections 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 >> 172.21.15.148:0/3677369379 conn(0x7f12440a5640 msgr2=0x7f12440a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 shutdown_connections 2024-09-16T04:20:50.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.382+0000 7f124c7d6640 1 -- 172.21.15.148:0/3677369379 wait complete. 2024-09-16T04:20:50.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:20:50.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:20:50.421 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:20:50.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.518+0000 7ff173c2e640 10 monclient: get_monmap_and_config 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.518+0000 7ff173c2e640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.518+0000 7ff173c2e640 10 monclient: monmap: 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:50.519097+0000 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:50.519097+0000 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:50.520 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.518+0000 7ff173c2e640 1 Processor -- start 2024-09-16T04:20:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 1 -- start start 2024-09-16T04:20:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient: init 2024-09-16T04:20:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff16c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff16c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff16c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c0a0e00 con 0x7ff16c0a2410 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c0a10e0 con 0x7ff16c13d060 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c0a4970 con 0x7ff16c0a1b10 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff173c2e640 10 monclient(hunting): authenticate will time out at 1671.144637s 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1711c9640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff16c0a1b10 0x7ff16c13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60646/0 (socket says 172.21.15.148:60646) 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1719ca640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff16c0a2410 0x7ff16c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48486/0 (socket says 172.21.15.148:48486) 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1711c9640 1 -- 172.21.15.148:0/1870588451 learned_addr learned my addr 172.21.15.148:0/1870588451 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2602293461 0 0) 0x7ff16c0a4970 con 0x7ff16c0a1b10 2024-09-16T04:20:50.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1709c8640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1709c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1709c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.519+0000 7ff1709c8640 10 monclient(hunting): my global_id is 124214 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff158003520 con 0x7ff16c0a1b10 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1462931630 0 0) 0x7ff16c0a0e00 con 0x7ff16c0a2410 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient(hunting): my global_id is 114360 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff16c0a4970 con 0x7ff16c0a2410 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3773646774 0 0) 0x7ff158003520 con 0x7ff16c0a1b10 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 >> v1:172.21.15.171:6789/0 conn(0x7ff16c13d060 legacy=0x7ff16c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 >> v1:172.21.15.148:6789/0 conn(0x7ff16c0a2410 legacy=0x7ff16c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 monclient: found mon.noname-b 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16c06dfd0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff15c001610 con 0x7ff16c0a1b10 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 5 monclient: authenticate success, global_id 124214 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:50.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: dump: 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff15c0018f0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 1 -- 172.21.15.148:0/1870588451 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff15c001d00 con 0x7ff16c0a1b10 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 4 set_mon_vals no callback set 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff1709c8640 10 monclient: dump: 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.534 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 10 monclient: shutdown 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 1 -- 172.21.15.148:0/1870588451 >> v1:172.21.15.148:6790/0 conn(0x7ff16c0a1b10 legacy=0x7ff16c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 1 -- 172.21.15.148:0/1870588451 shutdown_connections 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.520+0000 7ff173c2e640 1 -- 172.21.15.148:0/1870588451 >> 172.21.15.148:0/1870588451 conn(0x7ff16c0a97b0 msgr2=0x7ff16c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 1 -- 172.21.15.148:0/1870588451 shutdown_connections 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 1 -- 172.21.15.148:0/1870588451 wait complete. 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 10 monclient: monmap: 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:50.535 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 1 Processor -- start 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 1 -- start start 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.521+0000 7ff173c2e640 10 monclient: init 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:20:50.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff16c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff16c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff16c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c137fc0 con 0x7ff16c0a2410 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c137ba0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c09d9b0 con 0x7ff16c13d060 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff173c2e640 10 monclient(hunting): authenticate will time out at 1671.147455s 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1721cb640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff16c13d060 0x7ff16c1d3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60652/0 (socket says 172.21.15.148:60652) 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1721cb640 1 -- 172.21.15.148:0/4129451865 learned_addr learned my addr 172.21.15.148:0/4129451865 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2579808056 0 0) 0x7ff16c06dfd0 con 0x7ff16c0a2410 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): my global_id is 114363 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff150003550 con 0x7ff16c0a2410 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2650320902 0 0) 0x7ff16c09d9b0 con 0x7ff16c13d060 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): my global_id is 124217 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff16c06dfd0 con 0x7ff16c13d060 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1944224008 0 0) 0x7ff16c137ba0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): my global_id is 7027 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff16c09d9b0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 12416027 0 0) 0x7ff150003550 con 0x7ff16c0a2410 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff16c137ba0 con 0x7ff16c0a2410 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff15c001c40 con 0x7ff16c0a2410 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.522+0000 7ff1627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): dump: 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.539 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2452711060 0 0) 0x7ff16c06dfd0 con 0x7ff16c13d060 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff150003550 con 0x7ff16c13d060 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3368022880 0 0) 0x7ff16c09d9b0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff16c06dfd0 con 0x7ff16c0a1b10 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff164001830 con 0x7ff16c13d060 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): dump: 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.540 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff168001b90 con 0x7ff16c0a1b10 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient(hunting): dump: 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.541 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4191548145 0 0) 0x7ff16c137ba0 con 0x7ff16c0a2410 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.148:6790/0 conn(0x7ff16c13d060 legacy=0x7ff16c1d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.171:6789/0 conn(0x7ff16c0a1b10 legacy=0x7ff16c09cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 monclient: found mon.a 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16c13cae0 con 0x7ff16c0a2410 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 5 monclient: authenticate success, global_id 114363 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 10 monclient: _renew_subs 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff16c0a0130 con 0x7ff16c0a2410 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 10 monclient: _renew_subs 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff16c09f110 con 0x7ff16c0a2410 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff15c0018f0 con 0x7ff16c0a2410 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff15c004cc0 con 0x7ff16c0a2410 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff163fff640 4 set_mon_vals no callback set 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 10 monclient: dump: 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.544 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.523+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff15c00ff50 con 0x7ff16c0a2410 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff173c2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1540065c0 con 0x7ff154002860 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7ff15c047880 con 0x7ff16c0a2410 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1719ca640 10 monclient: get_auth_request con 0x7ff1500392b0 auth_method 0 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838663116 0 0) 0x7ff1540065c0 con 0x7ff154002860 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient(hunting): my global_id is 7030 2024-09-16T04:20:50.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff16c137ba0 con 0x7ff154002860 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2826529658 0 0) 0x7ff16c137ba0 con 0x7ff154002860 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1540065c0 con 0x7ff154002860 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff1640037b0 con 0x7ff154002860 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 10 monclient: dump: 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.546 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1603870608 0 0) 0x7ff1540065c0 con 0x7ff154002860 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.524+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff15400a150 con 0x7ff154002860 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.525+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff164003770 con 0x7ff154002860 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.525+0000 7ff1627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.525+0000 7ff1627fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.525+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.171:6789/0 conn(0x7ff154002860 legacy=0x7ff154004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.525+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.171:6789/0 conn(0x7ff154002860 legacy=0x7ff154004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff173c2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff154009e20 con 0x7ff1540090e0 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4138602981 0 0) 0x7ff154009e20 con 0x7ff1540090e0 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient(hunting): my global_id is 7033 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1540065c0 con 0x7ff1540090e0 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 109346008 0 0) 0x7ff1540065c0 con 0x7ff1540090e0 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff154009e20 con 0x7ff1540090e0 2024-09-16T04:20:50.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff164003df0 con 0x7ff1540090e0 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.531+0000 7ff1627fc640 10 monclient: dump: 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.550 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2935655280 0 0) 0x7ff154009e20 con 0x7ff1540090e0 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff15400c390 con 0x7ff1540090e0 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 1908780741) 0x7ff164003d00 con 0x7ff1540090e0 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.171:6789/0 conn(0x7ff1540090e0 legacy=0x7ff154009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1627fc640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.171:6789/0 conn(0x7ff1540090e0 legacy=0x7ff154009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:20:50.551 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 58, 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 874570, 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:20:50.552 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.553 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.554 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:20:50.555 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:20:50.556 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:20:50.557 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:20:50.558 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:20:50.560 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:20:50.561 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.562 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:20:50.563 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:20:50.564 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:20:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.171:6812/7327273 conn(0x7ff1500392b0 legacy=0x7ff15003b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 10 monclient: shutdown 2024-09-16T04:20:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 >> v1:172.21.15.148:6789/0 conn(0x7ff16c0a2410 legacy=0x7ff16c09d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1721cb640 1 -- 172.21.15.148:0/4129451865 reap_dead start 2024-09-16T04:20:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff1721cb640 1 -- 172.21.15.148:0/4129451865 reap_dead start 2024-09-16T04:20:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 shutdown_connections 2024-09-16T04:20:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 >> 172.21.15.148:0/4129451865 conn(0x7ff16c0a97b0 msgr2=0x7ff16c0a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 shutdown_connections 2024-09-16T04:20:50.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.532+0000 7ff173c2e640 1 -- 172.21.15.148:0/4129451865 wait complete. 2024-09-16T04:20:50.566 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:20:50.566 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:20:50.566 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:20:50.567 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:20:50.567 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:20:50.567 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:20:50.567 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:20:50.578 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:20:50.578 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:20:50.578 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.713+0000 7f6ee9ee4640 10 monclient: get_monmap_and_config 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.713+0000 7f6ee9ee4640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.713+0000 7f6ee9ee4640 10 monclient: monmap: 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:50.714755+0000 2024-09-16T04:20:50.715 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:50.714755+0000 2024-09-16T04:20:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:50.716 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:50.716 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.714+0000 7f6ee9ee4640 1 Processor -- start 2024-09-16T04:20:50.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.714+0000 7f6ee9ee4640 1 -- start start 2024-09-16T04:20:50.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.714+0000 7f6ee9ee4640 10 monclient: init 2024-09-16T04:20:50.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:20:50.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f6ee40a22b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f6ee40a1b70 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f6ee413d0d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee40a0440 con 0x7f6ee40a22b0 2024-09-16T04:20:50.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee40a4d90 con 0x7f6ee413d0d0 2024-09-16T04:20:50.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee40a08f0 con 0x7f6ee40a1b70 2024-09-16T04:20:50.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient(hunting): authenticate will time out at 1671.340174s 2024-09-16T04:20:50.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee37fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f6ee40a22b0 legacy=0x7f6ee40a2690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:50.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee2ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6ee40a1b70 0x7f6ee413b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60668/0 (socket says 172.21.15.148:60668) 2024-09-16T04:20:50.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee2ffd640 1 -- 172.21.15.148:0/4074473083 learned_addr learned my addr 172.21.15.148:0/4074473083 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635586147 0 0) 0x7f6ee40a08f0 con 0x7f6ee40a1b70 2024-09-16T04:20:50.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): my global_id is 124220 2024-09-16T04:20:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ecc003520 con 0x7f6ee40a1b70 2024-09-16T04:20:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228337692 0 0) 0x7f6ee40a4d90 con 0x7f6ee413d0d0 2024-09-16T04:20:50.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient(hunting): my global_id is 7036 2024-09-16T04:20:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ee40a08f0 con 0x7f6ee413d0d0 2024-09-16T04:20:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 713881153 0 0) 0x7f6ecc003520 con 0x7f6ee40a1b70 2024-09-16T04:20:50.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 >> v1:172.21.15.171:6789/0 conn(0x7f6ee413d0d0 legacy=0x7f6ee413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 >> v1:172.21.15.148:6789/0 conn(0x7f6ee40a22b0 legacy=0x7f6ee40a2690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 monclient: found mon.noname-b 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee40a2da0 con 0x7f6ee40a1b70 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 5 monclient: authenticate success, global_id 124220 2024-09-16T04:20:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ed8001610 con 0x7f6ee40a1b70 2024-09-16T04:20:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: dump: 2024-09-16T04:20:50.740 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ed8001930 con 0x7f6ee40a1b70 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.741 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 1 -- 172.21.15.148:0/4074473083 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ed8001d00 con 0x7f6ee40a1b70 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee9ee4640 4 set_mon_vals no callback set 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.715+0000 7f6ee27fc640 10 monclient: dump: 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.742 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 10 monclient: shutdown 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/4074473083 >> v1:172.21.15.148:6790/0 conn(0x7f6ee40a1b70 legacy=0x7f6ee413b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/4074473083 shutdown_connections 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/4074473083 >> 172.21.15.148:0/4074473083 conn(0x7f6ee40732c0 msgr2=0x7f6ee40736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/4074473083 shutdown_connections 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/4074473083 wait complete. 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 10 monclient: build_initial_monmap 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.716+0000 7f6ee9ee4640 10 monclient: monmap: 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:50.743 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 1 Processor -- start 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 1 -- start start 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient: init 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f6ee40a1b70 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f6ee40a22b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:50.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f6ee413d0d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee409d690 con 0x7f6ee413d0d0 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee4137a70 con 0x7f6ee40a1b70 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee4140720 con 0x7f6ee40a22b0 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee9ee4640 10 monclient(hunting): authenticate will time out at 1671.342902s 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee3fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f6ee413d0d0 legacy=0x7f6ee41d3d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee2ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6ee40a22b0 0x7f6ee409cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:60674/0 (socket says 172.21.15.148:60674) 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.717+0000 7f6ee2ffd640 1 -- 172.21.15.148:0/2885244819 learned_addr learned my addr 172.21.15.148:0/2885244819 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815132890 0 0) 0x7f6ee4140720 con 0x7f6ee40a22b0 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.745 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): my global_id is 124223 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ed0003520 con 0x7f6ee40a22b0 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735053835 0 0) 0x7f6ee4137a70 con 0x7f6ee40a1b70 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): my global_id is 7039 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ee4140720 con 0x7f6ee40a1b70 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4056483133 0 0) 0x7f6ed0003520 con 0x7f6ee40a22b0 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ee4137a70 con 0x7f6ee40a22b0 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ed80015d0 con 0x7f6ee40a22b0 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): dump: 2024-09-16T04:20:50.746 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 445923690 0 0) 0x7f6ee4140720 con 0x7f6ee40a1b70 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ed0003520 con 0x7f6ee40a1b70 2024-09-16T04:20:50.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ed4004040 con 0x7f6ee40a1b70 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient(hunting): dump: 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.748 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1944749675 0 0) 0x7f6ee4137a70 con 0x7f6ee40a22b0 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 >> v1:172.21.15.171:6789/0 conn(0x7f6ee40a1b70 legacy=0x7f6ee409c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 >> v1:172.21.15.148:6789/0 conn(0x7f6ee413d0d0 legacy=0x7f6ee41d3d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 monclient: found mon.c 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee413c860 con 0x7f6ee40a22b0 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: _finish_auth 0 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 5 monclient: authenticate success, global_id 124223 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 10 monclient: _renew_subs 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.749 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ee40a0130 con 0x7f6ee40a22b0 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 10 monclient: _renew_subs 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ee4137790 con 0x7f6ee40a22b0 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ed8001930 con 0x7f6ee40a22b0 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6ed8004e60 con 0x7f6ee40a22b0 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ee1ffb640 4 set_mon_vals no callback set 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.718+0000 7f6ecbfff640 10 monclient: dump: 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:50.750 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6ed80050c0 con 0x7f6ee40a22b0 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ee9ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ee9ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ee9ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec4005080 con 0x7f6ee40a22b0 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f6ed8047be0 con 0x7f6ee40a22b0 2024-09-16T04:20:50.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.719+0000 7f6ee37fe640 10 monclient: get_auth_request con 0x7f6ed0038e80 auth_method 0 2024-09-16T04:20:50.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.724+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f6ed8015f60 con 0x7f6ee40a22b0 2024-09-16T04:20:50.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.724+0000 7f6ecbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:50.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.724+0000 7f6ecbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:50.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ee9ee4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:20:50.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ee9ee4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:50.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ee9ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:50.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ec4005370 con 0x7f6ee40a22b0 2024-09-16T04:20:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ecbfff640 1 -- 172.21.15.148:0/2885244819 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 1127931544) 0x7f6ed8015860 con 0x7f6ee40a22b0 2024-09-16T04:20:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ecbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:50.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.951+0000 7f6ecbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:50.954 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:20:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.953+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 >> v1:172.21.15.171:6812/7327273 conn(0x7f6ed0038e80 legacy=0x7f6ed003b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.953+0000 7f6ee9ee4640 10 monclient: shutdown 2024-09-16T04:20:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.953+0000 7f6ee9ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:50.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.953+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 >> v1:172.21.15.148:6790/0 conn(0x7f6ee40a22b0 legacy=0x7f6ee409cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.953+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 shutdown_connections 2024-09-16T04:20:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.953+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 >> 172.21.15.148:0/2885244819 conn(0x7f6ee40732c0 msgr2=0x7f6ee4135a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.954+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 shutdown_connections 2024-09-16T04:20:50.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:50.954+0000 7f6ee9ee4640 1 -- 172.21.15.148:0/2885244819 wait complete. 2024-09-16T04:20:53.967 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.074+0000 7fdbcd7e4640 10 monclient: get_monmap_and_config 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.074+0000 7fdbcd7e4640 10 monclient: build_initial_monmap 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.074+0000 7fdbcd7e4640 10 monclient: monmap: 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:54.075707+0000 2024-09-16T04:20:54.076 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:54.075707+0000 2024-09-16T04:20:54.077 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:54.077 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.077 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:54.077 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:54.077 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:54.077 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.075+0000 7fdbcd7e4640 1 Processor -- start 2024-09-16T04:20:54.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.075+0000 7fdbcd7e4640 1 -- start start 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.075+0000 7fdbcd7e4640 10 monclient: init 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc80a1010 con 0x7fdbc809bb70 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc809f030 con 0x7fdbc809cc30 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc809dc60 con 0x7fdbc809c660 2024-09-16T04:20:54.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient(hunting): authenticate will time out at 1674.701102s 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc6ffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdbc809bb70 legacy=0x7fdbc809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbbffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fdbc809c660 0x7fdbc8138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37832/0 (socket says 172.21.15.148:37832) 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbbffff640 1 -- 172.21.15.148:0/2694075628 learned_addr learned my addr 172.21.15.148:0/2694075628 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360810772 0 0) 0x7fdbc809dc60 con 0x7fdbc809c660 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): my global_id is 124226 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbb4003520 con 0x7fdbc809c660 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2710669097 0 0) 0x7fdbc809f030 con 0x7fdbc809cc30 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:20:54.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient(hunting): my global_id is 7042 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc809dc60 con 0x7fdbc809cc30 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3538757530 0 0) 0x7fdbb4003520 con 0x7fdbc809c660 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 >> v1:172.21.15.171:6789/0 conn(0x7fdbc809cc30 legacy=0x7fdbc813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 >> v1:172.21.15.148:6789/0 conn(0x7fdbc809bb70 legacy=0x7fdbc809bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 monclient: found mon.noname-b 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc809df40 con 0x7fdbc809c660 2024-09-16T04:20:54.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: _finish_auth 0 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbb0001610 con 0x7fdbc809c660 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 5 monclient: authenticate success, global_id 124226 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: dump: 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.089 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbb00018c0 con 0x7fdbc809c660 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 1 -- 172.21.15.148:0/2694075628 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbb0001c90 con 0x7fdbc809c660 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbc67fc640 10 monclient: dump: 2024-09-16T04:20:54.090 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.076+0000 7fdbcd7e4640 4 set_mon_vals no callback set 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 10 monclient: shutdown 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:54.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/2694075628 >> v1:172.21.15.148:6790/0 conn(0x7fdbc809c660 legacy=0x7fdbc8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/2694075628 shutdown_connections 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/2694075628 >> 172.21.15.148:0/2694075628 conn(0x7fdbc80a5640 msgr2=0x7fdbc80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/2694075628 shutdown_connections 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/2694075628 wait complete. 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 10 monclient: build_initial_monmap 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.077+0000 7fdbcd7e4640 10 monclient: monmap: 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:20:54.092 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 1 Processor -- start 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 1 -- start start 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient: init 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:20:54.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc813bd40 con 0x7fdbc809c660 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc813c020 con 0x7fdbc809bb70 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc813a820 con 0x7fdbc809cc30 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbbffff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fdbc809c660 legacy=0x7fdbc81d0b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbcd7e4640 10 monclient(hunting): authenticate will time out at 1674.703838s 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbc77fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fdbc809cc30 0x7fdbc81d4240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37834/0 (socket says 172.21.15.148:37834) 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.078+0000 7fdbc77fe640 1 -- 172.21.15.148:0/3534565914 learned_addr learned my addr 172.21.15.148:0/3534565914 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158646593 0 0) 0x7fdbc813a820 con 0x7fdbc809cc30 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:20:54.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): my global_id is 124229 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba8003550 con 0x7fdbc809cc30 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259134351 0 0) 0x7fdbc813c020 con 0x7fdbc809bb70 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): my global_id is 7045 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc813a820 con 0x7fdbc809bb70 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3695170179 0 0) 0x7fdba8003550 con 0x7fdbc809cc30 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbc813c020 con 0x7fdbc809cc30 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbc0001c50 con 0x7fdbc809cc30 2024-09-16T04:20:54.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): dump: 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3343558103 0 0) 0x7fdbc813a820 con 0x7fdbc809bb70 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba8003550 con 0x7fdbc809bb70 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbb80040c0 con 0x7fdbc809bb70 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient(hunting): dump: 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.097 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2975364824 0 0) 0x7fdbc813c020 con 0x7fdbc809cc30 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 >> v1:172.21.15.171:6789/0 conn(0x7fdbc809bb70 legacy=0x7fdbc81cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 >> v1:172.21.15.148:6789/0 conn(0x7fdbc809c660 legacy=0x7fdbc81d0b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 monclient: found mon.c 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc813ba60 con 0x7fdbc809cc30 2024-09-16T04:20:54.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 5 monclient: authenticate success, global_id 124229 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 10 monclient: _renew_subs 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc813ac60 con 0x7fdbc809cc30 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 10 monclient: _renew_subs 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbc806f190 con 0x7fdbc809cc30 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbc0002020 con 0x7fdbc809cc30 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdbc0003de0 con 0x7fdbc809cc30 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:20:54.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbc5ffb640 4 set_mon_vals no callback set 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.079+0000 7fdbbf7fe640 10 monclient: dump: 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:54.100 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.080+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdbc0004000 con 0x7fdbc809cc30 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.080+0000 7fdbcd7e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.080+0000 7fdbcd7e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.080+0000 7fdbcd7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.080+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac005080 con 0x7fdbc809cc30 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.080+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7fdbc0047e20 con 0x7fdbc809cc30 2024-09-16T04:20:54.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.081+0000 7fdbc6ffd640 10 monclient: get_auth_request con 0x7fdba8038f40 auth_method 0 2024-09-16T04:20:54.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.085+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fdbc0016040 con 0x7fdbc809cc30 2024-09-16T04:20:54.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.085+0000 7fdbbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:20:54.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.085+0000 7fdbbf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:20:54.317 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbcd7e4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbcd7e4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbcd7e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdbac005370 con 0x7fdbc809cc30 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbbf7fe640 1 -- 172.21.15.148:0/3534565914 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3856482314) 0x7fdbc0019cd0 con 0x7fdbc809cc30 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.316+0000 7fdbbf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:20:54.318 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 >> v1:172.21.15.171:6812/7327273 conn(0x7fdba8038f40 legacy=0x7fdba803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 10 monclient: shutdown 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 >> v1:172.21.15.148:6790/0 conn(0x7fdbc809cc30 legacy=0x7fdbc81d4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 shutdown_connections 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 >> 172.21.15.148:0/3534565914 conn(0x7fdbc80a5640 msgr2=0x7fdbc80a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 shutdown_connections 2024-09-16T04:20:54.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:54.318+0000 7fdbcd7e4640 1 -- 172.21.15.148:0/3534565914 wait complete. 2024-09-16T04:20:57.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:20:57.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.439+0000 7ff10e1f6640 10 monclient: get_monmap_and_config 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.439+0000 7ff10e1f6640 10 monclient: build_initial_monmap 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.439+0000 7ff10e1f6640 10 monclient: monmap: 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:20:57.440087+0000 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:20:57.440087+0000 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:20:57.441 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 1 Processor -- start 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 1 -- start start 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient: init 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff1080a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1080a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff10813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:20:57.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080a0e00 con 0x7ff10813d060 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080a10e0 con 0x7ff1080a2410 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080a4970 con 0x7ff1080a1b10 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff10e1f6640 10 monclient(hunting): authenticate will time out at 1678.065740s 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff107fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff10813f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff0fffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff1080a1b10 0x7ff10813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37848/0 (socket says 172.21.15.148:37848) 2024-09-16T04:20:57.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.440+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 learned_addr learned my addr 172.21.15.148:0/3341286958 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:20:57.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:57.641+0000 7ff107fff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff10813f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:58.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:58.042+0000 7ff107fff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff10813f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:20:58.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:20:58.843+0000 7ff107fff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff10813f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:00.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): tick 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6790/0 conn(0x7ff1080a1b10 legacy=0x7ff10813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.171:6789/0 conn(0x7ff1080a2410 legacy=0x7ff1080a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff10813f450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e80036c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): picked mon.noname-a con 0x7ff10813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): picked mon.noname-b con 0x7ff1080a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10806dfd0 con 0x7ff10813d060 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080a0e00 con 0x7ff0e80036c0 2024-09-16T04:21:00.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0e8010420 con 0x7ff1080a1b10 2024-09-16T04:21:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff105ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:00.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.441+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:00.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:00.642+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:01.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:01.043+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:01.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:01.844+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): tick 2024-09-16T04:21:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:21:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:21:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:21:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6790/0 conn(0x7ff1080a1b10 legacy=0x7ff0e800cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:03.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.171:6789/0 conn(0x7ff0e80036c0 legacy=0x7ff0e8005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e800f4a0 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): picked mon.noname-a con 0x7ff10813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): picked mon.noname-b con 0x7ff1080a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0e8011670 con 0x7ff10813d060 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10806dfd0 con 0x7ff0e800f4a0 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0e8010110 con 0x7ff1080a1b10 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff105ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:03.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.441+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:03.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:03.642+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:04.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.043+0000 7ff0fffff640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.656+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 585138257 0 0) 0x7ff0e8010110 con 0x7ff1080a1b10 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.656+0000 7ff106ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.656+0000 7ff106ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.656+0000 7ff106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.656+0000 7ff106ffd640 10 monclient(hunting): my global_id is 124232 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.656+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f4003520 con 0x7ff1080a1b10 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3634072163 0 0) 0x7ff0f4003520 con 0x7ff1080a1b10 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.171:6789/0 conn(0x7ff0e800f4a0 legacy=0x7ff0e800f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6789/0 conn(0x7ff10813d060 legacy=0x7ff0e8009320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 monclient: found mon.noname-b 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e8004ab0 con 0x7ff1080a1b10 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: _finish_auth 0 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff1000044d0 con 0x7ff1080a1b10 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: dump: 2024-09-16T04:21:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:04.662 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff10e1f6640 5 monclient: authenticate success, global_id 124232 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff10e1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1000047b0 con 0x7ff1080a1b10 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 1 -- 172.21.15.148:0/3341286958 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff100004b80 con 0x7ff1080a1b10 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff106ffd640 10 monclient: dump: 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff10e1f6640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.657+0000 7ff10e1f6640 4 set_mon_vals no callback set 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 10 monclient: shutdown 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 1 -- 172.21.15.148:0/3341286958 >> v1:172.21.15.148:6790/0 conn(0x7ff1080a1b10 legacy=0x7ff0e800cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 1 -- 172.21.15.148:0/3341286958 shutdown_connections 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 1 -- 172.21.15.148:0/3341286958 >> 172.21.15.148:0/3341286958 conn(0x7ff1080a97b0 msgr2=0x7ff1080a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 1 -- 172.21.15.148:0/3341286958 shutdown_connections 2024-09-16T04:21:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.658+0000 7ff10e1f6640 1 -- 172.21.15.148:0/3341286958 wait complete. 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.659+0000 7ff10e1f6640 10 monclient: build_initial_monmap 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.659+0000 7ff10e1f6640 10 monclient: monmap: 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.659+0000 7ff10e1f6640 1 Processor -- start 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.659+0000 7ff10e1f6640 1 -- start start 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.659+0000 7ff10e1f6640 10 monclient: init 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e800f4a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1080a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff10813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff1077fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff0e800f4a0 legacy=0x7ff1081cb640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff108125030 con 0x7ff0e800f4a0 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080a8420 con 0x7ff1080a1b10 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10813cd50 con 0x7ff10813d060 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff10e1f6640 10 monclient(hunting): authenticate will time out at 1685.285311s 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff107fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff10813d060 0x7ff1081d39e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37008/0 (socket says 172.21.15.148:37008) 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.660+0000 7ff107fff640 1 -- 172.21.15.148:0/384048435 learned_addr learned my addr 172.21.15.148:0/384048435 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485531978 0 0) 0x7ff1080a8420 con 0x7ff1080a1b10 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): my global_id is 7048 2024-09-16T04:21:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e8010110 con 0x7ff1080a1b10 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309428425 0 0) 0x7ff10813cd50 con 0x7ff10813d060 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): my global_id is 124235 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1080a8420 con 0x7ff10813d060 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 570321332 0 0) 0x7ff0e8010110 con 0x7ff1080a1b10 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff10813cd50 con 0x7ff1080a1b10 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0f0001860 con 0x7ff1080a1b10 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): dump: 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3452451503 0 0) 0x7ff1080a8420 con 0x7ff10813d060 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0e8010110 con 0x7ff10813d060 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff100004230 con 0x7ff10813d060 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.664+0000 7ff104ff9640 10 monclient(hunting): dump: 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:04.676 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2365880247 0 0) 0x7ff10813cd50 con 0x7ff1080a1b10 2024-09-16T04:21:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 >> v1:172.21.15.148:6790/0 conn(0x7ff10813d060 legacy=0x7ff1081d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 >> v1:172.21.15.148:6789/0 conn(0x7ff0e800f4a0 legacy=0x7ff1081cb640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 monclient: found mon.b 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff108123be0 con 0x7ff1080a1b10 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: _finish_auth 0 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 5 monclient: authenticate success, global_id 7048 2024-09-16T04:21:04.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 10 monclient: _renew_subs 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff10813a360 con 0x7ff1080a1b10 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 10 monclient: _renew_subs 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1081d52e0 con 0x7ff1080a1b10 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff0f00047e0 con 0x7ff1080a1b10 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff0f0004b30 con 0x7ff1080a1b10 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff1067fc640 4 set_mon_vals no callback set 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.665+0000 7ff104ff9640 10 monclient: dump: 2024-09-16T04:21:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff0f000ff60 con 0x7ff1080a1b10 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff10e1f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff10e1f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff10e1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ec005080 con 0x7ff1080a1b10 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7ff0f00478e0 con 0x7ff1080a1b10 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.666+0000 7ff1077fe640 10 monclient: get_auth_request con 0x7ff0e8039d60 auth_method 0 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.671+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff0f0015a80 con 0x7ff1080a1b10 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.671+0000 7ff104ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:04.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.671+0000 7ff104ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:04.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.905+0000 7ff10e1f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:21:04.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.905+0000 7ff10e1f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:21:04.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.905+0000 7ff10e1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:21:04.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.905+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0ec005370 con 0x7ff1080a1b10 2024-09-16T04:21:04.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.906+0000 7ff104ff9640 1 -- 172.21.15.148:0/384048435 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3202399412) 0x7ff0f0019730 con 0x7ff1080a1b10 2024-09-16T04:21:04.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.906+0000 7ff104ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:21:04.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.906+0000 7ff104ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:21:04.908 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:21:04.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 >> v1:172.21.15.171:6812/7327273 conn(0x7ff0e8039d60 legacy=0x7ff0e803c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 10 monclient: shutdown 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 >> v1:172.21.15.171:6789/0 conn(0x7ff1080a1b10 legacy=0x7ff108139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 shutdown_connections 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 >> 172.21.15.148:0/384048435 conn(0x7ff1080a97b0 msgr2=0x7ff1080a75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 shutdown_connections 2024-09-16T04:21:04.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:04.908+0000 7ff10e1f6640 1 -- 172.21.15.148:0/384048435 wait complete. 2024-09-16T04:21:04.920 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:21:04.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:21:04.921 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.017+0000 7f419397f640 10 monclient: get_monmap_and_config 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.017+0000 7f419397f640 10 monclient: build_initial_monmap 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.017+0000 7f419397f640 10 monclient: monmap: 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:21:05.018524+0000 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:21:05.018524+0000 2024-09-16T04:21:05.019 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:05.020 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.020 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:21:05.020 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:21:05.020 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:21:05.020 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 1 Processor -- start 2024-09-16T04:21:05.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 1 -- start start 2024-09-16T04:21:05.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient: init 2024-09-16T04:21:05.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient(hunting): picked mon.noname-b con 0x7f418c09bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient(hunting): picked mon.noname-a con 0x7f418c09c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient(hunting): picked mon.noname-c con 0x7f418c09cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c0a1010 con 0x7f418c09c660 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c09f030 con 0x7f418c09cc30 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c09dc60 con 0x7f418c09bb70 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419397f640 10 monclient(hunting): authenticate will time out at 1685.643848s 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f4190f1a640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f418c09c660 legacy=0x7f418c138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419171b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f418c09bb70 0x7f418c09bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37016/0 (socket says 172.21.15.148:37016) 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.018+0000 7f419171b640 1 -- 172.21.15.148:0/2189085846 learned_addr learned my addr 172.21.15.148:0/2189085846 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86374519 0 0) 0x7f418c09dc60 con 0x7f418c09bb70 2024-09-16T04:21:05.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient(hunting): my global_id is 124238 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4178003520 con 0x7f418c09bb70 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1785819227 0 0) 0x7f4178003520 con 0x7f418c09bb70 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 >> v1:172.21.15.171:6789/0 conn(0x7f418c09cc30 legacy=0x7f418c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 >> v1:172.21.15.148:6789/0 conn(0x7f418c09c660 legacy=0x7f418c138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 monclient: found mon.noname-b 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418c09df40 con 0x7f418c09bb70 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: _finish_auth 0 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 5 monclient: authenticate success, global_id 124238 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4188001610 con 0x7f418c09bb70 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: dump: 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.031 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4188001880 con 0x7f418c09bb70 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 4 set_mon_vals no callback set 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 1 -- 172.21.15.148:0/2189085846 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4188001c90 con 0x7f418c09bb70 2024-09-16T04:21:05.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f4183fff640 10 monclient: dump: 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 10 monclient: shutdown 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.019+0000 7f419397f640 1 -- 172.21.15.148:0/2189085846 >> v1:172.21.15.148:6790/0 conn(0x7f418c09bb70 legacy=0x7f418c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 1 -- 172.21.15.148:0/2189085846 shutdown_connections 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 1 -- 172.21.15.148:0/2189085846 >> 172.21.15.148:0/2189085846 conn(0x7f418c0a5640 msgr2=0x7f418c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 1 -- 172.21.15.148:0/2189085846 shutdown_connections 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 1 -- 172.21.15.148:0/2189085846 wait complete. 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 10 monclient: build_initial_monmap 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 10 monclient: monmap: 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:05.034 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.020+0000 7f419397f640 1 Processor -- start 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 1 -- start start 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient: init 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient(hunting): picked mon.noname-a con 0x7f418c09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient(hunting): picked mon.noname-b con 0x7f418c09c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient(hunting): picked mon.noname-c con 0x7f418c09cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419171b640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f418c09bb70 legacy=0x7f418c1314b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c11e930 con 0x7f418c09bb70 2024-09-16T04:21:05.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c120c20 con 0x7f418c09c660 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418c131bc0 con 0x7f418c09cc30 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f419397f640 10 monclient(hunting): authenticate will time out at 1685.646469s 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4191f1c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f418c09cc30 0x7f418c1cfa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37026/0 (socket says 172.21.15.148:37026) 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4191f1c640 1 -- 172.21.15.148:0/370999257 learned_addr learned my addr 172.21.15.148:0/370999257 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752199543 0 0) 0x7f418c131bc0 con 0x7f418c09cc30 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): my global_id is 124241 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4170003520 con 0x7f418c09cc30 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3908188804 0 0) 0x7f4170003520 con 0x7f418c09cc30 2024-09-16T04:21:05.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f418c131bc0 con 0x7f418c09cc30 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4184001cb0 con 0x7f418c09cc30 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): dump: 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.037 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093765677 0 0) 0x7f418c120c20 con 0x7f418c09c660 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 10 monclient(hunting): my global_id is 7054 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.021+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4170003520 con 0x7f418c09c660 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2292053674 0 0) 0x7f418c131bc0 con 0x7f418c09cc30 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.171:6789/0 conn(0x7f418c09c660 legacy=0x7f418c1cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.148:6789/0 conn(0x7f418c09bb70 legacy=0x7f418c1314b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:21:05.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 monclient: found mon.c 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418c0a3c40 con 0x7f418c09cc30 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: _finish_auth 0 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 5 monclient: authenticate success, global_id 124241 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 10 monclient: _renew_subs 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f418c0740a0 con 0x7f418c09cc30 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4184003cc0 con 0x7f418c09cc30 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:05.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4184004090 con 0x7f418c09cc30 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f4181ffb640 10 monclient: dump: 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.040 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f41837fe640 4 set_mon_vals no callback set 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 10 monclient: _renew_subs 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.022+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f418c11f210 con 0x7f418c09cc30 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f41840042b0 con 0x7f418c09cc30 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f419397f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f418c073740 con 0x7f418c11dca0 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7f4184047e50 con 0x7f418c09cc30 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3958949197 0 0) 0x7f418c09df40 con 0x7f418c11dca0 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient(hunting): my global_id is 124244 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f418c131bc0 con 0x7f418c11dca0 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4190f1a640 10 monclient: get_auth_request con 0x7f4170038f40 auth_method 0 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2980190578 0 0) 0x7f418c131bc0 con 0x7f418c11dca0 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f418c09df40 con 0x7f418c11dca0 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4188001a20 con 0x7f418c11dca0 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 10 monclient: dump: 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.042 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1688304468 0 0) 0x7f418c09df40 con 0x7f418c11dca0 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.023+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f418c076f00 con 0x7f418c11dca0 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.024+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4188001880 con 0x7f418c11dca0 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.024+0000 7f4181ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:05.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.024+0000 7f4181ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:05.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.024+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.148:6790/0 conn(0x7f418c11dca0 legacy=0x7f418c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.024+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.148:6790/0 conn(0x7f418c11dca0 legacy=0x7f418c06f540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:21:05.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f419397f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:21:05.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f418c120480 con 0x7f418c02d650 2024-09-16T04:21:05.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896861147 0 0) 0x7f418c120480 con 0x7f418c02d650 2024-09-16T04:21:05.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient(hunting): my global_id is 124247 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f418c09df40 con 0x7f418c02d650 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 316837754 0 0) 0x7f418c09df40 con 0x7f418c02d650 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f418c120480 con 0x7f418c02d650 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4188004cd0 con 0x7f418c02d650 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 10 monclient: dump: 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.046 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 715821116 0 0) 0x7f418c120480 con 0x7f418c02d650 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.030+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f418c06bc20 con 0x7f418c02d650 2024-09-16T04:21:05.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3965 (unknown 0 0 4125391899) 0x7f4188005f90 con 0x7f418c02d650 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4181ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4181ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.148:6790/0 conn(0x7f418c02d650 legacy=0x7f418c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4181ffb640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.148:6790/0 conn(0x7f418c02d650 legacy=0x7f418c06fc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:21:05.048 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 60, 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 32379, 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:21:05.049 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:21:05.050 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:21:05.051 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:21:05.052 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:21:05.053 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:21:05.054 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:21:05.055 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:21:05.056 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:21:05.057 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:21:05.058 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.059 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.060 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:21:05.060 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.171:6812/7327273 conn(0x7f4170038f40 legacy=0x7f417003b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 10 monclient: shutdown 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 >> v1:172.21.15.148:6790/0 conn(0x7f418c09cc30 legacy=0x7f418c1cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4191f1c640 1 -- 172.21.15.148:0/370999257 reap_dead start 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f4191f1c640 1 -- 172.21.15.148:0/370999257 reap_dead start 2024-09-16T04:21:05.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 shutdown_connections 2024-09-16T04:21:05.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 >> 172.21.15.148:0/370999257 conn(0x7f418c0a5640 msgr2=0x7f418c0a29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:05.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 shutdown_connections 2024-09-16T04:21:05.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.031+0000 7f419397f640 1 -- 172.21.15.148:0/370999257 wait complete. 2024-09-16T04:21:05.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:21:05.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:21:05.062 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:21:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.159+0000 7ff48b55f640 10 monclient: get_monmap_and_config 2024-09-16T04:21:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.159+0000 7ff48b55f640 10 monclient: build_initial_monmap 2024-09-16T04:21:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.159+0000 7ff48b55f640 10 monclient: monmap: 2024-09-16T04:21:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:21:05.161 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:21:05.160816+0000 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:21:05.160816+0000 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:21:05.162 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.160+0000 7ff48b55f640 1 Processor -- start 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.160+0000 7ff48b55f640 1 -- start start 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.160+0000 7ff48b55f640 10 monclient: init 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient(hunting): picked mon.noname-a con 0x7ff48412abe0 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient(hunting): picked mon.noname-c con 0x7ff48409f360 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient(hunting): picked mon.noname-b con 0x7ff48409fb50 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:05.170 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4840a03f0 con 0x7ff48412abe0 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4840a06d0 con 0x7ff48409f360 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4840a09b0 con 0x7ff48409fb50 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient(hunting): authenticate will time out at 1685.786217s 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff4892fb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff48412abe0 legacy=0x7ff48412afc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff489afc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff48409fb50 0x7ff48413cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37056/0 (socket says 172.21.15.148:37056) 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff489afc640 1 -- 172.21.15.148:0/769249625 learned_addr learned my addr 172.21.15.148:0/769249625 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847382378 0 0) 0x7ff4840a09b0 con 0x7ff48409fb50 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient(hunting): my global_id is 124250 2024-09-16T04:21:05.171 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff470003520 con 0x7ff48409fb50 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 813285136 0 0) 0x7ff470003520 con 0x7ff48409fb50 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 >> v1:172.21.15.171:6789/0 conn(0x7ff48409f360 legacy=0x7ff4841397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 >> v1:172.21.15.148:6789/0 conn(0x7ff48412abe0 legacy=0x7ff48412afc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 monclient: found mon.noname-b 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4840a0c90 con 0x7ff48409fb50 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: _finish_auth 0 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff47c0019d0 con 0x7ff48409fb50 2024-09-16T04:21:05.172 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 5 monclient: authenticate success, global_id 124250 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: dump: 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.173 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff47c001c70 con 0x7ff48409fb50 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 1 -- 172.21.15.148:0/769249625 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff47c002080 con 0x7ff48409fb50 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 10 monclient: get_monmap_and_config success 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff48b55f640 4 set_mon_vals no callback set 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.161+0000 7ff47bfff640 10 monclient: dump: 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.174 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 10 monclient: shutdown 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 1 -- 172.21.15.148:0/769249625 >> v1:172.21.15.148:6790/0 conn(0x7ff48409fb50 legacy=0x7ff48413cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 1 -- 172.21.15.148:0/769249625 shutdown_connections 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 1 -- 172.21.15.148:0/769249625 >> 172.21.15.148:0/769249625 conn(0x7ff4840a5050 msgr2=0x7ff4840a5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:05.175 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 1 -- 172.21.15.148:0/769249625 shutdown_connections 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.162+0000 7ff48b55f640 1 -- 172.21.15.148:0/769249625 wait complete. 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient: build_initial_monmap 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient: monmap: 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:21:05.176 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 1 Processor -- start 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 1 -- start start 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient: init 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient: _reopen_session rank -1 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient(hunting): picked mon.noname-a con 0x7ff48409f360 addr v1:172.21.15.148:6789/0 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient(hunting): picked mon.noname-c con 0x7ff48409fb50 addr v1:172.21.15.148:6790/0 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient(hunting): picked mon.noname-b con 0x7ff48412abe0 addr v1:172.21.15.171:6789/0 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4841340d0 con 0x7ff48409f360 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff484072ea0 con 0x7ff48412abe0 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff4892fb640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7ff48409f360 legacy=0x7ff484072080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4841343b0 con 0x7ff48409fb50 2024-09-16T04:21:05.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient(hunting): _renew_subs 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff48b55f640 10 monclient(hunting): authenticate will time out at 1685.788843s 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff488afa640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff48409fb50 0x7ff484072790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37062/0 (socket says 172.21.15.148:37062) 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.163+0000 7ff488afa640 1 -- 172.21.15.148:0/2128002044 learned_addr learned my addr 172.21.15.148:0/2128002044 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782638340 0 0) 0x7ff4841343b0 con 0x7ff48409fb50 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): my global_id is 124253 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff468003520 con 0x7ff48409fb50 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626438095 0 0) 0x7ff484072ea0 con 0x7ff48412abe0 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.178 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): my global_id is 7060 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4841343b0 con 0x7ff48412abe0 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1086482650 0 0) 0x7ff468003520 con 0x7ff48409fb50 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff484072ea0 con 0x7ff48409fb50 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff474001930 con 0x7ff48409fb50 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): dump: 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.179 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2056966288 0 0) 0x7ff484072ea0 con 0x7ff48409fb50 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.171:6789/0 conn(0x7ff48412abe0 legacy=0x7ff4841d1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.148:6789/0 conn(0x7ff48409f360 legacy=0x7ff484072080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 monclient: found mon.c 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:21:05.180 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48409c240 con 0x7ff48409fb50 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: _finish_auth 0 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 5 monclient: authenticate success, global_id 124253 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 10 monclient: _renew_subs 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff484133df0 con 0x7ff48409fb50 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff474001d00 con 0x7ff48409fb50 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff474003a60 con 0x7ff48409fb50 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.181 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff479ffb640 10 monclient: dump: 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 10 monclient: _renew_subs 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff47b7fe640 4 set_mon_vals no callback set 2024-09-16T04:21:05.182 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.164+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff48413db40 con 0x7ff48409fb50 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff48b55f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff474003c00 con 0x7ff48409fb50 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff48412f5c0 con 0x7ff48413e610 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(255..255 src has 188..255) ==== 6817+0+0 (unknown 3831350731 0 0) 0x7ff4740479a0 con 0x7ff48409fb50 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff489afc640 10 monclient: get_auth_request con 0x7ff468038ef0 auth_method 0 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3605627221 0 0) 0x7ff48412f5c0 con 0x7ff48413e610 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 10 monclient(hunting): my global_id is 7063 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.165+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff484072ea0 con 0x7ff48413e610 2024-09-16T04:21:05.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 401855816 0 0) 0x7ff484072ea0 con 0x7ff48413e610 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff48412f5c0 con 0x7ff48413e610 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff480003f60 con 0x7ff48413e610 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 10 monclient: dump: 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.184 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1769618677 0 0) 0x7ff48412f5c0 con 0x7ff48413e610 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4840008d0 con 0x7ff48413e610 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff480006370 con 0x7ff48413e610 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.171:6789/0 conn(0x7ff48413e610 legacy=0x7ff48406eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.185 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.166+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.171:6789/0 conn(0x7ff48413e610 legacy=0x7ff48406eb90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.172+0000 7ff48b55f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.172+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff48402da60 con 0x7ff48402d650 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988372480 0 0) 0x7ff48402da60 con 0x7ff48402d650 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient(hunting): my global_id is 7066 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff48412f5c0 con 0x7ff48402d650 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 726745047 0 0) 0x7ff48412f5c0 con 0x7ff48402d650 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff48402da60 con 0x7ff48402d650 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff4800046a0 con 0x7ff48402d650 2024-09-16T04:21:05.187 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient: dump: 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:21:05.188 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3308148380 0 0) 0x7ff48402da60 con 0x7ff48402d650 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff48413ddd0 con 0x7ff48402d650 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 678625169) 0x7ff480005920 con 0x7ff48402d650 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.171:6789/0 conn(0x7ff48402d650 legacy=0x7ff48406f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.173+0000 7ff479ffb640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.171:6789/0 conn(0x7ff48402d650 legacy=0x7ff48406f2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:21:05.189 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 60, 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 889211, 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:21:05.190 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:21:05.191 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.192 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:21:05.193 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:21:05.194 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:21:05.195 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:21:05.196 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:21:05.197 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.198 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.199 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:21:05.200 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:21:05.201 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:21:05.202 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:21:05.202 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.171:6812/7327273 conn(0x7ff468038ef0 legacy=0x7ff46803b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 10 monclient: shutdown 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 >> v1:172.21.15.148:6790/0 conn(0x7ff48409fb50 legacy=0x7ff484072790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff489afc640 1 -- 172.21.15.148:0/2128002044 reap_dead start 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff489afc640 1 -- 172.21.15.148:0/2128002044 reap_dead start 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 shutdown_connections 2024-09-16T04:21:05.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 >> 172.21.15.148:0/2128002044 conn(0x7ff4840a5050 msgr2=0x7ff4840a3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:21:05.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 shutdown_connections 2024-09-16T04:21:05.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:21:05.174+0000 7ff48b55f640 1 -- 172.21.15.148:0/2128002044 wait complete. 2024-09-16T04:21:05.204 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:21:05.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:22:35.205 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:22:35.206 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:22:35.206 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:22:35.206 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:22:35.211 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:22:35.211 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:22:35.211 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:22:35.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.321+0000 7f1310d42640 10 monclient: get_monmap_and_config 2024-09-16T04:22:35.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.321+0000 7f1310d42640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.323 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.321+0000 7f1310d42640 10 monclient: monmap: 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:35.322868+0000 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:35.322868+0000 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:35.324 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.322+0000 7f1310d42640 1 Processor -- start 2024-09-16T04:22:35.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.322+0000 7f1310d42640 1 -- start start 2024-09-16T04:22:35.332 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.322+0000 7f1310d42640 10 monclient: init 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient(hunting): picked mon.noname-a con 0x7f130c09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient(hunting): picked mon.noname-b con 0x7f130c09c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient(hunting): picked mon.noname-c con 0x7f130c09cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c0a1010 con 0x7f130c09bb70 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c09f030 con 0x7f130c09cc30 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c09dc60 con 0x7f130c09c660 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1310d42640 10 monclient(hunting): authenticate will time out at 1775.948429s 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130a59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f130c09bb70 0x7f130c09bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36340/0 (socket says 172.21.15.148:36340) 2024-09-16T04:22:35.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f1309d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f130c09c660 0x7f130c138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33642/0 (socket says 172.21.15.148:33642) 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130a59c640 1 -- 172.21.15.148:0/3479672899 learned_addr learned my addr 172.21.15.148:0/3479672899 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855425480 0 0) 0x7f130c09f030 con 0x7f130c09cc30 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130959a640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130959a640 10 monclient(hunting): my global_id is 7069 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.323+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12f4003520 con 0x7f130c09cc30 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2028775160 0 0) 0x7f12f4003520 con 0x7f130c09cc30 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 >> v1:172.21.15.148:6790/0 conn(0x7f130c09c660 legacy=0x7f130c138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 >> v1:172.21.15.148:6789/0 conn(0x7f130c09bb70 legacy=0x7f130c09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 monclient: found mon.noname-c 2024-09-16T04:22:35.334 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c09df40 con 0x7f130c09cc30 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1304001990 con 0x7f130c09cc30 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: dump: 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.335 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 5 monclient: authenticate success, global_id 7069 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1304001c80 con 0x7f130c09cc30 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 1 -- 172.21.15.148:0/3479672899 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1304002010 con 0x7f130c09cc30 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 4 set_mon_vals no callback set 2024-09-16T04:22:35.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f130959a640 10 monclient: dump: 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 10 monclient: shutdown 2024-09-16T04:22:35.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.324+0000 7f1310d42640 1 -- 172.21.15.148:0/3479672899 >> v1:172.21.15.171:6789/0 conn(0x7f130c09cc30 legacy=0x7f130c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.325+0000 7f1310d42640 1 -- 172.21.15.148:0/3479672899 shutdown_connections 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.325+0000 7f1310d42640 1 -- 172.21.15.148:0/3479672899 >> 172.21.15.148:0/3479672899 conn(0x7f130c0a5640 msgr2=0x7f130c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.325+0000 7f1310d42640 1 -- 172.21.15.148:0/3479672899 shutdown_connections 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.325+0000 7f1310d42640 1 -- 172.21.15.148:0/3479672899 wait complete. 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.325+0000 7f1310d42640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.325+0000 7f1310d42640 10 monclient: monmap: 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.338 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 1 Processor -- start 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 1 -- start start 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient: init 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient(hunting): picked mon.noname-c con 0x7f130c09bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient(hunting): picked mon.noname-b con 0x7f130c09c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient(hunting): picked mon.noname-a con 0x7f130c09cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c11ba50 con 0x7f130c09cc30 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c13b9e0 con 0x7f130c09c660 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c06e5f0 con 0x7f130c09bb70 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f1310d42640 10 monclient(hunting): authenticate will time out at 1775.951785s 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f130a59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f130c09bb70 0x7f130c139490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33644/0 (socket says 172.21.15.148:33644) 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.326+0000 7f130a59c640 1 -- 172.21.15.148:0/1916704666 learned_addr learned my addr 172.21.15.148:0/1916704666 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2296234864 0 0) 0x7f130c06e5f0 con 0x7f130c09bb70 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): my global_id is 124256 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12ec003520 con 0x7f130c09bb70 2024-09-16T04:22:35.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993361667 0 0) 0x7f130c13b9e0 con 0x7f130c09c660 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): my global_id is 7072 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f130c06e5f0 con 0x7f130c09c660 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 622688229 0 0) 0x7f12ec003520 con 0x7f130c09bb70 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f130c13b9e0 con 0x7f130c09bb70 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f12fc0017a0 con 0x7f130c09bb70 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): dump: 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.341 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.342 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3588977418 0 0) 0x7f130c06e5f0 con 0x7f130c09c660 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12ec003520 con 0x7f130c09c660 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f13000017e0 con 0x7f130c09c660 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient(hunting): dump: 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.343 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3713447290 0 0) 0x7f130c13b9e0 con 0x7f130c09bb70 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 >> v1:172.21.15.171:6789/0 conn(0x7f130c09c660 legacy=0x7f130c139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 >> v1:172.21.15.148:6789/0 conn(0x7f130c09cc30 legacy=0x7f130c1d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 monclient: found mon.c 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c0740a0 con 0x7f130c09bb70 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 5 monclient: authenticate success, global_id 124256 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 10 monclient: _renew_subs 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f130c11b770 con 0x7f130c09bb70 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 10 monclient: _renew_subs 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f130c13c8d0 con 0x7f130c09bb70 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12fc0047e0 con 0x7f130c09bb70 2024-09-16T04:22:35.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f12fc004b30 con 0x7f130c09bb70 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f12faffd640 10 monclient: dump: 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.346 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.327+0000 7f1308d99640 4 set_mon_vals no callback set 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.328+0000 7f1310d42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.328+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f12fc004d90 con 0x7f130c09bb70 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.328+0000 7f1310d42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.328+0000 7f1310d42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.328+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f0005080 con 0x7f130c09bb70 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.329+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f12fc0479a0 con 0x7f130c09bb70 2024-09-16T04:22:35.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.329+0000 7f1309d9b640 10 monclient: get_auth_request con 0x7f12ec038ec0 auth_method 0 2024-09-16T04:22:35.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.336+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f12fc015d00 con 0x7f130c09bb70 2024-09-16T04:22:35.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.336+0000 7f12faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.336+0000 7f12faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f1310d42640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f1310d42640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f1310d42640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f12f0005370 con 0x7f130c09bb70 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f12faffd640 1 -- 172.21.15.148:0/1916704666 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 886448697) 0x7f12fc015600 con 0x7f130c09bb70 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f12faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:35.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.572+0000 7f12faffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:35.575 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:22:35.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 >> v1:172.21.15.171:6812/7327273 conn(0x7f12ec038ec0 legacy=0x7f12ec03b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 10 monclient: shutdown 2024-09-16T04:22:35.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 >> v1:172.21.15.148:6790/0 conn(0x7f130c09bb70 legacy=0x7f130c139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 shutdown_connections 2024-09-16T04:22:35.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 >> 172.21.15.148:0/1916704666 conn(0x7f130c0a5640 msgr2=0x7f130c0718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 shutdown_connections 2024-09-16T04:22:35.577 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.574+0000 7f1310d42640 1 -- 172.21.15.148:0/1916704666 wait complete. 2024-09-16T04:22:35.587 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:22:35.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:22:35.588 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:22:35.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.685+0000 7f00a354b640 10 monclient: get_monmap_and_config 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.685+0000 7f00a354b640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.685+0000 7f00a354b640 10 monclient: monmap: 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:35.686222+0000 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:35.686222+0000 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:35.687 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:35.688 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.685+0000 7f00a354b640 1 Processor -- start 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 1 -- start start 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient: init 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient(hunting): picked mon.noname-b con 0x7f009c134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient(hunting): picked mon.noname-a con 0x7f009c09e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.696 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient(hunting): picked mon.noname-c con 0x7f009c13ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c135360 con 0x7f009c09e240 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c134440 con 0x7f009c13ae80 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c09e650 con 0x7f009c134870 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a354b640 10 monclient(hunting): authenticate will time out at 1776.311657s 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a12e7640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f009c134870 0x7f009c134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33652/0 (socket says 172.21.15.148:33652) 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f00a12e7640 1 -- 172.21.15.148:0/1248576100 learned_addr learned my addr 172.21.15.148:0/1248576100 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024451228 0 0) 0x7f009c135360 con 0x7f009c09e240 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): my global_id is 144099 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0088003520 con 0x7f009c09e240 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926720956 0 0) 0x7f009c09e650 con 0x7f009c134870 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 10 monclient(hunting): my global_id is 124259 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.686+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c135360 con 0x7f009c134870 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900327434 0 0) 0x7f009c134440 con 0x7f009c13ae80 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient(hunting): my global_id is 7075 2024-09-16T04:22:35.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c09e650 con 0x7f009c13ae80 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 790292142 0 0) 0x7f0088003520 con 0x7f009c09e240 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 >> v1:172.21.15.148:6790/0 conn(0x7f009c134870 legacy=0x7f009c134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 >> v1:172.21.15.171:6789/0 conn(0x7f009c13ae80 legacy=0x7f009c13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 monclient: found mon.noname-a 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c073230 con 0x7f009c09e240 2024-09-16T04:22:35.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 5 monclient: authenticate success, global_id 144099 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0098001610 con 0x7f009c09e240 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: dump: 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.700 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0098001930 con 0x7f009c09e240 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 1 -- 172.21.15.148:0/1248576100 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0098001d00 con 0x7f009c09e240 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 4 set_mon_vals no callback set 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f0093fff640 10 monclient: dump: 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.702 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 10 monclient: shutdown 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 1 -- 172.21.15.148:0/1248576100 >> v1:172.21.15.148:6789/0 conn(0x7f009c09e240 legacy=0x7f009c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 1 -- 172.21.15.148:0/1248576100 shutdown_connections 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 1 -- 172.21.15.148:0/1248576100 >> 172.21.15.148:0/1248576100 conn(0x7f009c0a5640 msgr2=0x7f009c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 1 -- 172.21.15.148:0/1248576100 shutdown_connections 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.687+0000 7f00a354b640 1 -- 172.21.15.148:0/1248576100 wait complete. 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.688+0000 7f00a354b640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.688+0000 7f00a354b640 10 monclient: monmap: 2024-09-16T04:22:35.703 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.688+0000 7f00a354b640 1 Processor -- start 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.688+0000 7f00a354b640 1 -- start start 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.688+0000 7f00a354b640 10 monclient: init 2024-09-16T04:22:35.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient(hunting): picked mon.noname-c con 0x7f009c09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient(hunting): picked mon.noname-b con 0x7f009c134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient(hunting): picked mon.noname-a con 0x7f009c13ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c1ca100 con 0x7f009c13ae80 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c11cc20 con 0x7f009c134870 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f009c0740a0 con 0x7f009c09e240 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a354b640 10 monclient(hunting): authenticate will time out at 1776.314320s 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a12e7640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f009c09e240 0x7f009c131560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33660/0 (socket says 172.21.15.148:33660) 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f00a12e7640 1 -- 172.21.15.148:0/2225396023 learned_addr learned my addr 172.21.15.148:0/2225396023 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 12207445 0 0) 0x7f009c1ca100 con 0x7f009c13ae80 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): my global_id is 144102 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0080003520 con 0x7f009c13ae80 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085706785 0 0) 0x7f009c0740a0 con 0x7f009c09e240 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): my global_id is 124262 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c1ca100 con 0x7f009c09e240 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066825339 0 0) 0x7f009c11cc20 con 0x7f009c134870 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): my global_id is 7078 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c0740a0 con 0x7f009c134870 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3390692380 0 0) 0x7f0080003520 con 0x7f009c13ae80 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f009c11cc20 con 0x7f009c13ae80 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0094001c30 con 0x7f009c13ae80 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): dump: 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.707 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2061387523 0 0) 0x7f009c1ca100 con 0x7f009c09e240 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0080003520 con 0x7f009c09e240 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f008c0017f0 con 0x7f009c09e240 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): dump: 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2962998937 0 0) 0x7f009c0740a0 con 0x7f009c134870 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f009c1ca100 con 0x7f009c134870 2024-09-16T04:22:35.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f00980015d0 con 0x7f009c134870 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient(hunting): dump: 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.710 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1059914342 0 0) 0x7f009c11cc20 con 0x7f009c13ae80 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.148:6790/0 conn(0x7f009c09e240 legacy=0x7f009c131560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.171:6789/0 conn(0x7f009c134870 legacy=0x7f009c1ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.689+0000 7f0091ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 1 monclient: found mon.a 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c139f20 con 0x7f009c13ae80 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 5 monclient: authenticate success, global_id 144102 2024-09-16T04:22:35.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 10 monclient: _renew_subs 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f009c11c9a0 con 0x7f009c13ae80 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0094003990 con 0x7f009c13ae80 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0094003ce0 con 0x7f009c13ae80 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 10 monclient: dump: 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.712 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 10 monclient: _renew_subs 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00937fe640 4 set_mon_vals no callback set 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f009c11b770 con 0x7f009c13ae80 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.690+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f009400ef70 con 0x7f009c13ae80 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f0094046940 con 0x7f009c13ae80 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f00a354b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f009c0008d0 con 0x7f009c02d7c0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f00a12e7640 10 monclient: get_auth_request con 0x7f00800391a0 auth_method 0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699642883 0 0) 0x7f009c0008d0 con 0x7f009c02d7c0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient(hunting): my global_id is 144105 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c11cc20 con 0x7f009c02d7c0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1287762947 0 0) 0x7f009c11cc20 con 0x7f009c02d7c0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f009c0008d0 con 0x7f009c02d7c0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0098003920 con 0x7f009c02d7c0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient: dump: 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1633704263 0 0) 0x7f009c0008d0 con 0x7f009c02d7c0 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f009c02dba0 con 0x7f009c02d7c0 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0098005b80 con 0x7f009c02d7c0 2024-09-16T04:22:35.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.148:6789/0 conn(0x7f009c02d7c0 legacy=0x7f009c068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.691+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.148:6789/0 conn(0x7f009c02d7c0 legacy=0x7f009c068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:35.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.697+0000 7f00a354b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.697+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f009c071a10 con 0x7f009c1c91e0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 611878600 0 0) 0x7f009c071a10 con 0x7f009c1c91e0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient(hunting): my global_id is 144108 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c0008d0 con 0x7f009c1c91e0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4035578832 0 0) 0x7f009c0008d0 con 0x7f009c1c91e0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f009c071a10 con 0x7f009c1c91e0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0098005290 con 0x7f009c1c91e0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient: dump: 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 917719947 0 0) 0x7f009c071a10 con 0x7f009c1c91e0 2024-09-16T04:22:35.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f009c1c95c0 con 0x7f009c1c91e0 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 3453399432) 0x7f00980047d0 con 0x7f009c1c91e0 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.148:6789/0 conn(0x7f009c1c91e0 legacy=0x7f009c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.698+0000 7f0091ffb640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.148:6789/0 conn(0x7f009c1c91e0 legacy=0x7f009c071300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:35.720 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 64, 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 397, 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:35.721 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:35.722 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.723 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:35.724 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:35.725 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.726 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.727 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:35.728 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:35.729 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:35.730 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.731 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.732 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.732 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:35.732 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.171:6812/7327273 conn(0x7f00800391a0 legacy=0x7f008003b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 10 monclient: shutdown 2024-09-16T04:22:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 >> v1:172.21.15.148:6789/0 conn(0x7f009c13ae80 legacy=0x7f009c1d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a1ae8640 1 -- 172.21.15.148:0/2225396023 reap_dead start 2024-09-16T04:22:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a1ae8640 1 -- 172.21.15.148:0/2225396023 reap_dead start 2024-09-16T04:22:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 shutdown_connections 2024-09-16T04:22:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 >> 172.21.15.148:0/2225396023 conn(0x7f009c0a5640 msgr2=0x7f009c0a33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 shutdown_connections 2024-09-16T04:22:35.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.699+0000 7f00a354b640 1 -- 172.21.15.148:0/2225396023 wait complete. 2024-09-16T04:22:35.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:22:35.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:22:35.734 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:22:35.832 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.830+0000 7f8893f4b640 10 monclient: get_monmap_and_config 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.830+0000 7f8893f4b640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.830+0000 7f8893f4b640 10 monclient: monmap: 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:35.831469+0000 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:35.831469+0000 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.833 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 1 Processor -- start 2024-09-16T04:22:35.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 1 -- start start 2024-09-16T04:22:35.834 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient: init 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f888c134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f888c09e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f888c13ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c135360 con 0x7f888c09e240 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c134440 con 0x7f888c13ae80 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c09e650 con 0x7f888c134870 2024-09-16T04:22:35.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8893f4b640 10 monclient(hunting): authenticate will time out at 1776.456934s 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8891ce7640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f888c134870 0x7f888c134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33670/0 (socket says 172.21.15.148:33670) 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.831+0000 7f8891ce7640 1 -- 172.21.15.148:0/4046781611 learned_addr learned my addr 172.21.15.148:0/4046781611 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720179180 0 0) 0x7f888c135360 con 0x7f888c09e240 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): my global_id is 144111 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8878003520 con 0x7f888c09e240 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2835717339 0 0) 0x7f888c09e650 con 0x7f888c134870 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient(hunting): my global_id is 124265 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888c135360 con 0x7f888c134870 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 609797172 0 0) 0x7f8878003520 con 0x7f888c09e240 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 >> v1:172.21.15.148:6790/0 conn(0x7f888c134870 legacy=0x7f888c134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 >> v1:172.21.15.171:6789/0 conn(0x7f888c13ae80 legacy=0x7f888c13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 monclient: found mon.noname-a 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888c073230 con 0x7f888c09e240 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8893f4b640 5 monclient: authenticate success, global_id 144111 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8893f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f887c001610 con 0x7f888c09e240 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: dump: 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.845 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8893f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f887c001870 con 0x7f888c09e240 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 1 -- 172.21.15.148:0/4046781611 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f887c001c40 con 0x7f888c09e240 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8893f4b640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8893f4b640 4 set_mon_vals no callback set 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.832+0000 7f8890ce5640 10 monclient: dump: 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.846 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 10 monclient: shutdown 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 1 -- 172.21.15.148:0/4046781611 >> v1:172.21.15.148:6789/0 conn(0x7f888c09e240 legacy=0x7f888c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 1 -- 172.21.15.148:0/4046781611 shutdown_connections 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 1 -- 172.21.15.148:0/4046781611 >> 172.21.15.148:0/4046781611 conn(0x7f888c0a5640 msgr2=0x7f888c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 1 -- 172.21.15.148:0/4046781611 shutdown_connections 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 1 -- 172.21.15.148:0/4046781611 wait complete. 2024-09-16T04:22:35.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.833+0000 7f8893f4b640 10 monclient: monmap: 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 1 Processor -- start 2024-09-16T04:22:35.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 1 -- start start 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient: init 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f888c09e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f888c134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f888c13ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c0740a0 con 0x7f888c09e240 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c0720b0 con 0x7f888c134870 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c0a3d70 con 0x7f888c13ae80 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8893f4b640 10 monclient(hunting): authenticate will time out at 1776.459660s 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8891ce7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f888c09e240 0x7f888c133710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36422/0 (socket says 172.21.15.148:36422) 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f8891ce7640 1 -- 172.21.15.148:0/2169322132 learned_addr learned my addr 172.21.15.148:0/2169322132 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1442782238 0 0) 0x7f888c0a3d70 con 0x7f888c13ae80 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): my global_id is 124268 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8870003520 con 0x7f888c13ae80 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409063674 0 0) 0x7f888c0740a0 con 0x7f888c09e240 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 10 monclient(hunting): my global_id is 144114 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.834+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888c0a3d70 con 0x7f888c09e240 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3422670297 0 0) 0x7f888c0720b0 con 0x7f888c134870 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): my global_id is 7084 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888c0740a0 con 0x7f888c134870 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2733394789 0 0) 0x7f888c0a3d70 con 0x7f888c09e240 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f888c0720b0 con 0x7f888c09e240 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88880012e0 con 0x7f888c09e240 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): dump: 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.851 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 392622691 0 0) 0x7f8870003520 con 0x7f888c13ae80 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f888c0a3d70 con 0x7f888c13ae80 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8884001af0 con 0x7f888c13ae80 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): dump: 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.852 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88924e8640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f888c13ae80 legacy=0x7f888c1d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88924e8640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f888c13ae80 legacy=0x7f888c1d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88924e8640 1 --1- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f888c13ae80 0x7f888c1d1a50 :-1 s=OPENED pgs=772 cs=1 l=1).handle_message read tag failed 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88924e8640 1 --1- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f888c13ae80 0x7f888c1d1a50 :-1 s=OPENED pgs=772 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:22:35.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6790/0 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 935015750 0 0) 0x7f888c0740a0 con 0x7f888c134870 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8870003520 con 0x7f888c134870 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f887c001bc0 con 0x7f888c134870 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient(hunting): dump: 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.854 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1084981518 0 0) 0x7f888c0720b0 con 0x7f888c09e240 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f888c13ae80 legacy=0x7f888c1d1a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.171:6789/0 conn(0x7f888c134870 legacy=0x7f888c1ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 monclient: found mon.a 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888c11e880 con 0x7f888c09e240 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 5 monclient: authenticate success, global_id 144114 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 10 monclient: _renew_subs 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f888c11eb60 con 0x7f888c09e240 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 10 monclient: _renew_subs 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f888c11d430 con 0x7f888c09e240 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88880047e0 con 0x7f888c09e240 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8888004b30 con 0x7f888c09e240 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f88827fc640 10 monclient: dump: 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.835+0000 7f8883fff640 4 set_mon_vals no callback set 2024-09-16T04:22:35.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f888800fdc0 con 0x7f888c09e240 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f8893f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88740065c0 con 0x7f8874002860 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f88880477c0 con 0x7f888c09e240 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 839769632 0 0) 0x7f88740065c0 con 0x7f8874002860 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient(hunting): my global_id is 124271 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f888c0720b0 con 0x7f8874002860 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2731647327 0 0) 0x7f888c0720b0 con 0x7f8874002860 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88740065c0 con 0x7f8874002860 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88924e8640 10 monclient: get_auth_request con 0x7f8870039250 auth_method 0 2024-09-16T04:22:35.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f887c0038f0 con 0x7f8874002860 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.836+0000 7f88827fc640 10 monclient: dump: 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.859 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3740970443 0 0) 0x7f88740065c0 con 0x7f8874002860 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f887400a150 con 0x7f8874002860 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f887c005b50 con 0x7f8874002860 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f8874002860 legacy=0x7f8874004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.837+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f8874002860 legacy=0x7f8874004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:35.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f8893f4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:35.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8874009e40 con 0x7f8874009100 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75969177 0 0) 0x7f8874009e40 con 0x7f8874009100 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient(hunting): my global_id is 124274 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88740065c0 con 0x7f8874009100 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3538341116 0 0) 0x7f88740065c0 con 0x7f8874009100 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8874009e40 con 0x7f8874009100 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f887c005270 con 0x7f8874009100 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 10 monclient: dump: 2024-09-16T04:22:35.862 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3355625250 0 0) 0x7f8874009e40 con 0x7f8874009100 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.843+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f887400c3d0 con 0x7f8874009100 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3977 (unknown 0 0 3052659751) 0x7f887c005180 con 0x7f8874009100 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:35.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f8874009100 legacy=0x7f8874009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88827fc640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6790/0 conn(0x7f8874009100 legacy=0x7f8874009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 64, 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 123191, 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:35.864 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:35.865 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:35.866 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:35.867 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:35.868 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:35.869 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:35.870 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:35.871 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:35.872 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.873 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:35.874 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.171:6812/7327273 conn(0x7f8870039250 legacy=0x7f887003b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 10 monclient: shutdown 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 >> v1:172.21.15.148:6789/0 conn(0x7f888c09e240 legacy=0x7f888c133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88924e8640 1 -- 172.21.15.148:0/2169322132 reap_dead start 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f88924e8640 1 -- 172.21.15.148:0/2169322132 reap_dead start 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 shutdown_connections 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 >> 172.21.15.148:0/2169322132 conn(0x7f888c0a5640 msgr2=0x7f888c0a32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 shutdown_connections 2024-09-16T04:22:35.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.844+0000 7f8893f4b640 1 -- 172.21.15.148:0/2169322132 wait complete. 2024-09-16T04:22:35.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:22:35.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:22:35.877 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.974+0000 7f74956d0640 10 monclient: get_monmap_and_config 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.974+0000 7f74956d0640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.974+0000 7f74956d0640 10 monclient: monmap: 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:35.975349+0000 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:35.975349+0000 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.976 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.977 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:35.977 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:35.977 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.974+0000 7f74956d0640 1 Processor -- start 2024-09-16T04:22:35.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 1 -- start start 2024-09-16T04:22:35.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient: init 2024-09-16T04:22:35.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:35.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient(hunting): picked mon.noname-a con 0x7f74900a25a0 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient(hunting): picked mon.noname-b con 0x7f74900a2980 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient(hunting): picked mon.noname-c con 0x7f749013d0d0 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749009ff00 con 0x7f74900a25a0 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74900a4cd0 con 0x7f749013d0d0 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74900a07d0 con 0x7f74900a2980 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f74956d0640 10 monclient(hunting): authenticate will time out at 1776.600573s 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748effd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f74900a25a0 0x7f74900a3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36432/0 (socket says 172.21.15.148:36432) 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748effd640 1 -- 172.21.15.148:0/1194572428 learned_addr learned my addr 172.21.15.148:0/1194572428 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748e7fc640 1 --1- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6790/0 conn(0x7f74900a2980 0x7f749013b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33712/0 (socket says 172.21.15.148:33712) 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748effd640 1 -- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6789/0 conn(0x7f74900a25a0 legacy=0x7f74900a3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748effd640 1 -- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6789/0 conn(0x7f74900a25a0 legacy=0x7f74900a3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748effd640 1 --1- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6789/0 conn(0x7f74900a25a0 0x7f74900a3d10 :-1 s=OPENED pgs=569 cs=1 l=1).handle_message read tag failed 2024-09-16T04:22:35.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748effd640 1 --1- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6789/0 conn(0x7f74900a25a0 0x7f74900a3d10 :-1 s=OPENED pgs=569 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4023852574 0 0) 0x7f74900a07d0 con 0x7f74900a2980 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 10 monclient(hunting): my global_id is 124277 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.975+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f747c003520 con 0x7f74900a2980 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824000446 0 0) 0x7f74900a4cd0 con 0x7f749013d0d0 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient(hunting): my global_id is 7087 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74900a07d0 con 0x7f749013d0d0 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3670627925 0 0) 0x7f747c003520 con 0x7f74900a2980 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 >> v1:172.21.15.171:6789/0 conn(0x7f749013d0d0 legacy=0x7f749013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6789/0 conn(0x7f74900a25a0 legacy=0x7f74900a3d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 monclient: found mon.noname-b 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74900a4970 con 0x7f74900a2980 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: _finish_auth 0 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7478001540 con 0x7f74900a2980 2024-09-16T04:22:35.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 5 monclient: authenticate success, global_id 124277 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: dump: 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.990 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7478001860 con 0x7f74900a2980 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 1 -- 172.21.15.148:0/1194572428 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7478001c30 con 0x7f74900a2980 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 4 set_mon_vals no callback set 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f748dffb640 10 monclient: dump: 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.991 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 10 monclient: shutdown 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 1 -- 172.21.15.148:0/1194572428 >> v1:172.21.15.148:6790/0 conn(0x7f74900a2980 legacy=0x7f749013b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 1 -- 172.21.15.148:0/1194572428 shutdown_connections 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 1 -- 172.21.15.148:0/1194572428 >> 172.21.15.148:0/1194572428 conn(0x7f74900a5960 msgr2=0x7f74900a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 1 -- 172.21.15.148:0/1194572428 shutdown_connections 2024-09-16T04:22:35.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.976+0000 7f74956d0640 1 -- 172.21.15.148:0/1194572428 wait complete. 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.977+0000 7f74956d0640 10 monclient: build_initial_monmap 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.977+0000 7f74956d0640 10 monclient: monmap: 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.977+0000 7f74956d0640 1 Processor -- start 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.977+0000 7f74956d0640 1 -- start start 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.977+0000 7f74956d0640 10 monclient: init 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient(hunting): picked mon.noname-c con 0x7f74900a25a0 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient(hunting): picked mon.noname-b con 0x7f74900a2980 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient(hunting): picked mon.noname-a con 0x7f749013d0d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749013acd0 con 0x7f749013d0d0 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7490137e40 con 0x7f74900a2980 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74900a8670 con 0x7f74900a25a0 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74956d0640 10 monclient(hunting): authenticate will time out at 1776.603456s 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f748effd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f74900a25a0 0x7f7490137020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33724/0 (socket says 172.21.15.148:33724) 2024-09-16T04:22:35.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f748effd640 1 -- 172.21.15.148:0/1692624945 learned_addr learned my addr 172.21.15.148:0/1692624945 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208659850 0 0) 0x7f74900a8670 con 0x7f74900a25a0 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): my global_id is 124280 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7470003520 con 0x7f74900a25a0 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320861709 0 0) 0x7f749013acd0 con 0x7f749013d0d0 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): my global_id is 144117 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74900a8670 con 0x7f749013d0d0 2024-09-16T04:22:35.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3606267237 0 0) 0x7f7490137e40 con 0x7f74900a2980 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): my global_id is 7090 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f749013acd0 con 0x7f74900a2980 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3946066995 0 0) 0x7f7470003520 con 0x7f74900a25a0 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7490137e40 con 0x7f74900a25a0 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7480001830 con 0x7f74900a25a0 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): dump: 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.996 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.978+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2940055699 0 0) 0x7f74900a8670 con 0x7f749013d0d0 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7470003520 con 0x7f749013d0d0 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7488001bb0 con 0x7f749013d0d0 2024-09-16T04:22:35.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient(hunting): dump: 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:35.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2976789428 0 0) 0x7f749013acd0 con 0x7f74900a2980 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74900a8670 con 0x7f74900a2980 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7478001500 con 0x7f74900a2980 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient(hunting): dump: 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:35.999 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3378073370 0 0) 0x7f7490137e40 con 0x7f74900a25a0 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.171:6789/0 conn(0x7f74900a2980 legacy=0x7f7490137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.148:6789/0 conn(0x7f749013d0d0 legacy=0x7f74901d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: _finish_hunting 0 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 monclient: found mon.c 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74900a8140 con 0x7f74900a25a0 2024-09-16T04:22:36.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: _finish_auth 0 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: _check_auth_tickets 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 5 monclient: authenticate success, global_id 124280 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 10 monclient: _renew_subs 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7490139a40 con 0x7f74900a25a0 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 10 monclient: _renew_subs 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f749013b3d0 con 0x7f74900a25a0 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74800047e0 con 0x7f74900a25a0 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:36.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7480004bb0 con 0x7f74900a25a0 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f748d7fa640 4 set_mon_vals no callback set 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74877fe640 10 monclient: dump: 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.002 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.979+0000 7f74956d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f74740065c0 con 0x7f7474002860 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7480004bb0 con 0x7f74900a25a0 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f7480047820 con 0x7f74900a25a0 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3324186865 0 0) 0x7f74740065c0 con 0x7f7474002860 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient(hunting): my global_id is 7093 2024-09-16T04:22:36.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7490137e40 con 0x7f7474002860 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f748f7fe640 10 monclient: get_auth_request con 0x7f74700391c0 auth_method 0 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 569163954 0 0) 0x7f7490137e40 con 0x7f7474002860 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f74740065c0 con 0x7f7474002860 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7478004a70 con 0x7f7474002860 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 10 monclient: dump: 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.004 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2106395456 0 0) 0x7f74740065c0 con 0x7f7474002860 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.980+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f747400a150 con 0x7f7474002860 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.981+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7478005f70 con 0x7f7474002860 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.981+0000 7f74877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.981+0000 7f74877fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.981+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.171:6789/0 conn(0x7f7474002860 legacy=0x7f7474004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.981+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.171:6789/0 conn(0x7f7474002860 legacy=0x7f7474004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:36.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74956d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:36.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7474009e20 con 0x7f74740090e0 2024-09-16T04:22:36.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666645133 0 0) 0x7f7474009e20 con 0x7f74740090e0 2024-09-16T04:22:36.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74877fe640 10 monclient(hunting): my global_id is 7096 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.987+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74740065c0 con 0x7f74740090e0 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3708924043 0 0) 0x7f74740065c0 con 0x7f74740090e0 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7474009e20 con 0x7f74740090e0 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7478006580 con 0x7f74740090e0 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 10 monclient: dump: 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.008 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2186228427 0 0) 0x7f7474009e20 con 0x7f74740090e0 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f747400c390 con 0x7f74740090e0 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 3548380080) 0x7f74780067b0 con 0x7f74740090e0 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.171:6789/0 conn(0x7f74740090e0 legacy=0x7f7474009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74877fe640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.171:6789/0 conn(0x7f74740090e0 legacy=0x7f7474009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 64, 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 980026, 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:36.010 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:36.011 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:36.012 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:36.013 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:36.014 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:36.015 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.016 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:36.017 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:36.018 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.019 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.020 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:36.021 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:36.022 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.171:6812/7327273 conn(0x7f74700391c0 legacy=0x7f747003b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 10 monclient: shutdown 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 >> v1:172.21.15.148:6790/0 conn(0x7f74900a25a0 legacy=0x7f7490137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f748f7fe640 1 -- 172.21.15.148:0/1692624945 reap_dead start 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f748f7fe640 1 -- 172.21.15.148:0/1692624945 reap_dead start 2024-09-16T04:22:36.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 shutdown_connections 2024-09-16T04:22:36.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 >> 172.21.15.148:0/1692624945 conn(0x7f74900a5960 msgr2=0x7f749013ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:36.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 shutdown_connections 2024-09-16T04:22:36.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:35.988+0000 7f74956d0640 1 -- 172.21.15.148:0/1692624945 wait complete. 2024-09-16T04:22:36.024 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:22:36.024 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:22:36.024 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:22:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.120+0000 7fd647b33640 10 monclient: get_monmap_and_config 2024-09-16T04:22:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.120+0000 7fd647b33640 10 monclient: build_initial_monmap 2024-09-16T04:22:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.120+0000 7fd647b33640 10 monclient: monmap: 2024-09-16T04:22:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:36.122 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:36.121793+0000 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:36.121793+0000 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:36.123 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.121+0000 7fd647b33640 1 Processor -- start 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.121+0000 7fd647b33640 1 -- start start 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.121+0000 7fd647b33640 10 monclient: init 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient(hunting): picked mon.noname-a con 0x7fd640134b40 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient(hunting): picked mon.noname-c con 0x7fd640134610 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient(hunting): picked mon.noname-b con 0x7fd64013b050 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6400a1910 con 0x7fd640134b40 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd640099690 con 0x7fd640134610 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd640099970 con 0x7fd64013b050 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:36.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 10 monclient(hunting): authenticate will time out at 1776.747189s 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6458cf640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd640134b40 0x7fd640134f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36456/0 (socket says 172.21.15.148:36456) 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6458cf640 1 -- 172.21.15.148:0/2446266450 learned_addr learned my addr 172.21.15.148:0/2446266450 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2619586175 0 0) 0x7fd6400a1910 con 0x7fd640134b40 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): my global_id is 144120 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd62c003520 con 0x7fd640134b40 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900220647 0 0) 0x7fd640099970 con 0x7fd64013b050 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): my global_id is 124283 2024-09-16T04:22:36.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6400a1910 con 0x7fd64013b050 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4261112 0 0) 0x7fd640099690 con 0x7fd640134610 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): my global_id is 7099 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd640099970 con 0x7fd640134610 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 823733936 0 0) 0x7fd6400a1910 con 0x7fd64013b050 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.171:6789/0 conn(0x7fd640134610 legacy=0x7fd640139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6789/0 conn(0x7fd640134b40 legacy=0x7fd640134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 monclient: found mon.noname-b 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:36.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64009f160 con 0x7fd64013b050 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: _finish_auth 0 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd638001970 con 0x7fd64013b050 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 5 monclient: authenticate success, global_id 124283 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: dump: 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.136 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd647b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6460d0640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6790/0 conn(0x7fd64013b050 legacy=0x7fd64013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6460d0640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6790/0 conn(0x7fd64013b050 legacy=0x7fd64013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6460d0640 1 --1- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6790/0 conn(0x7fd64013b050 0x7fd64013d440 :-1 s=OPENED pgs=777 cs=1 l=1).handle_message read tag failed 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6460d0640 1 --1- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6790/0 conn(0x7fd64013b050 0x7fd64013d440 :-1 s=OPENED pgs=777 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: ms_handle_reset current mon v1:172.21.15.148:6790/0 2024-09-16T04:22:36.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6790/0 conn(0x7fd64013b050 legacy=0x7fd64013d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): picked mon.c con 0x7fd62c004e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): picked mon.b con 0x7fd640134610 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 10 monclient(hunting): picked mon.a con 0x7fd640134b40 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd6400a1910 con 0x7fd640134b40 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.122+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd62c003520 con 0x7fd640134610 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd640099690 con 0x7fd62c004e90 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6790/0 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831911426 0 0) 0x7fd640099690 con 0x7fd62c004e90 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd62c00f100 con 0x7fd62c004e90 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515748818 0 0) 0x7fd6400a1910 con 0x7fd640134b40 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd640099690 con 0x7fd640134b40 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911556374 0 0) 0x7fd62c003520 con 0x7fd640134610 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.171:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fd6400a1910 con 0x7fd640134610 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 4161369374 0 0) 0x7fd62c00f100 con 0x7fd62c004e90 2024-09-16T04:22:36.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.171:6789/0 conn(0x7fd640134610 legacy=0x7fd62c003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6789/0 conn(0x7fd640134b40 legacy=0x7fd62c00d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 monclient: found mon.c 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=2+}) -- 0x7fd62c005600 con 0x7fd62c004e90 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: _finish_auth 0 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd63c001c40 con 0x7fd62c004e90 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:36.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: dump: 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd647b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 1 -- 172.21.15.148:0/2446266450 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd63c004910 con 0x7fd62c004e90 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd6448cd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:36.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd647b33640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd647b33640 4 set_mon_vals no callback set 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd647b33640 10 monclient: shutdown 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd647b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.123+0000 7fd647b33640 1 -- 172.21.15.148:0/2446266450 >> v1:172.21.15.148:6790/0 conn(0x7fd62c004e90 legacy=0x7fd62c0037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 1 -- 172.21.15.148:0/2446266450 shutdown_connections 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 1 -- 172.21.15.148:0/2446266450 >> 172.21.15.148:0/2446266450 conn(0x7fd6400a5860 msgr2=0x7fd6400a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 1 -- 172.21.15.148:0/2446266450 shutdown_connections 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 1 -- 172.21.15.148:0/2446266450 wait complete. 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 10 monclient: build_initial_monmap 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 10 monclient: monmap: 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:36.142 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.124+0000 7fd647b33640 1 Processor -- start 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 1 -- start start 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient: init 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient(hunting): picked mon.noname-b con 0x7fd62c004e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient(hunting): picked mon.noname-a con 0x7fd640134610 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:36.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient(hunting): picked mon.noname-c con 0x7fd640134b40 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd640132aa0 con 0x7fd640134610 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd640131e00 con 0x7fd62c004e90 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6401320e0 con 0x7fd640134b40 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd647b33640 10 monclient(hunting): authenticate will time out at 1776.750493s 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6460d0640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd640134b40 0x7fd6401d1e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33740/0 (socket says 172.21.15.148:33740) 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6460d0640 1 -- 172.21.15.148:0/3898738529 learned_addr learned my addr 172.21.15.148:0/3898738529 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3930354499 0 0) 0x7fd6401320e0 con 0x7fd640134b40 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): my global_id is 124286 2024-09-16T04:22:36.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd624003520 con 0x7fd640134b40 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405160712 0 0) 0x7fd6400a1910 con 0x7fd640134610 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): my global_id is 144123 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6401320e0 con 0x7fd640134610 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1915028209 0 0) 0x7fd640131e00 con 0x7fd62c004e90 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _negotiate 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): my global_id is 7102 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6400a1910 con 0x7fd62c004e90 2024-09-16T04:22:36.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1446311328 0 0) 0x7fd624003520 con 0x7fd640134b40 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd640131e00 con 0x7fd640134b40 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd6380020b0 con 0x7fd640134b40 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.125+0000 7fd6367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient(hunting): dump: 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.146 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1783894898 0 0) 0x7fd6401320e0 con 0x7fd640134610 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd624003520 con 0x7fd640134610 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd6300018f0 con 0x7fd640134610 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient(hunting): dump: 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.147 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4290372435 0 0) 0x7fd640131e00 con 0x7fd640134b40 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 >> v1:172.21.15.171:6789/0 conn(0x7fd62c004e90 legacy=0x7fd6401c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 >> v1:172.21.15.148:6789/0 conn(0x7fd640134610 legacy=0x7fd6401ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: _finish_hunting 0 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 monclient: found mon.c 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd640073910 con 0x7fd640134b40 2024-09-16T04:22:36.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: _finish_auth 0 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: _check_auth_tickets 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 5 monclient: authenticate success, global_id 124286 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 10 monclient: _renew_subs 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6400a40d0 con 0x7fd640134b40 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 10 monclient: _renew_subs 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd640131b20 con 0x7fd640134b40 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd638004da0 con 0x7fd640134b40 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd6380050f0 con 0x7fd640134b40 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:36.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd637fff640 4 set_mon_vals no callback set 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.126+0000 7fd6367fc640 10 monclient: dump: 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd638010520 con 0x7fd640134b40 2024-09-16T04:22:36.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd647b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd647b33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd647b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd628005080 con 0x7fd640134b40 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fd638047bd0 con 0x7fd640134b40 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.127+0000 7fd6458cf640 10 monclient: get_auth_request con 0x7fd624038ec0 auth_method 0 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.131+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd638015900 con 0x7fd640134b40 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.131+0000 7fd6367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:36.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.131+0000 7fd6367fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:36.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.363+0000 7fd647b33640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:22:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.363+0000 7fd647b33640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:22:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.363+0000 7fd647b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.363+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd628005b80 con 0x7fd640134b40 2024-09-16T04:22:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.364+0000 7fd6367fc640 1 -- 172.21.15.148:0/3898738529 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fd638044d70 con 0x7fd640134b40 2024-09-16T04:22:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.364+0000 7fd6367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:22:36.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.364+0000 7fd6367fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:36.366 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.364+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.366 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.364+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=3555037336,config=692619927,health=224038855,kv=3693054442,logm=1397008739}) 2024-09-16T04:22:36.366 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.364+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=9,logm=50} crc {auth=222861610,config=692619927,health=4076111,kv=2272835141,logm=3670601959}) 2024-09-16T04:22:36.366 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.364+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.366 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.364+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=3555037336,config=692619927,health=224038855,kv=3693054442,logm=1397008739}) 2024-09-16T04:22:36.366 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.364+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=1,health=11,kv=10,logm=49} crc {auth=1064118143,config=1228700967,health=224038855,kv=2270513034,logm=206135404}) 2024-09-16T04:22:36.367 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.365+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.367 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.365+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957737349}) 2024-09-16T04:22:36.367 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.365+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312115295}) 2024-09-16T04:22:36.367 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.365+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.367 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.365+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957737349}) 2024-09-16T04:22:36.367 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.365+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352449536}) 2024-09-16T04:22:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 >> v1:172.21.15.171:6812/7327273 conn(0x7fd624038ec0 legacy=0x7fd62403b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 10 monclient: shutdown 2024-09-16T04:22:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 >> v1:172.21.15.148:6790/0 conn(0x7fd640134b40 legacy=0x7fd6401d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 shutdown_connections 2024-09-16T04:22:36.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 >> 172.21.15.148:0/3898738529 conn(0x7fd6400a5860 msgr2=0x7fd640071630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:36.369 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.366+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.369 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.366+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806578984}) 2024-09-16T04:22:36.369 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.366+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282663895}) 2024-09-16T04:22:36.369 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.366+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.370 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.366+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806578984}) 2024-09-16T04:22:36.370 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.366+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2280986842}) 2024-09-16T04:22:36.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 shutdown_connections 2024-09-16T04:22:36.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:36.366+0000 7fd647b33640 1 -- 172.21.15.148:0/3898738529 wait complete. 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.367+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.367+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479095026}) 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.367+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474570318}) 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.367+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.367+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479095026}) 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.367+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2198061622}) 2024-09-16T04:22:36.372 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.368+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.368+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474886211}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.368+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3329148235}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.368+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.368+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474886211}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.368+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661439402}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.369+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.369+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247639755}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.369+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102603236}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.369+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.369+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247639755}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.369+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3589715222}) 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.373 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278694187}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240866222}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278694187}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152749211}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666711990}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=204749904}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666711990}) 2024-09-16T04:22:36.374 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.370+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235409930}) 2024-09-16T04:22:36.376 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.371+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.376 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.371+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026935767}) 2024-09-16T04:22:36.376 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.371+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3587380850}) 2024-09-16T04:22:36.376 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.371+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.371+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4026935767}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.371+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219382907}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.372+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.372+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382463954}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.372+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077909536}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.372+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.372+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382463954}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.372+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361184445}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.373+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.373+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584546719}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.373+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=385995495}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.373+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.373+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584546719}) 2024-09-16T04:22:36.377 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.373+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798186883}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.374+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.374+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460095642}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.374+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=131063369}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.374+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.374+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460095642}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.374+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624133195}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590315635}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385893885}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590315635}) 2024-09-16T04:22:36.378 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416514992}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422134507}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746465130}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422134507}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.375+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3847170840}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.376+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.376+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600119890}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.376+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639157603}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.376+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.376+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600119890}) 2024-09-16T04:22:36.383 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.376+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1825714506}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.377+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.377+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904153269}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.377+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327067217}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.377+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.377+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904153269}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.377+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2390023611}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.378+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.378+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038606807}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.378+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812605518}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.378+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.378+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038606807}) 2024-09-16T04:22:36.384 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.378+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=59830155}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303082418}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28031058}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303082418}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939974474}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229665647}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1172343593}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229665647}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.379+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578704635}) 2024-09-16T04:22:36.385 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.380+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.386 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.380+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903775426}) 2024-09-16T04:22:36.386 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.380+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241376011}) 2024-09-16T04:22:36.386 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.380+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.386 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.380+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903775426}) 2024-09-16T04:22:36.386 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.380+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577292017}) 2024-09-16T04:22:36.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:22:36.397 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.381+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.381+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217281306}) 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.381+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237312750}) 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.381+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.381+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217281306}) 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.381+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140610682}) 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.400 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547522677}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926272765}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547522677}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480702838}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760664106}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866015921}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760664106}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.382+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4160613742}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.383+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.383+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533691012}) 2024-09-16T04:22:36.401 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.383+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911787076}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.383+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.383+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533691012}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.383+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298948286}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.384+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.384+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777432303}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.384+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030035958}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.384+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.384+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777432303}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.384+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781833279}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799555018}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253321738}) 2024-09-16T04:22:36.402 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799555018}) 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836520399}) 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034610050}) 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3600001660}) 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034610050}) 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.385+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2308144451}) 2024-09-16T04:22:36.403 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.386+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.386+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326551678}) 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.386+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972147829}) 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.386+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.386+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326551678}) 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.386+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1905223476}) 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502515617}) 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=560950954}) 2024-09-16T04:22:36.404 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502515617}) 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225849452}) 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665879431}) 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264682346}) 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665879431}) 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.387+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815951511}) 2024-09-16T04:22:36.405 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.388+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.388+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388736704}) 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.388+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1120851761}) 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.388+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.388+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388736704}) 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.388+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292325007}) 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.389+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.389+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477013458}) 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.389+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705081381}) 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.389+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.406 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.389+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477013458}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.389+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470462396}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.390+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.390+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072461816}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.390+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=569048175}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.390+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.390+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072461816}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.390+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083709832}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180572107}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168646424}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180572107}) 2024-09-16T04:22:36.407 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=745758973}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481763016}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2535465184}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481763016}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.391+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052825647}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.392+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.392+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120095858}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.392+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86076102}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.392+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.392+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120095858}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.392+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175055444}) 2024-09-16T04:22:36.408 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.393+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.393+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709676125}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.393+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=673108377}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.393+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.393+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709676125}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.393+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3785964091}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893862916}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041283779}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893862916}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857272714}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229273026}) 2024-09-16T04:22:36.409 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1424977726}) 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229273026}) 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.394+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195724039}) 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.395+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.395+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507965964}) 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.395+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610929858}) 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.395+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.395+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507965964}) 2024-09-16T04:22:36.410 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.395+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202731013}) 2024-09-16T04:22:36.419 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243059361}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863716631}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243059361}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845553974}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572702714}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.396+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3897408150}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572702714}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226820229}) 2024-09-16T04:22:36.420 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163799879}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=897632953}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163799879}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.397+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3761113803}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.398+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.398+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200066089}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.398+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3090743258}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.398+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.398+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200066089}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.398+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007003402}) 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.421 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061486846}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149993784}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061486846}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258144785}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577326325}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=361938620}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577326325}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.399+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4183178551}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.400+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.400+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229906495}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.400+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3001563653}) 2024-09-16T04:22:36.422 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.400+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.400+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229906495}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.400+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402137732}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234000639}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607218008}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234000639}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1194234861}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128516788}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.401+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619720887}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128516788}) 2024-09-16T04:22:36.423 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383250630}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260685599}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290640152}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260685599}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.402+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355156461}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288018528}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848368976}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288018528}) 2024-09-16T04:22:36.424 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447497101}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.403+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253724428}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546785250}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253724428}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1580797814}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32791938}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963527225}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32791938}) 2024-09-16T04:22:36.425 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.404+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587122025}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.405+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.405+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336696383}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.405+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590506717}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.405+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.405+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336696383}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.405+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1861119230}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.406+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.406+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850729410}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.406+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3665768312,mdsmap=1201742139}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.406+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.406+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850729410}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.406+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846595734}) 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.407+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.426 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.407+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {logm=2771266781,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=190158057,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=227010982}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.407+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {logm=2771266781,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=1418076518,mon_config_key=193808867,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=2728575215}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.407+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.407+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {logm=2771266781,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=190158057,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=227010982}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.407+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {logm=2771266781,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=317834704,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=3583698692}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.409+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.409+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=11898749}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.409+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=3387306513}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.409+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.409+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=66} crc {osdmap=11898749}) 2024-09-16T04:22:36.427 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:22:36.409+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=68} crc {osdmap=505252375}) 2024-09-16T04:22:37.388 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:22:37.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:22:37.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.487+0000 7f02beb52640 10 monclient: get_monmap_and_config 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.487+0000 7f02beb52640 10 monclient: build_initial_monmap 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.487+0000 7f02beb52640 10 monclient: monmap: 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:37.488163+0000 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:37.488163+0000 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:37.489 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.487+0000 7f02beb52640 1 Processor -- start 2024-09-16T04:22:37.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 1 -- start start 2024-09-16T04:22:37.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient: init 2024-09-16T04:22:37.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient(hunting): picked mon.noname-a con 0x7f02b80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient(hunting): picked mon.noname-b con 0x7f02b80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient(hunting): picked mon.noname-c con 0x7f02b813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b80a0e00 con 0x7f02b80a2410 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b80a10e0 con 0x7f02b813d060 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b80a4970 con 0x7f02b80a1b10 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02beb52640 10 monclient(hunting): authenticate will time out at 1778.113533s 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02bc8ee640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f02b80a2410 0x7f02b80a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36482/0 (socket says 172.21.15.148:36482) 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02bc8ee640 1 -- 172.21.15.148:0/3473093482 learned_addr learned my addr 172.21.15.148:0/3473093482 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 281163015 0 0) 0x7f02b80a4970 con 0x7f02b80a1b10 2024-09-16T04:22:37.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): my global_id is 124289 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02a8003520 con 0x7f02b80a1b10 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503845702 0 0) 0x7f02b80a0e00 con 0x7f02b80a2410 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 10 monclient(hunting): my global_id is 144126 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.488+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b80a4970 con 0x7f02b80a2410 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305959233 0 0) 0x7f02b80a10e0 con 0x7f02b813d060 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient(hunting): my global_id is 7105 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b80a0e00 con 0x7f02b813d060 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 724359182 0 0) 0x7f02b80a4970 con 0x7f02b80a2410 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 >> v1:172.21.15.148:6790/0 conn(0x7f02b80a1b10 legacy=0x7f02b813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 >> v1:172.21.15.171:6789/0 conn(0x7f02b813d060 legacy=0x7f02b813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 monclient: found mon.noname-a 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b806dfd0 con 0x7f02b80a2410 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: _finish_auth 0 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:22:37.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 5 monclient: authenticate success, global_id 144126 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02ac001590 con 0x7f02b80a2410 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: dump: 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.501 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02ac0017f0 con 0x7f02b80a2410 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 1 -- 172.21.15.148:0/3473093482 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02ac001c00 con 0x7f02b80a2410 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 4 set_mon_vals no callback set 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:37.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02b77fe640 10 monclient: dump: 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 10 monclient: shutdown 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:37.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 1 -- 172.21.15.148:0/3473093482 >> v1:172.21.15.148:6789/0 conn(0x7f02b80a2410 legacy=0x7f02b80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 1 -- 172.21.15.148:0/3473093482 shutdown_connections 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 1 -- 172.21.15.148:0/3473093482 >> 172.21.15.148:0/3473093482 conn(0x7f02b80a97b0 msgr2=0x7f02b80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 1 -- 172.21.15.148:0/3473093482 shutdown_connections 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.489+0000 7f02beb52640 1 -- 172.21.15.148:0/3473093482 wait complete. 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.490+0000 7f02beb52640 10 monclient: build_initial_monmap 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.490+0000 7f02beb52640 10 monclient: monmap: 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:37.504 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.490+0000 7f02beb52640 1 Processor -- start 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.490+0000 7f02beb52640 1 -- start start 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.490+0000 7f02beb52640 10 monclient: init 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient(hunting): picked mon.noname-c con 0x7f02b80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient(hunting): picked mon.noname-a con 0x7f02b80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:37.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient(hunting): picked mon.noname-b con 0x7f02b813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b8137ba0 con 0x7f02b80a2410 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b809d6f0 con 0x7f02b813d060 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b813ca70 con 0x7f02b80a1b10 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02beb52640 10 monclient(hunting): authenticate will time out at 1778.116459s 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02bc8ee640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f02b80a1b10 0x7f02b809c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33768/0 (socket says 172.21.15.148:33768) 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02bc8ee640 1 -- 172.21.15.148:0/838366204 learned_addr learned my addr 172.21.15.148:0/838366204 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315443722 0 0) 0x7f02b813ca70 con 0x7f02b80a1b10 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): my global_id is 124292 2024-09-16T04:22:37.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f029c003550 con 0x7f02b80a1b10 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317705982 0 0) 0x7f02b8137ba0 con 0x7f02b80a2410 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): my global_id is 144129 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b813ca70 con 0x7f02b80a2410 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 666790023 0 0) 0x7f02b809d6f0 con 0x7f02b813d060 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): my global_id is 7108 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02b8137ba0 con 0x7f02b813d060 2024-09-16T04:22:37.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1302395415 0 0) 0x7f02b813ca70 con 0x7f02b80a2410 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02b809d6f0 con 0x7f02b80a2410 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02a4001890 con 0x7f02b80a2410 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): dump: 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.508 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.491+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1769605433 0 0) 0x7f029c003550 con 0x7f02b80a1b10 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f02b813ca70 con 0x7f02b80a1b10 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02ac001b40 con 0x7f02b80a1b10 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient(hunting): dump: 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1619444857 0 0) 0x7f02b8137ba0 con 0x7f02b813d060 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f029c003550 con 0x7f02b813d060 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02b0001b70 con 0x7f02b813d060 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient(hunting): dump: 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.510 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 265257450 0 0) 0x7f02b809d6f0 con 0x7f02b80a2410 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 >> v1:172.21.15.148:6790/0 conn(0x7f02b80a1b10 legacy=0x7f02b809c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 >> v1:172.21.15.171:6789/0 conn(0x7f02b813d060 legacy=0x7f02b81d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: _finish_hunting 0 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 monclient: found mon.a 2024-09-16T04:22:37.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b80a0130 con 0x7f02b80a2410 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: _finish_auth 0 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 5 monclient: authenticate success, global_id 144129 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 10 monclient: _renew_subs 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f02b81378c0 con 0x7f02b80a2410 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 10 monclient: _renew_subs 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f02b8136df0 con 0x7f02b80a2410 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02a4001c40 con 0x7f02b80a2410 2024-09-16T04:22:37.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f02a4003a40 con 0x7f02b80a2410 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b6ffd640 4 set_mon_vals no callback set 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.492+0000 7f02b57fa640 10 monclient: dump: 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.513 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f02a400ecd0 con 0x7f02b80a2410 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02beb52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02beb52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02beb52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0298005080 con 0x7f02b80a2410 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f02a4046490 con 0x7f02b80a2410 2024-09-16T04:22:37.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.493+0000 7f02bc8ee640 10 monclient: get_auth_request con 0x7f029c039130 auth_method 0 2024-09-16T04:22:37.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.497+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f02a40147d0 con 0x7f02b80a2410 2024-09-16T04:22:37.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.497+0000 7f02b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:37.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.497+0000 7f02b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02beb52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02beb52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02beb52640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0298005370 con 0x7f02b80a2410 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02b57fa640 1 -- 172.21.15.148:0/838366204 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 589537192) 0x7f02a40140d0 con 0x7f02b80a2410 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.730+0000 7f02b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:37.732 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 >> v1:172.21.15.171:6812/7327273 conn(0x7f029c039130 legacy=0x7f029c03b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 10 monclient: shutdown 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 >> v1:172.21.15.148:6789/0 conn(0x7f02b80a2410 legacy=0x7f02b809cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 shutdown_connections 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 >> 172.21.15.148:0/838366204 conn(0x7f02b80a97b0 msgr2=0x7f02b8135ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 shutdown_connections 2024-09-16T04:22:37.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.732+0000 7f02beb52640 1 -- 172.21.15.148:0/838366204 wait complete. 2024-09-16T04:22:37.744 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:22:37.744 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:22:37.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:22:37.745 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.840+0000 7fa32aaac640 10 monclient: get_monmap_and_config 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.840+0000 7fa32aaac640 10 monclient: build_initial_monmap 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.840+0000 7fa32aaac640 10 monclient: monmap: 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:37.841575+0000 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:37.841575+0000 2024-09-16T04:22:37.842 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:37.843 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.843 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:37.843 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:37.843 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:37.843 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 1 Processor -- start 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 1 -- start start 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient: init 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient(hunting): picked mon.noname-a con 0x7fa324134870 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:37.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient(hunting): picked mon.noname-b con 0x7fa32409e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient(hunting): picked mon.noname-c con 0x7fa32413ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa324135360 con 0x7fa324134870 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa324134440 con 0x7fa32413ae80 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa32409e650 con 0x7fa32409e240 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.841+0000 7fa32aaac640 10 monclient(hunting): authenticate will time out at 1778.466899s 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31bfff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa32409e240 0x7fa324139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33772/0 (socket says 172.21.15.148:33772) 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31bfff640 1 -- 172.21.15.148:0/3791616877 learned_addr learned my addr 172.21.15.148:0/3791616877 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa328848640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa324134870 0x7fa324134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36502/0 (socket says 172.21.15.148:36502) 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2434284827 0 0) 0x7fa32409e650 con 0x7fa32409e240 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): my global_id is 124295 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa30c003520 con 0x7fa32409e240 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815073170 0 0) 0x7fa324135360 con 0x7fa324134870 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): my global_id is 144132 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa32409e650 con 0x7fa324134870 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3685410261 0 0) 0x7fa324134440 con 0x7fa32413ae80 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient(hunting): my global_id is 7111 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa324135360 con 0x7fa32413ae80 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3219978040 0 0) 0x7fa30c003520 con 0x7fa32409e240 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 >> v1:172.21.15.171:6789/0 conn(0x7fa32413ae80 legacy=0x7fa32413d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 >> v1:172.21.15.148:6789/0 conn(0x7fa324134870 legacy=0x7fa324134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 monclient: found mon.noname-b 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa324073230 con 0x7fa32409e240 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa314001610 con 0x7fa32409e240 2024-09-16T04:22:37.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa32aaac640 5 monclient: authenticate success, global_id 124295 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa32aaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: dump: 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.855 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa32aaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa314001930 con 0x7fa32409e240 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa32aaac640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 1 -- 172.21.15.148:0/3791616877 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa314001d40 con 0x7fa32409e240 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa32aaac640 4 set_mon_vals no callback set 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.842+0000 7fa31b7fe640 10 monclient: dump: 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.856 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 10 monclient: shutdown 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 1 -- 172.21.15.148:0/3791616877 >> v1:172.21.15.148:6790/0 conn(0x7fa32409e240 legacy=0x7fa324139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 1 -- 172.21.15.148:0/3791616877 shutdown_connections 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 1 -- 172.21.15.148:0/3791616877 >> 172.21.15.148:0/3791616877 conn(0x7fa3240a5640 msgr2=0x7fa3240a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 1 -- 172.21.15.148:0/3791616877 shutdown_connections 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 1 -- 172.21.15.148:0/3791616877 wait complete. 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 10 monclient: build_initial_monmap 2024-09-16T04:22:37.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.843+0000 7fa32aaac640 10 monclient: monmap: 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 1 Processor -- start 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 1 -- start start 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient: init 2024-09-16T04:22:37.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient(hunting): picked mon.noname-b con 0x7fa32409e240 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient(hunting): picked mon.noname-c con 0x7fa324134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient(hunting): picked mon.noname-a con 0x7fa32413ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa32411d7d0 con 0x7fa32413ae80 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3240a3f10 con 0x7fa32409e240 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa32411e6c0 con 0x7fa324134870 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa32aaac640 10 monclient(hunting): authenticate will time out at 1778.469723s 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa329049640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa32413ae80 0x7fa3241d19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36510/0 (socket says 172.21.15.148:36510) 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa31bfff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa324134870 0x7fa3241ce2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33776/0 (socket says 172.21.15.148:33776) 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa329049640 1 -- 172.21.15.148:0/3643148926 learned_addr learned my addr 172.21.15.148:0/3643148926 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4278912770 0 0) 0x7fa32411e6c0 con 0x7fa324134870 2024-09-16T04:22:37.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 10 monclient(hunting): my global_id is 124298 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa310003520 con 0x7fa324134870 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501949088 0 0) 0x7fa32411d7d0 con 0x7fa32413ae80 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.844+0000 7fa3197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): my global_id is 144135 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa32411e6c0 con 0x7fa32413ae80 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436823386 0 0) 0x7fa3240a3f10 con 0x7fa32409e240 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): my global_id is 7114 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa32411d7d0 con 0x7fa32409e240 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1500759263 0 0) 0x7fa310003520 con 0x7fa324134870 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3240a3f10 con 0x7fa324134870 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa314001430 con 0x7fa324134870 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:22:37.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): dump: 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3294691425 0 0) 0x7fa32411e6c0 con 0x7fa32413ae80 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa310003520 con 0x7fa32413ae80 2024-09-16T04:22:37.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa31c001c30 con 0x7fa32413ae80 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): dump: 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1726420076 0 0) 0x7fa32411d7d0 con 0x7fa32409e240 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa32411e6c0 con 0x7fa32409e240 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa320001890 con 0x7fa32409e240 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient(hunting): dump: 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.864 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3014092710 0 0) 0x7fa3240a3f10 con 0x7fa324134870 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.171:6789/0 conn(0x7fa32409e240 legacy=0x7fa324133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.148:6789/0 conn(0x7fa32413ae80 legacy=0x7fa3241d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: _finish_hunting 0 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 monclient: found mon.c 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32411ec80 con 0x7fa324134870 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: _finish_auth 0 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: _check_auth_tickets 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 5 monclient: authenticate success, global_id 124298 2024-09-16T04:22:37.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 10 monclient: _renew_subs 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa32411d0d0 con 0x7fa324134870 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 10 monclient: _renew_subs 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa324133d90 con 0x7fa324134870 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa314004cc0 con 0x7fa324134870 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa314005010 con 0x7fa324134870 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa3197fa640 10 monclient: dump: 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.866 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.845+0000 7fa31affd640 4 set_mon_vals no callback set 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa32aaac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa3240008d0 con 0x7fa32402d7c0 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa3140102a0 con 0x7fa324134870 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fa3140476e0 con 0x7fa324134870 2024-09-16T04:22:37.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461448926 0 0) 0x7fa3240008d0 con 0x7fa32402d7c0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient(hunting): my global_id is 144138 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3240a3f10 con 0x7fa32402d7c0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa329049640 10 monclient: get_auth_request con 0x7fa3100391f0 auth_method 0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1803120809 0 0) 0x7fa3240a3f10 con 0x7fa32402d7c0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3240008d0 con 0x7fa32402d7c0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa3200047e0 con 0x7fa32402d7c0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.846+0000 7fa3197fa640 10 monclient: dump: 2024-09-16T04:22:37.868 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 899653858 0 0) 0x7fa3240008d0 con 0x7fa32402d7c0 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa32402dba0 con 0x7fa32402d7c0 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa320006b10 con 0x7fa32402d7c0 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:37.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:37.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.148:6789/0 conn(0x7fa32402d7c0 legacy=0x7fa324068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.847+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.148:6789/0 conn(0x7fa32402d7c0 legacy=0x7fa324068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa32aaac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:37.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa324136c50 con 0x7fa32411c250 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224071181 0 0) 0x7fa324136c50 con 0x7fa32411c250 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient(hunting): my global_id is 144141 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3240008d0 con 0x7fa32411c250 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1053252362 0 0) 0x7fa3240008d0 con 0x7fa32411c250 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa324136c50 con 0x7fa32411c250 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa320004e70 con 0x7fa32411c250 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:37.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.853+0000 7fa3197fa640 10 monclient: dump: 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1753373687 0 0) 0x7fa324136c50 con 0x7fa32411c250 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa32411e900 con 0x7fa32411c250 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3977 (unknown 0 0 2391251442) 0x7fa3200061e0 con 0x7fa32411c250 2024-09-16T04:22:37.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.148:6789/0 conn(0x7fa32411c250 legacy=0x7fa324136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa3197fa640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.148:6789/0 conn(0x7fa32411c250 legacy=0x7fa324136540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 64, 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 2553, 2024-09-16T04:22:37.874 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:37.875 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:37.876 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:37.877 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:37.878 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.879 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:37.880 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:37.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:37.882 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:37.883 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:37.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:37.885 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:37.885 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:37.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.171:6812/7327273 conn(0x7fa3100391f0 legacy=0x7fa31003b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 10 monclient: shutdown 2024-09-16T04:22:37.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:37.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 >> v1:172.21.15.148:6790/0 conn(0x7fa324134870 legacy=0x7fa3241ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa329049640 1 -- 172.21.15.148:0/3643148926 reap_dead start 2024-09-16T04:22:37.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa329049640 1 -- 172.21.15.148:0/3643148926 reap_dead start 2024-09-16T04:22:37.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 shutdown_connections 2024-09-16T04:22:37.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 >> 172.21.15.148:0/3643148926 conn(0x7fa3240a5640 msgr2=0x7fa324071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:37.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 shutdown_connections 2024-09-16T04:22:37.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.854+0000 7fa32aaac640 1 -- 172.21.15.148:0/3643148926 wait complete. 2024-09-16T04:22:37.887 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:22:37.887 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:22:37.887 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.984+0000 7fdac6cd9640 10 monclient: get_monmap_and_config 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.984+0000 7fdac6cd9640 10 monclient: build_initial_monmap 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.984+0000 7fdac6cd9640 10 monclient: monmap: 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:37.985349+0000 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:37.985349+0000 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:37.986 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:37.987 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 1 Processor -- start 2024-09-16T04:22:37.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 1 -- start start 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient: init 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient(hunting): picked mon.noname-b con 0x7fdac0134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient(hunting): picked mon.noname-a con 0x7fdac009e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient(hunting): picked mon.noname-c con 0x7fdac013ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac0135360 con 0x7fdac009e240 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac0134440 con 0x7fdac013ae80 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac009e650 con 0x7fdac0134870 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac6cd9640 10 monclient(hunting): authenticate will time out at 1778.610707s 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdabffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdac009e240 0x7fdac0139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36546/0 (socket says 172.21.15.148:36546) 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdac4a75640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fdac0134870 0x7fdac0134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33792/0 (socket says 172.21.15.148:33792) 2024-09-16T04:22:37.995 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.985+0000 7fdabffff640 1 -- 172.21.15.148:0/3128437271 learned_addr learned my addr 172.21.15.148:0/3128437271 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208982010 0 0) 0x7fdac009e650 con 0x7fdac0134870 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): my global_id is 124301 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab0003520 con 0x7fdac0134870 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119262057 0 0) 0x7fdac0135360 con 0x7fdac009e240 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): my global_id is 144144 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac009e650 con 0x7fdac009e240 2024-09-16T04:22:37.996 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037745901 0 0) 0x7fdac0134440 con 0x7fdac013ae80 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient(hunting): my global_id is 7117 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac0135360 con 0x7fdac013ae80 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 616640724 0 0) 0x7fdac009e650 con 0x7fdac009e240 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 >> v1:172.21.15.148:6790/0 conn(0x7fdac0134870 legacy=0x7fdac0134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 >> v1:172.21.15.171:6789/0 conn(0x7fdac013ae80 legacy=0x7fdac013d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 monclient: found mon.noname-a 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac0073230 con 0x7fdac009e240 2024-09-16T04:22:37.997 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 5 monclient: authenticate success, global_id 144144 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdaac0015f0 con 0x7fdac009e240 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: dump: 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:37.998 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdaac001950 con 0x7fdac009e240 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 1 -- 172.21.15.148:0/3128437271 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdaac0048a0 con 0x7fdac009e240 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 4 set_mon_vals no callback set 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdabf7fe640 10 monclient: dump: 2024-09-16T04:22:37.999 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 10 monclient: shutdown 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.986+0000 7fdac6cd9640 1 -- 172.21.15.148:0/3128437271 >> v1:172.21.15.148:6789/0 conn(0x7fdac009e240 legacy=0x7fdac0139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 1 -- 172.21.15.148:0/3128437271 shutdown_connections 2024-09-16T04:22:38.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 1 -- 172.21.15.148:0/3128437271 >> 172.21.15.148:0/3128437271 conn(0x7fdac00a5640 msgr2=0x7fdac00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 1 -- 172.21.15.148:0/3128437271 shutdown_connections 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 1 -- 172.21.15.148:0/3128437271 wait complete. 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 10 monclient: build_initial_monmap 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 10 monclient: monmap: 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:38.001 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 1 Processor -- start 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 1 -- start start 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.987+0000 7fdac6cd9640 10 monclient: init 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient(hunting): picked mon.noname-a con 0x7fdac009e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient(hunting): picked mon.noname-c con 0x7fdac0134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient(hunting): picked mon.noname-b con 0x7fdac013ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac011e970 con 0x7fdac009e240 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac011d7a0 con 0x7fdac013ae80 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac011e690 con 0x7fdac0134870 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac6cd9640 10 monclient(hunting): authenticate will time out at 1778.613622s 2024-09-16T04:22:38.002 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac4a75640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fdac009e240 0x7fdac0133550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36556/0 (socket says 172.21.15.148:36556) 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdac4a75640 1 -- 172.21.15.148:0/2262678144 learned_addr learned my addr 172.21.15.148:0/2262678144 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879858019 0 0) 0x7fdac011e970 con 0x7fdac009e240 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): my global_id is 144147 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaa4003520 con 0x7fdac009e240 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2176432291 0 0) 0x7fdac011e690 con 0x7fdac0134870 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 10 monclient(hunting): my global_id is 124304 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.988+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac011e970 con 0x7fdac0134870 2024-09-16T04:22:38.003 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1315040818 0 0) 0x7fdaa4003520 con 0x7fdac009e240 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdac011e690 con 0x7fdac009e240 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdab4001ab0 con 0x7fdac009e240 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): dump: 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.004 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2771515715 0 0) 0x7fdac011e970 con 0x7fdac0134870 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdaa4003520 con 0x7fdac0134870 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdaac001410 con 0x7fdac0134870 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): dump: 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.005 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317067746 0 0) 0x7fdac011d7a0 con 0x7fdac013ae80 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient(hunting): my global_id is 7120 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac011e970 con 0x7fdac013ae80 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3477070416 0 0) 0x7fdac011e690 con 0x7fdac009e240 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.148:6790/0 conn(0x7fdac0134870 legacy=0x7fdac01ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.006 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.171:6789/0 conn(0x7fdac013ae80 legacy=0x7fdac01d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 monclient: found mon.a 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac011ec50 con 0x7fdac009e240 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 5 monclient: authenticate success, global_id 144147 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 10 monclient: _renew_subs 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdac011d0a0 con 0x7fdac009e240 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdab4001d20 con 0x7fdac009e240 2024-09-16T04:22:38.007 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdab4004c30 con 0x7fdac009e240 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 10 monclient: _renew_subs 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdac0133c60 con 0x7fdac009e240 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabeffd640 4 set_mon_vals no callback set 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.989+0000 7fdabd7fa640 10 monclient: dump: 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.008 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fdab4010060 con 0x7fdac009e240 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdac6cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdac0071f90 con 0x7fdac002d7c0 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fdab40479f0 con 0x7fdac009e240 2024-09-16T04:22:38.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3605180811 0 0) 0x7fdaa4003520 con 0x7fdac002d7c0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient(hunting): my global_id is 124307 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdac011e690 con 0x7fdac002d7c0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdac5276640 10 monclient: get_auth_request con 0x7fdaa40390e0 auth_method 0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3293980196 0 0) 0x7fdac011e690 con 0x7fdac002d7c0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdaa4003520 con 0x7fdac002d7c0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdaac004ba0 con 0x7fdac002d7c0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 10 monclient: dump: 2024-09-16T04:22:38.010 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1498230634 0 0) 0x7fdaa4003520 con 0x7fdac002d7c0 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.990+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdac002dba0 con 0x7fdac002d7c0 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.991+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdaac004e50 con 0x7fdac002d7c0 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.991+0000 7fdabd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.011 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.991+0000 7fdabd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:38.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.991+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.148:6790/0 conn(0x7fdac002d7c0 legacy=0x7fdac0068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.012 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.991+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.148:6790/0 conn(0x7fdac002d7c0 legacy=0x7fdac0068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:38.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdac6cd9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:38.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdac00740a0 con 0x7fdac011c250 2024-09-16T04:22:38.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 634313461 0 0) 0x7fdac00740a0 con 0x7fdac011c250 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient(hunting): my global_id is 124310 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaa4003520 con 0x7fdac011c250 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3297519545 0 0) 0x7fdaa4003520 con 0x7fdac011c250 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdac00740a0 con 0x7fdac011c250 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fdaac005410 con 0x7fdac011c250 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient: dump: 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.014 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 832019184 0 0) 0x7fdac00740a0 con 0x7fdac011c250 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdac011c630 con 0x7fdac011c250 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3977 (unknown 0 0 3341493458) 0x7fdaac006780 con 0x7fdac011c250 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:38.015 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.148:6790/0 conn(0x7fdac011c250 legacy=0x7fdac0136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.997+0000 7fdabd7fa640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.148:6790/0 conn(0x7fdac011c250 legacy=0x7fdac0136540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 64, 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 125345, 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:38.016 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:38.017 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:38.018 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:38.019 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:38.020 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:38.021 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.022 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:38.023 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:38.024 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:38.025 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:38.026 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:38.027 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:38.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.171:6812/7327273 conn(0x7fdaa40390e0 legacy=0x7fdaa403b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 10 monclient: shutdown 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 >> v1:172.21.15.148:6789/0 conn(0x7fdac009e240 legacy=0x7fdac0133550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac5276640 1 -- 172.21.15.148:0/2262678144 reap_dead start 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac5276640 1 -- 172.21.15.148:0/2262678144 reap_dead start 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 shutdown_connections 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 >> 172.21.15.148:0/2262678144 conn(0x7fdac00a5640 msgr2=0x7fdac0071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 shutdown_connections 2024-09-16T04:22:38.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:37.998+0000 7fdac6cd9640 1 -- 172.21.15.148:0/2262678144 wait complete. 2024-09-16T04:22:38.029 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:22:38.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:22:38.030 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:22:38.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.127+0000 7f2b5e328640 10 monclient: get_monmap_and_config 2024-09-16T04:22:38.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.127+0000 7f2b5e328640 10 monclient: build_initial_monmap 2024-09-16T04:22:38.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.127+0000 7f2b5e328640 10 monclient: monmap: 2024-09-16T04:22:38.139 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:38.128853+0000 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:38.128853+0000 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.128+0000 7f2b5e328640 1 Processor -- start 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.128+0000 7f2b5e328640 1 -- start start 2024-09-16T04:22:38.140 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.128+0000 7f2b5e328640 10 monclient: init 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient(hunting): picked mon.noname-a con 0x7f2b58134870 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient(hunting): picked mon.noname-c con 0x7f2b5809e240 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient(hunting): picked mon.noname-b con 0x7f2b5813ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b58135360 con 0x7f2b58134870 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b58134440 con 0x7f2b5809e240 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b5809e650 con 0x7f2b5813ae80 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 10 monclient(hunting): authenticate will time out at 1778.754289s 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b57fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2b58134870 0x7f2b58134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36564/0 (socket says 172.21.15.148:36564) 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b57fff640 1 -- 172.21.15.148:0/3392370386 learned_addr learned my addr 172.21.15.148:0/3392370386 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:38.141 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982015631 0 0) 0x7f2b58135360 con 0x7f2b58134870 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): my global_id is 144150 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b40003520 con 0x7f2b58134870 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1085046102 0 0) 0x7f2b5809e650 con 0x7f2b5813ae80 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): my global_id is 124313 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b58135360 con 0x7f2b5813ae80 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423404174 0 0) 0x7f2b58134440 con 0x7f2b5809e240 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient(hunting): my global_id is 7123 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b5809e650 con 0x7f2b5809e240 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2310194091 0 0) 0x7f2b40003520 con 0x7f2b58134870 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 >> v1:172.21.15.148:6790/0 conn(0x7f2b5813ae80 legacy=0x7f2b5813d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 >> v1:172.21.15.171:6789/0 conn(0x7f2b5809e240 legacy=0x7f2b58139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 monclient: found mon.noname-a 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b58073230 con 0x7f2b58134870 2024-09-16T04:22:38.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: _finish_auth 0 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b48001610 con 0x7f2b58134870 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 5 monclient: authenticate success, global_id 144150 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b56ffd640 10 monclient: dump: 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.144 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.129+0000 7f2b5e328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b48001870 con 0x7f2b58134870 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b56ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b56ffd640 1 -- 172.21.15.148:0/3392370386 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b48001c80 con 0x7f2b58134870 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 4 set_mon_vals no callback set 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:38.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b56ffd640 10 monclient: dump: 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 10 monclient: shutdown 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:38.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 1 -- 172.21.15.148:0/3392370386 >> v1:172.21.15.148:6789/0 conn(0x7f2b58134870 legacy=0x7f2b58134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 1 -- 172.21.15.148:0/3392370386 shutdown_connections 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 1 -- 172.21.15.148:0/3392370386 >> 172.21.15.148:0/3392370386 conn(0x7f2b580a5640 msgr2=0x7f2b580a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 1 -- 172.21.15.148:0/3392370386 shutdown_connections 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.130+0000 7f2b5e328640 1 -- 172.21.15.148:0/3392370386 wait complete. 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.131+0000 7f2b5e328640 10 monclient: build_initial_monmap 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.131+0000 7f2b5e328640 10 monclient: monmap: 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:38.147 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.131+0000 7f2b5e328640 1 Processor -- start 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.131+0000 7f2b5e328640 1 -- start start 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.131+0000 7f2b5e328640 10 monclient: init 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient(hunting): picked mon.noname-c con 0x7f2b5809e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient(hunting): picked mon.noname-b con 0x7f2b58134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient(hunting): picked mon.noname-a con 0x7f2b5813ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:38.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b581cc0b0 con 0x7f2b5813ae80 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b58120bd0 con 0x7f2b58134870 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b58120eb0 con 0x7f2b5809e240 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient(hunting): authenticate will time out at 1778.757133s 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b57fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2b5809e240 0x7f2b581337d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:33844/0 (socket says 172.21.15.148:33844) 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b57fff640 1 -- 172.21.15.148:0/3644290898 learned_addr learned my addr 172.21.15.148:0/3644290898 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174516999 0 0) 0x7f2b58120eb0 con 0x7f2b5809e240 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): my global_id is 124316 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b44003520 con 0x7f2b5809e240 2024-09-16T04:22:38.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603490792 0 0) 0x7f2b581cc0b0 con 0x7f2b5813ae80 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): my global_id is 144153 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b58120eb0 con 0x7f2b5813ae80 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739393303 0 0) 0x7f2b58120bd0 con 0x7f2b58134870 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): my global_id is 7126 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b581cc0b0 con 0x7f2b58134870 2024-09-16T04:22:38.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1138066573 0 0) 0x7f2b44003520 con 0x7f2b5809e240 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b58120bd0 con 0x7f2b5809e240 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b48001bc0 con 0x7f2b5809e240 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): dump: 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.151 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2625037051 0 0) 0x7f2b58120eb0 con 0x7f2b5813ae80 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b44003520 con 0x7f2b5813ae80 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b50001d90 con 0x7f2b5813ae80 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient(hunting): dump: 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.152 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 173225040 0 0) 0x7f2b58120bd0 con 0x7f2b5809e240 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.171:6789/0 conn(0x7f2b58134870 legacy=0x7f2b581d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.148:6789/0 conn(0x7f2b5813ae80 legacy=0x7f2b581d3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 monclient: found mon.c 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:38.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b58072240 con 0x7f2b5809e240 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient: _finish_auth 0 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 5 monclient: authenticate success, global_id 124316 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient: _renew_subs 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5806e660 con 0x7f2b5809e240 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient: _renew_subs 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.132+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b5811fa00 con 0x7f2b5809e240 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b48001870 con 0x7f2b5809e240 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:38.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b48004d90 con 0x7f2b5809e240 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b567fc640 4 set_mon_vals no callback set 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 10 monclient: dump: 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.155 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b5e328640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2b380065c0 con 0x7f2b38002860 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.133+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2b48010020 con 0x7f2b5809e240 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f2b48047790 con 0x7f2b5809e240 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b5c8c5640 10 monclient: get_auth_request con 0x7f2b44038e30 auth_method 0 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2456370001 0 0) 0x7f2b380065c0 con 0x7f2b38002860 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient(hunting): my global_id is 7129 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b58120bd0 con 0x7f2b38002860 2024-09-16T04:22:38.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2236445192 0 0) 0x7f2b58120bd0 con 0x7f2b38002860 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b380065c0 con 0x7f2b38002860 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b4c0035f0 con 0x7f2b38002860 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 10 monclient: dump: 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.157 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2020011734 0 0) 0x7f2b380065c0 con 0x7f2b38002860 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.134+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b3800a150 con 0x7f2b38002860 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.135+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2b4c0059a0 con 0x7f2b38002860 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.135+0000 7f2b54ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.135+0000 7f2b54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.135+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.171:6789/0 conn(0x7f2b38002860 legacy=0x7f2b38004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.135+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.171:6789/0 conn(0x7f2b38002860 legacy=0x7f2b38004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.140+0000 7f2b5e328640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2b3800c4e0 con 0x7f2b380090a0 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182499237 0 0) 0x7f2b3800c4e0 con 0x7f2b380090a0 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient(hunting): my global_id is 7132 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b380065c0 con 0x7f2b380090a0 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2347063320 0 0) 0x7f2b380065c0 con 0x7f2b380090a0 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b3800c4e0 con 0x7f2b380090a0 2024-09-16T04:22:38.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b4c003c70 con 0x7f2b380090a0 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 10 monclient: dump: 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 729603317 0 0) 0x7f2b3800c4e0 con 0x7f2b380090a0 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.141+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b38009c80 con 0x7f2b380090a0 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 1524319608) 0x7f2b4c005bd0 con 0x7f2b380090a0 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b54ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.171:6789/0 conn(0x7f2b380090a0 legacy=0x7f2b38009530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b54ff9640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.171:6789/0 conn(0x7f2b380090a0 legacy=0x7f2b38009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:38.162 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 64, 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 982180, 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:38.163 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.164 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:38.165 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:38.166 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:38.167 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.168 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:38.169 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:38.170 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:38.171 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:38.172 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:22:38.173 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.174 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:38.175 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:38.176 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.171:6812/7327273 conn(0x7f2b44038e30 legacy=0x7f2b4403b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 10 monclient: shutdown 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 >> v1:172.21.15.148:6790/0 conn(0x7f2b5809e240 legacy=0x7f2b581337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5c8c5640 1 -- 172.21.15.148:0/3644290898 reap_dead start 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5c8c5640 1 -- 172.21.15.148:0/3644290898 reap_dead start 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 shutdown_connections 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 >> 172.21.15.148:0/3644290898 conn(0x7f2b580a5640 msgr2=0x7f2b580a33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 shutdown_connections 2024-09-16T04:22:38.177 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.142+0000 7f2b5e328640 1 -- 172.21.15.148:0/3644290898 wait complete. 2024-09-16T04:22:38.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:22:38.178 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:22:38.178 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:22:38.178 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:22:38.178 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:22:38.178 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:22:38.178 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:22:38.189 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:22:38.189 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:22:38.189 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.325+0000 7f7dbce4c640 10 monclient: get_monmap_and_config 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.325+0000 7f7dbce4c640 10 monclient: build_initial_monmap 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.325+0000 7f7dbce4c640 10 monclient: monmap: 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:38.326251+0000 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:38.326251+0000 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.327 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:38.328 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:38.328 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:38.328 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 1 Processor -- start 2024-09-16T04:22:38.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 1 -- start start 2024-09-16T04:22:38.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient: init 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient(hunting): picked mon.noname-b con 0x7f7db8133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient(hunting): picked mon.noname-a con 0x7f7db806c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient(hunting): picked mon.noname-c con 0x7f7db813d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db809b0d0 con 0x7f7db806c430 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db809b340 con 0x7f7db813d0e0 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db809b620 con 0x7f7db8133910 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7dbce4c640 10 monclient(hunting): authenticate will time out at 1778.951607s 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db659c640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7db8133910 legacy=0x7f7db8133cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7daffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7db806c430 0x7f7db813b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36582/0 (socket says 172.21.15.148:36582) 2024-09-16T04:22:38.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7daffff640 1 -- 172.21.15.148:0/2758778929 learned_addr learned my addr 172.21.15.148:0/2758778929 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1701147780 0 0) 0x7f7db809b0d0 con 0x7f7db806c430 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db5d9b640 10 monclient(hunting): my global_id is 144156 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.326+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7da4003520 con 0x7f7db806c430 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3157277078 0 0) 0x7f7da4003520 con 0x7f7db806c430 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 >> v1:172.21.15.148:6790/0 conn(0x7f7db8133910 legacy=0x7f7db8133cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 >> v1:172.21.15.171:6789/0 conn(0x7f7db813d0e0 legacy=0x7f7db813f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:22:38.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db80633c0 con 0x7f7db806c430 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7da0001610 con 0x7f7db806c430 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 5 monclient: authenticate success, global_id 144156 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: dump: 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.338 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7da0001970 con 0x7f7db806c430 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 1 -- 172.21.15.148:0/2758778929 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7da00048a0 con 0x7f7db806c430 2024-09-16T04:22:38.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 4 set_mon_vals no callback set 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7db5d9b640 10 monclient: dump: 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.340 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 10 monclient: shutdown 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 1 -- 172.21.15.148:0/2758778929 >> v1:172.21.15.148:6789/0 conn(0x7f7db806c430 legacy=0x7f7db813b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 1 -- 172.21.15.148:0/2758778929 shutdown_connections 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 1 -- 172.21.15.148:0/2758778929 >> 172.21.15.148:0/2758778929 conn(0x7f7db8062630 msgr2=0x7f7db8062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 1 -- 172.21.15.148:0/2758778929 shutdown_connections 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.327+0000 7f7dbce4c640 1 -- 172.21.15.148:0/2758778929 wait complete. 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.328+0000 7f7dbce4c640 10 monclient: build_initial_monmap 2024-09-16T04:22:38.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.328+0000 7f7dbce4c640 10 monclient: monmap: 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.328+0000 7f7dbce4c640 1 Processor -- start 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.328+0000 7f7dbce4c640 1 -- start start 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.328+0000 7f7dbce4c640 10 monclient: init 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient(hunting): picked mon.noname-a con 0x7f7db806c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient(hunting): picked mon.noname-c con 0x7f7db8133910 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient(hunting): picked mon.noname-b con 0x7f7db813d0e0 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db8134630 con 0x7f7db806c430 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db81308d0 con 0x7f7db813d0e0 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daffff640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7db8133910 legacy=0x7f7db81301c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db81322b0 con 0x7f7db8133910 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:38.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7dbce4c640 10 monclient(hunting): authenticate will time out at 1778.954453s 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7db659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7db806c430 0x7f7db812fab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36598/0 (socket says 172.21.15.148:36598) 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7db659c640 1 -- 172.21.15.148:0/4200178398 learned_addr learned my addr 172.21.15.148:0/4200178398 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922855392 0 0) 0x7f7db8134630 con 0x7f7db806c430 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): my global_id is 144159 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d98003520 con 0x7f7db806c430 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2753281753 0 0) 0x7f7d98003520 con 0x7f7db806c430 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7db8134630 con 0x7f7db806c430 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7da8004150 con 0x7f7db806c430 2024-09-16T04:22:38.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): dump: 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.345 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.329+0000 7f7daeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1666255808 0 0) 0x7f7db81308d0 con 0x7f7db813d0e0 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient(hunting): my global_id is 7138 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d98003520 con 0x7f7db813d0e0 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2919984417 0 0) 0x7f7db8134630 con 0x7f7db806c430 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 >> v1:172.21.15.148:6790/0 conn(0x7f7db8133910 legacy=0x7f7db81301c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 >> v1:172.21.15.171:6789/0 conn(0x7f7db813d0e0 legacy=0x7f7db81d3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:38.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 monclient: found mon.a 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db8134910 con 0x7f7db806c430 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: _finish_auth 0 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 5 monclient: authenticate success, global_id 144159 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 monclient: _renew_subs 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7db80a5180 con 0x7f7db806c430 2024-09-16T04:22:38.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 monclient: _renew_subs 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7db8132030 con 0x7f7db806c430 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7da8004480 con 0x7f7db806c430 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7da80047d0 con 0x7f7db806c430 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7db559a640 4 set_mon_vals no callback set 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 10 monclient: dump: 2024-09-16T04:22:38.348 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7da80049f0 con 0x7f7db806c430 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.330+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d9c005080 con 0x7f7db806c430 2024-09-16T04:22:38.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.331+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f7da8047b40 con 0x7f7db806c430 2024-09-16T04:22:38.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.331+0000 7f7daffff640 10 monclient: get_auth_request con 0x7f7d98038f70 auth_method 0 2024-09-16T04:22:38.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.335+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7da8015da0 con 0x7f7db806c430 2024-09-16T04:22:38.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.335+0000 7f7daeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:38.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.335+0000 7f7daeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7dbce4c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7dbce4c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7dbce4c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d9c005370 con 0x7f7db806c430 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7daeffd640 1 -- 172.21.15.148:0/4200178398 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2855238168) 0x7f7da80156a0 con 0x7f7db806c430 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7daeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:38.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.570+0000 7f7daeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:38.573 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:22:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.572+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 >> v1:172.21.15.171:6812/7327273 conn(0x7f7d98038f70 legacy=0x7f7d9803b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.572+0000 7f7dbce4c640 10 monclient: shutdown 2024-09-16T04:22:38.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.572+0000 7f7dbce4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.572+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 >> v1:172.21.15.148:6789/0 conn(0x7f7db806c430 legacy=0x7f7db812fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.573+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 shutdown_connections 2024-09-16T04:22:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.573+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 >> 172.21.15.148:0/4200178398 conn(0x7f7db8062630 msgr2=0x7f7db80a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.573+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 shutdown_connections 2024-09-16T04:22:38.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:38.573+0000 7f7dbce4c640 1 -- 172.21.15.148:0/4200178398 wait complete. 2024-09-16T04:22:41.585 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:22:41.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.691+0000 7f9edd4ba640 10 monclient: get_monmap_and_config 2024-09-16T04:22:41.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.691+0000 7f9edd4ba640 10 monclient: build_initial_monmap 2024-09-16T04:22:41.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.691+0000 7f9edd4ba640 10 monclient: monmap: 2024-09-16T04:22:41.693 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:41.693 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:41.692811+0000 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:41.692811+0000 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:41.694 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.692+0000 7f9edd4ba640 1 Processor -- start 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.692+0000 7f9edd4ba640 1 -- start start 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.692+0000 7f9edd4ba640 10 monclient: init 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f9ed809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f9ed809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9ed809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:41.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed80a1010 con 0x7f9ed809c660 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed809f030 con 0x7f9ed809cc30 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed6ffd640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f9ed809bb70 legacy=0x7f9ed809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed809dc60 con 0x7f9ed809bb70 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient(hunting): authenticate will time out at 1782.318190s 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed67fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9ed809c660 0x7f9ed8138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36600/0 (socket says 172.21.15.148:36600) 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed67fc640 1 -- 172.21.15.148:0/3184051828 learned_addr learned my addr 172.21.15.148:0/3184051828 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447908181 0 0) 0x7f9ed80a1010 con 0x7f9ed809c660 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient(hunting): my global_id is 144162 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ec0003520 con 0x7f9ed809c660 2024-09-16T04:22:41.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2744572409 0 0) 0x7f9ec0003520 con 0x7f9ed809c660 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 >> v1:172.21.15.148:6790/0 conn(0x7f9ed809bb70 legacy=0x7f9ed809bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 >> v1:172.21.15.171:6789/0 conn(0x7f9ed809cc30 legacy=0x7f9ed813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 monclient: found mon.noname-a 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed809df40 con 0x7f9ed809c660 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ecc001990 con 0x7f9ed809c660 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: dump: 2024-09-16T04:22:41.703 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 5 monclient: authenticate success, global_id 144162 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ecc001c80 con 0x7f9ed809c660 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:41.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 1 -- 172.21.15.148:0/3184051828 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ecc002090 con 0x7f9ed809c660 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9edd4ba640 4 set_mon_vals no callback set 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.693+0000 7f9ed5ffb640 10 monclient: dump: 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:41.705 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 10 monclient: shutdown 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 1 -- 172.21.15.148:0/3184051828 >> v1:172.21.15.148:6789/0 conn(0x7f9ed809c660 legacy=0x7f9ed8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 1 -- 172.21.15.148:0/3184051828 shutdown_connections 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 1 -- 172.21.15.148:0/3184051828 >> 172.21.15.148:0/3184051828 conn(0x7f9ed80a5640 msgr2=0x7f9ed80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 1 -- 172.21.15.148:0/3184051828 shutdown_connections 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 1 -- 172.21.15.148:0/3184051828 wait complete. 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 10 monclient: build_initial_monmap 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.694+0000 7f9edd4ba640 10 monclient: monmap: 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:41.706 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 1 Processor -- start 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 1 -- start start 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 10 monclient: init 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 10 monclient(hunting): picked mon.noname-b con 0x7f9ed809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:41.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9ed809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 10 monclient(hunting): picked mon.noname-a con 0x7f9ed809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed811f800 con 0x7f9ed809cc30 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.695+0000 7f9edd4ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed8121370 con 0x7f9ed809bb70 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ed67fc640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f9ed809c660 legacy=0x7f9ed811ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed811e4e0 con 0x7f9ed809c660 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 10 monclient(hunting): authenticate will time out at 1782.321063s 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ed77fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9ed809cc30 0x7f9ed81cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:36608/0 (socket says 172.21.15.148:36608) 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ed77fe640 1 -- 172.21.15.148:0/150250794 learned_addr learned my addr 172.21.15.148:0/150250794 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056840610 0 0) 0x7f9ed811f800 con 0x7f9ed809cc30 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:41.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): my global_id is 144165 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ec4003520 con 0x7f9ed809cc30 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1823235955 0 0) 0x7f9ec4003520 con 0x7f9ed809cc30 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ed811f800 con 0x7f9ed809cc30 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ed00018b0 con 0x7f9ed809cc30 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): dump: 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:41.709 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 528050539 0 0) 0x7f9ed811f800 con 0x7f9ed809cc30 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 >> v1:172.21.15.148:6790/0 conn(0x7f9ed809c660 legacy=0x7f9ed811ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 >> v1:172.21.15.171:6789/0 conn(0x7f9ed809bb70 legacy=0x7f9ed811d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 monclient: found mon.a 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed811fec0 con 0x7f9ed809cc30 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:22:41.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 5 monclient: authenticate success, global_id 144165 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 10 monclient: _renew_subs 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed80a4040 con 0x7f9ed809cc30 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 10 monclient: _renew_subs 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ed8121090 con 0x7f9ed809cc30 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9ed0003610 con 0x7f9ed809cc30 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9ed00039e0 con 0x7f9ed809cc30 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ed57fa640 4 set_mon_vals no callback set 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.696+0000 7f9ebf7fe640 10 monclient: dump: 2024-09-16T04:22:41.711 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9ed0003b80 con 0x7f9ed809cc30 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f9ed0047960 con 0x7f9ed809cc30 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9edd4ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9edd4ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9edd4ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:41.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb8005080 con 0x7f9ed809cc30 2024-09-16T04:22:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.697+0000 7f9ed6ffd640 10 monclient: get_auth_request con 0x7f9ec4038f30 auth_method 0 2024-09-16T04:22:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.702+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9ed0015c20 con 0x7f9ed809cc30 2024-09-16T04:22:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.702+0000 7f9ebf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:41.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.702+0000 7f9ebf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:41.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9edd4ba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:22:41.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9edd4ba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:41.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9edd4ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:41.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9eb8005370 con 0x7f9ed809cc30 2024-09-16T04:22:41.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9ebf7fe640 1 -- 172.21.15.148:0/150250794 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 214492298) 0x7f9ed00152b0 con 0x7f9ed809cc30 2024-09-16T04:22:41.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9ebf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:41.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.931+0000 7f9ebf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:41.938 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:22:41.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 >> v1:172.21.15.171:6812/7327273 conn(0x7f9ec4038f30 legacy=0x7f9ec403b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 10 monclient: shutdown 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 >> v1:172.21.15.148:6789/0 conn(0x7f9ed809cc30 legacy=0x7f9ed81cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 shutdown_connections 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 >> 172.21.15.148:0/150250794 conn(0x7f9ed80a5640 msgr2=0x7f9ed80a3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 shutdown_connections 2024-09-16T04:22:41.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:41.933+0000 7f9edd4ba640 1 -- 172.21.15.148:0/150250794 wait complete. 2024-09-16T04:22:44.947 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.054+0000 7f7b9cd59640 10 monclient: get_monmap_and_config 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.054+0000 7f7b9cd59640 10 monclient: build_initial_monmap 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.054+0000 7f7b9cd59640 10 monclient: monmap: 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:45.055692+0000 2024-09-16T04:22:45.056 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:45.055692+0000 2024-09-16T04:22:45.057 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:45.057 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:45.057 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:45.057 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:45.057 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:45.057 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:45.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.054+0000 7f7b9cd59640 1 Processor -- start 2024-09-16T04:22:45.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.054+0000 7f7b9cd59640 1 -- start start 2024-09-16T04:22:45.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.054+0000 7f7b9cd59640 10 monclient: init 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b980a1010 con 0x7f7b9809cc30 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9809f030 con 0x7f7b9809bb70 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9809dc60 con 0x7f7b9809c660 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b95d9b640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b98138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b9cd59640 10 monclient(hunting): authenticate will time out at 1785.681171s 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b96d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7b9809cc30 0x7f7b9813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:47214/0 (socket says 172.21.15.148:47214) 2024-09-16T04:22:45.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.055+0000 7f7b96d9d640 1 -- 172.21.15.148:0/4287371999 learned_addr learned my addr 172.21.15.148:0/4287371999 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:45.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.256+0000 7f7b95d9b640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b98138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:45.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:45.657+0000 7f7b95d9b640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b98138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:46.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:46.459+0000 7f7b95d9b640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b98138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): tick 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b98138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.171:6789/0 conn(0x7f7b9809bb70 legacy=0x7f7b9809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6789/0 conn(0x7f7b9809cc30 legacy=0x7f7b9813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:48.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9809df40 con 0x7f7b9809cc30 2024-09-16T04:22:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9809dc60 con 0x7f7b9809bb70 2024-09-16T04:22:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7800fe70 con 0x7f7b9809c660 2024-09-16T04:22:48.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.056+0000 7f7b87fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:48.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.257+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:48.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:48.658+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:49.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:49.459+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78005710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:51.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): tick 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78005710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.171:6789/0 conn(0x7f7b9809bb70 legacy=0x7f7b78008ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6789/0 conn(0x7f7b9809cc30 legacy=0x7f7b7800c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7b78010850 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b78011120 con 0x7f7b78010850 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7800fe70 con 0x7f7b9809bb70 2024-09-16T04:22:51.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b78010e70 con 0x7f7b9809c660 2024-09-16T04:22:51.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b87fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:51.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.056+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78010140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:51.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.257+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78010140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:51.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:51.658+0000 7f7b9659c640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78010140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:52.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925239747 0 0) 0x7f7b78011120 con 0x7f7b78010850 2024-09-16T04:22:52.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient(hunting): my global_id is 144168 2024-09-16T04:22:52.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b80005730 con 0x7f7b78010850 2024-09-16T04:22:52.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 470274382 0 0) 0x7f7b80005730 con 0x7f7b78010850 2024-09-16T04:22:52.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b78010140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:52.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.171:6789/0 conn(0x7f7b9809bb70 legacy=0x7f7b78008ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient: _finish_hunting 0 2024-09-16T04:22:52.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 monclient: found mon.noname-a 2024-09-16T04:22:52.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7800c6a0 con 0x7f7b78010850 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient: _finish_auth 0 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 10 monclient: _check_auth_tickets 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.429+0000 7f7b9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b8c004150 con 0x7f7b78010850 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9cd59640 5 monclient: authenticate success, global_id 144168 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: dump: 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.433 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9cd59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b8c0043b0 con 0x7f7b78010850 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 1 -- 172.21.15.148:0/4287371999 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b8c004800 con 0x7f7b78010850 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9cd59640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9cd59640 4 set_mon_vals no callback set 2024-09-16T04:22:52.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.430+0000 7f7b9559a640 10 monclient: dump: 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.435 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 10 monclient: shutdown 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 1 -- 172.21.15.148:0/4287371999 >> v1:172.21.15.148:6789/0 conn(0x7f7b78010850 legacy=0x7f7b78005710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 1 -- 172.21.15.148:0/4287371999 shutdown_connections 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 1 -- 172.21.15.148:0/4287371999 >> 172.21.15.148:0/4287371999 conn(0x7f7b980a5640 msgr2=0x7f7b980a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 1 -- 172.21.15.148:0/4287371999 shutdown_connections 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.431+0000 7f7b9cd59640 1 -- 172.21.15.148:0/4287371999 wait complete. 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.432+0000 7f7b9cd59640 10 monclient: build_initial_monmap 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.432+0000 7f7b9cd59640 10 monclient: monmap: 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:52.441 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 1 Processor -- start 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 1 -- start start 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 10 monclient: init 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 10 monclient(hunting): picked mon.noname-b con 0x7f7b78010850 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:52.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b980740a0 con 0x7f7b9809bb70 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.433+0000 7f7b9cd59640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98120cf0 con 0x7f7b78010850 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b9cd59640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98072230 con 0x7f7b9809c660 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b9cd59640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b9cd59640 10 monclient(hunting): authenticate will time out at 1793.059035s 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b96d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b981cf890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b95d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7b9809bb70 0x7f7b981cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41928/0 (socket says 172.21.15.148:41928) 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b95d9b640 1 -- 172.21.15.148:0/2974525612 learned_addr learned my addr 172.21.15.148:0/2974525612 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182044227 0 0) 0x7f7b98120cf0 con 0x7f7b78010850 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): my global_id is 7144 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b78010e70 con 0x7f7b78010850 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 37002512 0 0) 0x7f7b7800c6a0 con 0x7f7b9809bb70 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): my global_id is 144171 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b98120cf0 con 0x7f7b9809bb70 2024-09-16T04:22:52.444 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1863941400 0 0) 0x7f7b78010e70 con 0x7f7b78010850 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b7800c6a0 con 0x7f7b78010850 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b880018b0 con 0x7f7b78010850 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): dump: 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.445 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.434+0000 7f7b86ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3050317261 0 0) 0x7f7b7800c6a0 con 0x7f7b78010850 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 >> v1:172.21.15.148:6790/0 conn(0x7f7b9809c660 legacy=0x7f7b981cf890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 >> v1:172.21.15.148:6789/0 conn(0x7f7b9809bb70 legacy=0x7f7b981cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 monclient: found mon.b 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b98120fd0 con 0x7f7b78010850 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: _finish_auth 0 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:52.446 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 5 monclient: authenticate success, global_id 7144 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 10 monclient: _renew_subs 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9811f8a0 con 0x7f7b78010850 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 10 monclient: _renew_subs 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b981341e0 con 0x7f7b78010850 2024-09-16T04:22:52.447 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7b88003610 con 0x7f7b78010850 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7b88003960 con 0x7f7b78010850 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b94d99640 4 set_mon_vals no callback set 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 10 monclient: dump: 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.448 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.435+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7b88003b80 con 0x7f7b78010850 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.436+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f7b88047a20 con 0x7f7b78010850 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.436+0000 7f7b9cd59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.436+0000 7f7b9cd59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.436+0000 7f7b9cd59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.436+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b7c005080 con 0x7f7b78010850 2024-09-16T04:22:52.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.436+0000 7f7b95d9b640 10 monclient: get_auth_request con 0x7f7b780399f0 auth_method 0 2024-09-16T04:22:52.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.441+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f7b88015bc0 con 0x7f7b78010850 2024-09-16T04:22:52.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.441+0000 7f7b86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.441+0000 7f7b86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b9cd59640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:22:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b9cd59640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b9cd59640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:22:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7b7c005370 con 0x7f7b78010850 2024-09-16T04:22:52.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b86ffd640 1 -- 172.21.15.148:0/2974525612 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2054664614) 0x7f7b88019870 con 0x7f7b78010850 2024-09-16T04:22:52.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:22:52.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.674+0000 7f7b86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:52.677 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":68,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 >> v1:172.21.15.171:6812/7327273 conn(0x7f7b780399f0 legacy=0x7f7b7803beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 10 monclient: shutdown 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 >> v1:172.21.15.171:6789/0 conn(0x7f7b78010850 legacy=0x7f7b981338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 shutdown_connections 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 >> 172.21.15.148:0/2974525612 conn(0x7f7b980a5640 msgr2=0x7f7b981b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 shutdown_connections 2024-09-16T04:22:52.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.676+0000 7f7b9cd59640 1 -- 172.21.15.148:0/2974525612 wait complete. 2024-09-16T04:22:52.689 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:22:52.689 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:22:52.689 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:22:52.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.786+0000 7f102c568640 10 monclient: get_monmap_and_config 2024-09-16T04:22:52.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.786+0000 7f102c568640 10 monclient: build_initial_monmap 2024-09-16T04:22:52.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.786+0000 7f102c568640 10 monclient: monmap: 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:52.786983+0000 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:52.786983+0000 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:52.788 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.786+0000 7f102c568640 1 Processor -- start 2024-09-16T04:22:52.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.786+0000 7f102c568640 1 -- start start 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.786+0000 7f102c568640 10 monclient: init 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient(hunting): picked mon.noname-a con 0x7f10240a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient(hunting): picked mon.noname-b con 0x7f10240a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient(hunting): picked mon.noname-c con 0x7f102413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10240a0e00 con 0x7f10240a2410 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10240a10e0 con 0x7f102413d060 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10240a4970 con 0x7f10240a1b10 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 10 monclient(hunting): authenticate will time out at 1793.412332s 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029b03640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f10240a1b10 legacy=0x7f102413b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:52.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102a304640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f10240a2410 0x7f10240a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41944/0 (socket says 172.21.15.148:41944) 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102a304640 1 -- 172.21.15.148:0/2955372646 learned_addr learned my addr 172.21.15.148:0/2955372646 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663480052 0 0) 0x7f10240a0e00 con 0x7f10240a2410 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): my global_id is 144174 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1010003520 con 0x7f10240a2410 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757890612 0 0) 0x7f10240a10e0 con 0x7f102413d060 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient(hunting): my global_id is 7147 2024-09-16T04:22:52.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10240a0e00 con 0x7f102413d060 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2471509430 0 0) 0x7f1010003520 con 0x7f10240a2410 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 >> v1:172.21.15.148:6790/0 conn(0x7f10240a1b10 legacy=0x7f102413b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 >> v1:172.21.15.171:6789/0 conn(0x7f102413d060 legacy=0x7f102413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: _finish_hunting 0 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 monclient: found mon.noname-a 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102406dfd0 con 0x7f10240a2410 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: _finish_auth 0 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: _check_auth_tickets 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:52.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1018001990 con 0x7f10240a2410 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f1029302640 10 monclient: dump: 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.801 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.803 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 5 monclient: authenticate success, global_id 144174 2024-09-16T04:22:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.787+0000 7f102c568640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1018001cb0 con 0x7f10240a2410 2024-09-16T04:22:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f1029302640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:52.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f1029302640 1 -- 172.21.15.148:0/2955372646 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1018002080 con 0x7f10240a2410 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f1029302640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 4 set_mon_vals no callback set 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f1029302640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f1029302640 10 monclient: dump: 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.804 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 10 monclient: shutdown 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 1 -- 172.21.15.148:0/2955372646 >> v1:172.21.15.148:6789/0 conn(0x7f10240a2410 legacy=0x7f10240a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 1 -- 172.21.15.148:0/2955372646 shutdown_connections 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 1 -- 172.21.15.148:0/2955372646 >> 172.21.15.148:0/2955372646 conn(0x7f10240a97b0 msgr2=0x7f10240a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:52.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 1 -- 172.21.15.148:0/2955372646 shutdown_connections 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.788+0000 7f102c568640 1 -- 172.21.15.148:0/2955372646 wait complete. 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient: build_initial_monmap 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient: monmap: 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:52.806 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 1 Processor -- start 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 1 -- start start 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient: init 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient(hunting): picked mon.noname-b con 0x7f10240a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient(hunting): picked mon.noname-c con 0x7f10240a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient(hunting): picked mon.noname-a con 0x7f102413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:52.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10241378c0 con 0x7f102413d060 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f102413cc90 con 0x7f10240a1b10 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f1029b03640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7f10240a2410 legacy=0x7f10241d0fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f102409ec20 con 0x7f10240a2410 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.789+0000 7f102c568640 10 monclient(hunting): authenticate will time out at 1793.414988s 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102ab05640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f102413d060 0x7f10241d46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41958/0 (socket says 172.21.15.148:41958) 2024-09-16T04:22:52.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102ab05640 1 -- 172.21.15.148:0/210411796 learned_addr learned my addr 172.21.15.148:0/210411796 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688306562 0 0) 0x7f10240a0e00 con 0x7f102413d060 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): my global_id is 144177 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1008003520 con 0x7f102413d060 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 149380348 0 0) 0x7f102413cc90 con 0x7f10240a1b10 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): my global_id is 7150 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10240a0e00 con 0x7f10240a1b10 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2509170441 0 0) 0x7f1008003520 con 0x7f102413d060 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f102413cc90 con 0x7f102413d060 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f101c0018b0 con 0x7f102413d060 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): dump: 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.810 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:52.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 438740045 0 0) 0x7f10240a0e00 con 0x7f10240a1b10 2024-09-16T04:22:52.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1008003520 con 0x7f10240a1b10 2024-09-16T04:22:52.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f10180017b0 con 0x7f10240a1b10 2024-09-16T04:22:52.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:52.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:52.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient(hunting): dump: 2024-09-16T04:22:52.812 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.813 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.813 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.813 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.813 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.813 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1436936119 0 0) 0x7f102413cc90 con 0x7f102413d060 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.148:6790/0 conn(0x7f10240a2410 legacy=0x7f10241d0fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.171:6789/0 conn(0x7f10240a1b10 legacy=0x7f10241328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 monclient: found mon.a 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102409f780 con 0x7f102413d060 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: _finish_auth 0 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 5 monclient: authenticate success, global_id 144177 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 10 monclient: _renew_subs 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f102413c4e0 con 0x7f102413d060 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 10 monclient: _renew_subs 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f102409e1f0 con 0x7f102413d060 2024-09-16T04:22:52.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f101c001c60 con 0x7f102413d060 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f101c003a60 con 0x7f102413d060 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1028b01640 4 set_mon_vals no callback set 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.790+0000 7f1016ffd640 10 monclient: dump: 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.816 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f102c568640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f100c0065c0 con 0x7f100c002860 2024-09-16T04:22:52.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f101c003c80 con 0x7f102413d060 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f101c047a00 con 0x7f102413d060 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803664366 0 0) 0x7f100c0065c0 con 0x7f100c002860 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 10 monclient(hunting): my global_id is 144180 2024-09-16T04:22:52.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f102413cc90 con 0x7f100c002860 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.791+0000 7f1029b03640 10 monclient: get_auth_request con 0x7f1008038ec0 auth_method 0 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3981208357 0 0) 0x7f102413cc90 con 0x7f100c002860 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f100c0065c0 con 0x7f100c002860 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1018001a10 con 0x7f100c002860 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 10 monclient: dump: 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.819 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 86343342 0 0) 0x7f100c0065c0 con 0x7f100c002860 2024-09-16T04:22:52.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f100c00a150 con 0x7f100c002860 2024-09-16T04:22:52.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f10180070a0 con 0x7f100c002860 2024-09-16T04:22:52.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:52.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.148:6789/0 conn(0x7f100c002860 legacy=0x7f100c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.792+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.148:6789/0 conn(0x7f100c002860 legacy=0x7f100c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f102c568640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f100c009e40 con 0x7f100c009100 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680326449 0 0) 0x7f100c009e40 con 0x7f100c009100 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient(hunting): my global_id is 144183 2024-09-16T04:22:52.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f100c0065c0 con 0x7f100c009100 2024-09-16T04:22:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3273709761 0 0) 0x7f100c0065c0 con 0x7f100c009100 2024-09-16T04:22:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f100c009e40 con 0x7f100c009100 2024-09-16T04:22:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1018005590 con 0x7f100c009100 2024-09-16T04:22:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.798+0000 7f1016ffd640 10 monclient: dump: 2024-09-16T04:22:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.826 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.827 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.827 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.827 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.827 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.827 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 588769444 0 0) 0x7f100c009e40 con 0x7f100c009100 2024-09-16T04:22:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f100c00c3d0 con 0x7f100c009100 2024-09-16T04:22:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3967 (unknown 0 0 492239623) 0x7f1018006890 con 0x7f100c009100 2024-09-16T04:22:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:52.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.148:6789/0 conn(0x7f100c009100 legacy=0x7f100c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f1016ffd640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.148:6789/0 conn(0x7f100c009100 legacy=0x7f100c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:52.829 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:52.829 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:52.830 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:52.830 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:22:52.830 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 68, 2024-09-16T04:22:52.830 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 17499, 2024-09-16T04:22:52.830 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:52.831 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:52.831 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:22:52.831 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.831 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:22:52.831 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:52.831 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:52.832 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:52.833 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:52.834 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:52.835 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:52.836 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:52.837 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:52.838 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:52.839 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.840 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.841 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.842 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:52.842 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:52.842 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:52.842 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:52.842 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:52.843 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.844 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.844 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:52.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.844 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:52.844 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:52.845 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:22:52.845 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.845 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.845 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.845 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:52.845 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.171:6812/7327273 conn(0x7f1008038ec0 legacy=0x7f100803b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 10 monclient: shutdown 2024-09-16T04:22:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:52.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 >> v1:172.21.15.148:6789/0 conn(0x7f102413d060 legacy=0x7f10241d46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102ab05640 1 -- 172.21.15.148:0/210411796 reap_dead start 2024-09-16T04:22:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102ab05640 1 -- 172.21.15.148:0/210411796 reap_dead start 2024-09-16T04:22:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 shutdown_connections 2024-09-16T04:22:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 >> 172.21.15.148:0/210411796 conn(0x7f10240a97b0 msgr2=0x7f10241359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 shutdown_connections 2024-09-16T04:22:52.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.799+0000 7f102c568640 1 -- 172.21.15.148:0/210411796 wait complete. 2024-09-16T04:22:52.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:22:52.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:22:52.849 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:22:52.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.946+0000 7fd9d918d640 10 monclient: get_monmap_and_config 2024-09-16T04:22:52.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.946+0000 7fd9d918d640 10 monclient: build_initial_monmap 2024-09-16T04:22:52.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.946+0000 7fd9d918d640 10 monclient: monmap: 2024-09-16T04:22:52.948 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:52.948 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.948 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:22:52.947573+0000 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:22:52.947573+0000 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:22:52.949 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 1 Processor -- start 2024-09-16T04:22:52.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 1 -- start start 2024-09-16T04:22:52.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient: init 2024-09-16T04:22:52.958 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d409bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d409c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d40a1010 con 0x7fd9d409bb70 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d409f030 con 0x7fd9d409c660 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d409dc60 con 0x7fd9d409cc30 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d918d640 10 monclient(hunting): authenticate will time out at 1793.572972s 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.947+0000 7fd9d359e640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fd9d409cc30 legacy=0x7fd9d413c090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d2d9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd9d409bb70 0x7fd9d409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:41994/0 (socket says 172.21.15.148:41994) 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d2d9d640 1 -- 172.21.15.148:0/729539495 learned_addr learned my addr 172.21.15.148:0/729539495 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:52.959 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827948147 0 0) 0x7fd9d40a1010 con 0x7fd9d409bb70 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): my global_id is 144186 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c0003520 con 0x7fd9d409bb70 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005520812 0 0) 0x7fd9d409f030 con 0x7fd9d409c660 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient(hunting): my global_id is 7153 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d40a1010 con 0x7fd9d409c660 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1688260646 0 0) 0x7fd9c0003520 con 0x7fd9d409bb70 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 >> v1:172.21.15.148:6790/0 conn(0x7fd9d409cc30 legacy=0x7fd9d413c090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 >> v1:172.21.15.171:6789/0 conn(0x7fd9d409c660 legacy=0x7fd9d4138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: _finish_hunting 0 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 monclient: found mon.noname-a 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:52.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d409df40 con 0x7fd9d409bb70 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: _finish_auth 0 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: _check_auth_tickets 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9c4001610 con 0x7fd9d409bb70 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d918d640 5 monclient: authenticate success, global_id 144186 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d918d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: dump: 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.962 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d918d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9c40018c0 con 0x7fd9d409bb70 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 1 -- 172.21.15.148:0/729539495 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9c4001c90 con 0x7fd9d409bb70 2024-09-16T04:22:52.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d259c640 10 monclient: dump: 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d918d640 10 monclient: get_monmap_and_config success 2024-09-16T04:22:52.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.948+0000 7fd9d918d640 4 set_mon_vals no callback set 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 10 monclient: shutdown 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 1 -- 172.21.15.148:0/729539495 >> v1:172.21.15.148:6789/0 conn(0x7fd9d409bb70 legacy=0x7fd9d409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 1 -- 172.21.15.148:0/729539495 shutdown_connections 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 1 -- 172.21.15.148:0/729539495 >> 172.21.15.148:0/729539495 conn(0x7fd9d40a5640 msgr2=0x7fd9d40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 1 -- 172.21.15.148:0/729539495 shutdown_connections 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 1 -- 172.21.15.148:0/729539495 wait complete. 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 10 monclient: build_initial_monmap 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.949+0000 7fd9d918d640 10 monclient: monmap: 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:22:52.965 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 1 Processor -- start 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 1 -- start start 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient: init 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient: _reopen_session rank -1 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:22:52.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d413bfa0 con 0x7fd9d409c660 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d411cf10 con 0x7fd9d409cc30 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d2d9d640 1 -- >> v1:172.21.15.148:6790/0 conn(0x7fd9d409bb70 legacy=0x7fd9d413a610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6790/0 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d411caf0 con 0x7fd9d409bb70 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient(hunting): _renew_subs 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9d918d640 10 monclient(hunting): authenticate will time out at 1793.575670s 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cbfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd9d409c660 0x7fd9d413ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:42002/0 (socket says 172.21.15.148:42002) 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cbfff640 1 -- 172.21.15.148:0/1277757766 learned_addr learned my addr 172.21.15.148:0/1277757766 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626150141 0 0) 0x7fd9d413bfa0 con 0x7fd9d409c660 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cb7fe640 10 monclient(hunting): my global_id is 144189 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.950+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b4003520 con 0x7fd9d409c660 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834429509 0 0) 0x7fd9cc003d30 con 0x7fd9d409cc30 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): my global_id is 7156 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d413bfa0 con 0x7fd9d409cc30 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1970311941 0 0) 0x7fd9b4003520 con 0x7fd9d409c660 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9cc003d30 con 0x7fd9d409c660 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9bc001910 con 0x7fd9d409c660 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:52.968 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): dump: 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.969 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2878006242 0 0) 0x7fd9d413bfa0 con 0x7fd9d409cc30 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9b4003520 con 0x7fd9d409cc30 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9cc004500 con 0x7fd9d409cc30 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient(hunting): dump: 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.970 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3821372311 0 0) 0x7fd9cc003d30 con 0x7fd9d409c660 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.148:6790/0 conn(0x7fd9d409bb70 legacy=0x7fd9d413a610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.171:6789/0 conn(0x7fd9d409cc30 legacy=0x7fd9d41d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 monclient: found mon.a 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d40a4030 con 0x7fd9d409c660 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:22:52.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 5 monclient: authenticate success, global_id 144189 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 10 monclient: _renew_subs 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d413ca50 con 0x7fd9d409c660 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 10 monclient: _renew_subs 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9d4134460 con 0x7fd9d409c660 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9bc003610 con 0x7fd9d409c660 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9bc003960 con 0x7fd9d409c660 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:22:52.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9cb7fe640 10 monclient: dump: 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.951+0000 7fd9d1d9b640 4 set_mon_vals no callback set 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd9bc003b80 con 0x7fd9d409c660 2024-09-16T04:22:52.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fd9bc047a80 con 0x7fd9d409c660 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9d918d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd9d40008d0 con 0x7fd9d413b430 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9d2d9d640 10 monclient: get_auth_request con 0x7fd9b4038ef0 auth_method 0 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089353894 0 0) 0x7fd9d40008d0 con 0x7fd9d413b430 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 10 monclient(hunting): my global_id is 7159 2024-09-16T04:22:52.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.952+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9cc003d30 con 0x7fd9d413b430 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4144131769 0 0) 0x7fd9cc003d30 con 0x7fd9d413b430 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9d40008d0 con 0x7fd9d413b430 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9cc004b30 con 0x7fd9d413b430 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 10 monclient: dump: 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.975 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4138491325 0 0) 0x7fd9d40008d0 con 0x7fd9d413b430 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9d4068890 con 0x7fd9d413b430 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd9cc004de0 con 0x7fd9d413b430 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.171:6789/0 conn(0x7fd9d413b430 legacy=0x7fd9d4068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.953+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.171:6789/0 conn(0x7fd9d413b430 legacy=0x7fd9d4068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:52.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.959+0000 7fd9d918d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:22:52.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.959+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd9d4137070 con 0x7fd9d402d7c0 2024-09-16T04:22:52.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932876335 0 0) 0x7fd9d4137070 con 0x7fd9d402d7c0 2024-09-16T04:22:52.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:22:52.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient(hunting): my global_id is 7162 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9d40008d0 con 0x7fd9d402d7c0 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3200141821 0 0) 0x7fd9d40008d0 con 0x7fd9d402d7c0 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9d4137070 con 0x7fd9d402d7c0 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9cc005530 con 0x7fd9d402d7c0 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 10 monclient: dump: 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:22:52.982 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 989607247 0 0) 0x7fd9d4137070 con 0x7fd9d402d7c0 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.960+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd9d402dba0 con 0x7fd9d402d7c0 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4300 (unknown 0 0 4254084917) 0x7fd9cc005760 con 0x7fd9d402d7c0 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9cb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.171:6789/0 conn(0x7fd9d402d7c0 legacy=0x7fd9d4136960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9cb7fe640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.171:6789/0 conn(0x7fd9d402d7c0 legacy=0x7fd9d4136960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 68, 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 996999, 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: 1 2024-09-16T04:22:52.984 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:52.985 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:52.986 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:22:52.987 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:22:52.988 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:22:52.989 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.990 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:52.991 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.992 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:22:52.993 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:22:52.994 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.995 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:22:52.996 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:22:52.997 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:22:52.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.171:6812/7327273 conn(0x7fd9b4038ef0 legacy=0x7fd9b403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 10 monclient: shutdown 2024-09-16T04:22:52.998 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 >> v1:172.21.15.148:6789/0 conn(0x7fd9d409c660 legacy=0x7fd9d413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d359e640 1 -- 172.21.15.148:0/1277757766 reap_dead start 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d359e640 1 -- 172.21.15.148:0/1277757766 reap_dead start 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 shutdown_connections 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 >> 172.21.15.148:0/1277757766 conn(0x7fd9d40a5640 msgr2=0x7fd9d413cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 shutdown_connections 2024-09-16T04:22:52.999 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:22:52.961+0000 7fd9d918d640 1 -- 172.21.15.148:0/1277757766 wait complete. 2024-09-16T04:22:52.999 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:22:52.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:24:23.000 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:24:23.001 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:24:23.001 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:24:23.001 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:24:23.005 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:24:23.005 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:24:23.005 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:24:23.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.114+0000 7f2b9a48d640 10 monclient: get_monmap_and_config 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.114+0000 7f2b9a48d640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.114+0000 7f2b9a48d640 10 monclient: monmap: 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:23.115183+0000 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:23.115183+0000 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:23.116 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.124 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 1 Processor -- start 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 1 -- start start 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient: init 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b940a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b940a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b9413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b940a0e00 con 0x7f2b940a1b10 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b940a10e0 con 0x7f2b9413d060 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b940a4970 con 0x7f2b940a2410 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.125 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b9a48d640 10 monclient(hunting): authenticate will time out at 1883.740731s 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b93fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2b940a2410 0x7f2b940a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37448/0 (socket says 172.21.15.148:37448) 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.115+0000 7f2b93fff640 1 -- 172.21.15.148:0/4052734706 learned_addr learned my addr 172.21.15.148:0/4052734706 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3106339978 0 0) 0x7f2b940a0e00 con 0x7f2b940a1b10 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient(hunting): my global_id is 144192 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b7c003520 con 0x7f2b940a1b10 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1440791837 0 0) 0x7f2b7c003520 con 0x7f2b940a1b10 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 >> v1:172.21.15.148:6790/0 conn(0x7f2b940a2410 legacy=0x7f2b940a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 >> v1:172.21.15.171:6789/0 conn(0x7f2b9413d060 legacy=0x7f2b9413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.126 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 monclient: found mon.noname-a 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9406dfd0 con 0x7f2b940a1b10 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b88001610 con 0x7f2b940a1b10 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b9a48d640 5 monclient: authenticate success, global_id 144192 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b9a48d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: dump: 2024-09-16T04:24:23.127 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b9a48d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b88001970 con 0x7f2b940a1b10 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 1 -- 172.21.15.148:0/4052734706 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b88001d40 con 0x7f2b940a1b10 2024-09-16T04:24:23.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b9a48d640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b9a48d640 4 set_mon_vals no callback set 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.116+0000 7f2b92ffd640 10 monclient: dump: 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.129 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 10 monclient: shutdown 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 1 -- 172.21.15.148:0/4052734706 >> v1:172.21.15.148:6789/0 conn(0x7f2b940a1b10 legacy=0x7f2b9413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 1 -- 172.21.15.148:0/4052734706 shutdown_connections 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 1 -- 172.21.15.148:0/4052734706 >> 172.21.15.148:0/4052734706 conn(0x7f2b940a97b0 msgr2=0x7f2b940a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 1 -- 172.21.15.148:0/4052734706 shutdown_connections 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 1 -- 172.21.15.148:0/4052734706 wait complete. 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.117+0000 7f2b9a48d640 10 monclient: monmap: 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:23.130 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 1 Processor -- start 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 1 -- start start 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient: init 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient(hunting): picked mon.noname-a con 0x7f2b940a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient(hunting): picked mon.noname-c con 0x7f2b940a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient(hunting): picked mon.noname-b con 0x7f2b9413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b9413cd20 con 0x7f2b940a1b10 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b940a0130 con 0x7f2b9413d060 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b9409f510 con 0x7f2b940a2410 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.118+0000 7f2b9a48d640 10 monclient(hunting): authenticate will time out at 1883.743990s 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b937fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2b940a2410 0x7f2b941d0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37460/0 (socket says 172.21.15.148:37460) 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b937fe640 1 -- 172.21.15.148:0/67231912 learned_addr learned my addr 172.21.15.148:0/67231912 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053262848 0 0) 0x7f2b940a4970 con 0x7f2b940a1b10 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): my global_id is 144195 2024-09-16T04:24:23.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b80003520 con 0x7f2b940a1b10 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404433337 0 0) 0x7f2b940a0130 con 0x7f2b9413d060 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): my global_id is 7168 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b940a4970 con 0x7f2b9413d060 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 198831467 0 0) 0x7f2b80003520 con 0x7f2b940a1b10 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b940a0130 con 0x7f2b940a1b10 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b840017e0 con 0x7f2b940a1b10 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): dump: 2024-09-16T04:24:23.133 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2173554387 0 0) 0x7f2b940a4970 con 0x7f2b9413d060 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b80003520 con 0x7f2b9413d060 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b8c001b70 con 0x7f2b9413d060 2024-09-16T04:24:23.134 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient(hunting): dump: 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.135 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 40771125 0 0) 0x7f2b940a0130 con 0x7f2b940a1b10 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 >> v1:172.21.15.148:6790/0 conn(0x7f2b940a2410 legacy=0x7f2b941d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 >> v1:172.21.15.171:6789/0 conn(0x7f2b9413d060 legacy=0x7f2b941d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 monclient: found mon.a 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9409e340 con 0x7f2b940a1b10 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 5 monclient: authenticate success, global_id 144195 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 10 monclient: _renew_subs 2024-09-16T04:24:23.136 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9409f7f0 con 0x7f2b940a1b10 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 10 monclient: _renew_subs 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.119+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b9409ce00 con 0x7f2b940a1b10 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.120+0000 7f2b9a48d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.120+0000 7f2b9a48d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.120+0000 7f2b9a48d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.120+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b94136ad0 con 0x7f2b940a1b10 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.121+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b840047e0 con 0x7f2b940a1b10 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.121+0000 7f2b90ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.122+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2b84004b30 con 0x7f2b940a1b10 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.122+0000 7f2b90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.137 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.122+0000 7f2b90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.122+0000 7f2b90ff9640 10 monclient: dump: 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.122+0000 7f2b927fc640 4 set_mon_vals no callback set 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.122+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2b84004d50 con 0x7f2b940a1b10 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.123+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f2b84047930 con 0x7f2b940a1b10 2024-09-16T04:24:23.138 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.123+0000 7f2b937fe640 10 monclient: get_auth_request con 0x7f2b80038ef0 auth_method 0 2024-09-16T04:24:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.127+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2b9413cd20 con 0x7f2b940a1b10 2024-09-16T04:24:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.127+0000 7f2b90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.139 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.127+0000 7f2b90ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b9a48d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b9a48d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b9a48d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2b94062630 con 0x7f2b940a1b10 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b90ff9640 1 -- 172.21.15.148:0/67231912 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 1814380985) 0x7f2b84015b90 con 0x7f2b940a1b10 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:23.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.363+0000 7f2b90ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:23.366 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:24:23.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 >> v1:172.21.15.171:6812/7327273 conn(0x7f2b80038ef0 legacy=0x7f2b8003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 10 monclient: shutdown 2024-09-16T04:24:23.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 >> v1:172.21.15.148:6789/0 conn(0x7f2b940a1b10 legacy=0x7f2b941cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 shutdown_connections 2024-09-16T04:24:23.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 >> 172.21.15.148:0/67231912 conn(0x7f2b940a97b0 msgr2=0x7f2b94135c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 shutdown_connections 2024-09-16T04:24:23.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.365+0000 7f2b9a48d640 1 -- 172.21.15.148:0/67231912 wait complete. 2024-09-16T04:24:23.378 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:24:23.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:24:23.378 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:24:23.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.475+0000 7fe0c77c0640 10 monclient: get_monmap_and_config 2024-09-16T04:24:23.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.475+0000 7fe0c77c0640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.475+0000 7fe0c77c0640 10 monclient: monmap: 2024-09-16T04:24:23.477 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.477 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.477 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:23.476798+0000 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:23.476798+0000 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:23.478 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.476+0000 7fe0c77c0640 1 Processor -- start 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.476+0000 7fe0c77c0640 1 -- start start 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.476+0000 7fe0c77c0640 10 monclient: init 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c00a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c009b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c009b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c009e040 con 0x7fe0c009b170 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c009d660 con 0x7fe0c009b5c0 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c009c0d0 con 0x7fe0c00a0e90 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c77c0640 10 monclient(hunting): authenticate will time out at 1884.102163s 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c555c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe0c00a0e90 0x7fe0c009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37472/0 (socket says 172.21.15.148:37472) 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c555c640 1 -- 172.21.15.148:0/1995059455 learned_addr learned my addr 172.21.15.148:0/1995059455 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0c4d5b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe0c009b170 0x7fe0c01383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46540/0 (socket says 172.21.15.148:46540) 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098606076 0 0) 0x7fe0b0001110 con 0x7fe0c009b170 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.488 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): my global_id is 144198 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0ac003440 con 0x7fe0c009b170 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345187613 0 0) 0x7fe0b8001110 con 0x7fe0c009b5c0 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient(hunting): my global_id is 7171 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0b0001110 con 0x7fe0c009b5c0 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2143606809 0 0) 0x7fe0b0001110 con 0x7fe0c009b5c0 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 >> v1:172.21.15.148:6790/0 conn(0x7fe0c00a0e90 legacy=0x7fe0c009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 >> v1:172.21.15.148:6789/0 conn(0x7fe0c009b170 legacy=0x7fe0c01383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 monclient: found mon.noname-c 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.477+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c009d380 con 0x7fe0c009b5c0 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b80017c0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: dump: 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.490 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 5 monclient: authenticate success, global_id 7171 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0b8001ab0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 4 set_mon_vals no callback set 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 1 -- 172.21.15.148:0/1995059455 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b80035f0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0b7fff640 10 monclient: dump: 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.492 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 10 monclient: shutdown 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1995059455 >> v1:172.21.15.171:6789/0 conn(0x7fe0c009b5c0 legacy=0x7fe0c013bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1995059455 shutdown_connections 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.478+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1995059455 >> 172.21.15.148:0/1995059455 conn(0x7fe0c00a5640 msgr2=0x7fe0c00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.479+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1995059455 shutdown_connections 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.479+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1995059455 wait complete. 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.479+0000 7fe0c77c0640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.479+0000 7fe0c77c0640 10 monclient: monmap: 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:23.494 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.479+0000 7fe0c77c0640 1 Processor -- start 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 1 -- start start 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient: init 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c009b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c009b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c00a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c01c6060 con 0x7fe0c009b170 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c01aa7f0 con 0x7fe0c00a0e90 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c01ad2b0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c77c0640 10 monclient(hunting): authenticate will time out at 1884.105591s 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c555c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe0c009b170 0x7fe0c01c5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46552/0 (socket says 172.21.15.148:46552) 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0c555c640 1 -- 172.21.15.148:0/1583702997 learned_addr learned my addr 172.21.15.148:0/1583702997 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195275298 0 0) 0x7fe0c01c6060 con 0x7fe0c009b170 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 144201 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a4003520 con 0x7fe0c009b170 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727508278 0 0) 0x7fe0c01ad2b0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 154100 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c01c6060 con 0x7fe0c009b5c0 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955627066 0 0) 0x7fe0c01aa7f0 con 0x7fe0c00a0e90 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 7174 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.480+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c01ad2b0 con 0x7fe0c00a0e90 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2166577922 0 0) 0x7fe0c01c6060 con 0x7fe0c009b5c0 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0c01aa7f0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3186358736 0 0) 0x7fe0a4003520 con 0x7fe0c009b170 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0c01c6060 con 0x7fe0c009b170 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b0001aa0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): dump: 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.498 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0bc001b20 con 0x7fe0c009b170 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): dump: 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.499 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3457186810 0 0) 0x7fe0c01ad2b0 con 0x7fe0c00a0e90 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0a4003520 con 0x7fe0c00a0e90 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b8001c60 con 0x7fe0c00a0e90 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient(hunting): dump: 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1200424980 0 0) 0x7fe0c01aa7f0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.171:6789/0 conn(0x7fe0c00a0e90 legacy=0x7fe0c01d43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.148:6789/0 conn(0x7fe0c009b170 legacy=0x7fe0c01c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 monclient: found mon.c 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c01aafe0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 5 monclient: authenticate success, global_id 154100 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 10 monclient: _renew_subs 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c01acfd0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe0b0001c80 con 0x7fe0c009b5c0 2024-09-16T04:24:23.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b0004a90 con 0x7fe0c009b5c0 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b5ffb640 10 monclient: dump: 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 10 monclient: _renew_subs 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0b77fe640 4 set_mon_vals no callback set 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.481+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe0c01c83c0 con 0x7fe0c009b5c0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe0b000fd20 con 0x7fe0c009b5c0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fe0b0047800 con 0x7fe0c009b5c0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0c77c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0c009bc80 con 0x7fe0c002d7c0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0c555c640 10 monclient: get_auth_request con 0x7fe0a4039180 auth_method 0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3517625746 0 0) 0x7fe0c009bc80 con 0x7fe0c002d7c0 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 144204 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c01aa7f0 con 0x7fe0c002d7c0 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2849552759 0 0) 0x7fe0c01aa7f0 con 0x7fe0c002d7c0 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0c009bc80 con 0x7fe0c002d7c0 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b8003900 con 0x7fe0c002d7c0 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.482+0000 7fe0b5ffb640 10 monclient: dump: 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.505 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 698065147 0 0) 0x7fe0c009bc80 con 0x7fe0c002d7c0 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0c00a44c0 con 0x7fe0c002d7c0 2024-09-16T04:24:23.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe0b8003aa0 con 0x7fe0c002d7c0 2024-09-16T04:24:23.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:23.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.148:6789/0 conn(0x7fe0c002d7c0 legacy=0x7fe0c0068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.483+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.148:6789/0 conn(0x7fe0c002d7c0 legacy=0x7fe0c0068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0c77c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:23.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe0c00740a0 con 0x7fe0c01a0f60 2024-09-16T04:24:23.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 523652627 0 0) 0x7fe0c00740a0 con 0x7fe0c01a0f60 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient(hunting): my global_id is 144207 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0c009bc80 con 0x7fe0c01a0f60 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 192698299 0 0) 0x7fe0c009bc80 con 0x7fe0c01a0f60 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0c00740a0 con 0x7fe0c01a0f60 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe0b8004060 con 0x7fe0c01a0f60 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 10 monclient: dump: 2024-09-16T04:24:23.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1880714354 0 0) 0x7fe0c00740a0 con 0x7fe0c01a0f60 2024-09-16T04:24:23.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.489+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe0c002dba0 con 0x7fe0c01a0f60 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3979 (unknown 0 0 3400851068) 0x7fe0b80042a0 con 0x7fe0c01a0f60 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0b5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.148:6789/0 conn(0x7fe0c01a0f60 legacy=0x7fe0c01d9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0b5ffb640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.148:6789/0 conn(0x7fe0c01a0f60 legacy=0x7fe0c01d9af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:23.511 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 70, 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 108189, 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.512 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.513 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.514 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.515 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:23.516 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.517 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.518 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:23.519 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.520 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.521 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.522 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:23.523 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:23.524 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:23.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.171:6812/7327273 conn(0x7fe0a4039180 legacy=0x7fe0a403b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 10 monclient: shutdown 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 >> v1:172.21.15.148:6790/0 conn(0x7fe0c009b5c0 legacy=0x7fe0c01d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c5d5d640 1 -- 172.21.15.148:0/1583702997 reap_dead start 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c5d5d640 1 -- 172.21.15.148:0/1583702997 reap_dead start 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 shutdown_connections 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 >> 172.21.15.148:0/1583702997 conn(0x7fe0c00a5640 msgr2=0x7fe0c00715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 shutdown_connections 2024-09-16T04:24:23.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.490+0000 7fe0c77c0640 1 -- 172.21.15.148:0/1583702997 wait complete. 2024-09-16T04:24:23.526 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:24:23.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:24:23.527 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:24:23.625 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.624+0000 7fc2f75d6640 10 monclient: get_monmap_and_config 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.624+0000 7fc2f75d6640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.624+0000 7fc2f75d6640 10 monclient: monmap: 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:23.625022+0000 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:23.625022+0000 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:23.626 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:23.627 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.624+0000 7fc2f75d6640 1 Processor -- start 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.624+0000 7fc2f75d6640 1 -- start start 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.624+0000 7fc2f75d6640 10 monclient: init 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f00a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f00a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00a0e00 con 0x7fc2f013d060 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00a10e0 con 0x7fc2f00a1b10 2024-09-16T04:24:23.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00a4970 con 0x7fc2f00a2410 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f75d6640 10 monclient(hunting): authenticate will time out at 1884.250425s 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f5372640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc2f00a2410 0x7fc2f00a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37494/0 (socket says 172.21.15.148:37494) 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f5372640 1 -- 172.21.15.148:0/4179308131 learned_addr learned my addr 172.21.15.148:0/4179308131 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2f5b73640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc2f013d060 0x7fc2f013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46576/0 (socket says 172.21.15.148:46576) 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3829265718 0 0) 0x7fc2f00a4970 con 0x7fc2f00a2410 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): my global_id is 154103 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d8003520 con 0x7fc2f00a2410 2024-09-16T04:24:23.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442453404 0 0) 0x7fc2f00a0e00 con 0x7fc2f013d060 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 10 monclient(hunting): my global_id is 144210 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f00a4970 con 0x7fc2f013d060 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 549309486 0 0) 0x7fc2d8003520 con 0x7fc2f00a2410 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.625+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 >> v1:172.21.15.171:6789/0 conn(0x7fc2f00a1b10 legacy=0x7fc2f013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 >> v1:172.21.15.148:6789/0 conn(0x7fc2f013d060 legacy=0x7fc2f013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 1 monclient: found mon.noname-b 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f006dfd0 con 0x7fc2f00a2410 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e0001990 con 0x7fc2f00a2410 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 5 monclient: authenticate success, global_id 154103 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: dump: 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.639 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2e0001c70 con 0x7fc2f00a2410 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 1 -- 172.21.15.148:0/4179308131 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e0002080 con 0x7fc2f00a2410 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 4 set_mon_vals no callback set 2024-09-16T04:24:23.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2effff640 10 monclient: dump: 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 10 monclient: shutdown 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 1 -- 172.21.15.148:0/4179308131 >> v1:172.21.15.148:6790/0 conn(0x7fc2f00a2410 legacy=0x7fc2f00a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 1 -- 172.21.15.148:0/4179308131 shutdown_connections 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 1 -- 172.21.15.148:0/4179308131 >> 172.21.15.148:0/4179308131 conn(0x7fc2f00a97b0 msgr2=0x7fc2f00a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 1 -- 172.21.15.148:0/4179308131 shutdown_connections 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.626+0000 7fc2f75d6640 1 -- 172.21.15.148:0/4179308131 wait complete. 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.627+0000 7fc2f75d6640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.627+0000 7fc2f75d6640 10 monclient: monmap: 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:23.642 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.627+0000 7fc2f75d6640 1 Processor -- start 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.627+0000 7fc2f75d6640 1 -- start start 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.627+0000 7fc2f75d6640 10 monclient: init 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f00a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f00a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f013d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f0137a10 con 0x7fc2f00a2410 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f009d8e0 con 0x7fc2f00a1b10 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f0140340 con 0x7fc2f013d060 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient(hunting): authenticate will time out at 1884.253189s 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f5b73640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc2f013d060 0x7fc2f01d3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37510/0 (socket says 172.21.15.148:37510) 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f4b71640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc2f00a2410 0x7fc2f009d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46584/0 (socket says 172.21.15.148:46584) 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f5b73640 1 -- 172.21.15.148:0/1261302800 learned_addr learned my addr 172.21.15.148:0/1261302800 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169865567 0 0) 0x7fc2f0140340 con 0x7fc2f013d060 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): my global_id is 154106 2024-09-16T04:24:23.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2dc003520 con 0x7fc2f013d060 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181686636 0 0) 0x7fc2f0137a10 con 0x7fc2f00a2410 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): my global_id is 144213 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f0140340 con 0x7fc2f00a2410 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256921864 0 0) 0x7fc2f009d8e0 con 0x7fc2f00a1b10 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): my global_id is 7180 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f0137a10 con 0x7fc2f00a1b10 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 748476645 0 0) 0x7fc2f0140340 con 0x7fc2f00a2410 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2f009d8e0 con 0x7fc2f00a2410 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e40018d0 con 0x7fc2f00a2410 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): dump: 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.646 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 734860199 0 0) 0x7fc2dc003520 con 0x7fc2f013d060 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2f0140340 con 0x7fc2f013d060 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3639409163 0 0) 0x7fc2f0137a10 con 0x7fc2f00a1b10 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2dc003520 con 0x7fc2f00a1b10 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e8001180 con 0x7fc2f013d060 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): dump: 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.648 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e0001e20 con 0x7fc2f00a1b10 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient(hunting): dump: 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.649 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2320199236 0 0) 0x7fc2f009d8e0 con 0x7fc2f00a2410 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.148:6790/0 conn(0x7fc2f013d060 legacy=0x7fc2f01d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.171:6789/0 conn(0x7fc2f00a1b10 legacy=0x7fc2f009cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 monclient: found mon.a 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f013cd30 con 0x7fc2f00a2410 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 5 monclient: authenticate success, global_id 144213 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient: _renew_subs 2024-09-16T04:24:23.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.628+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f00a0130 con 0x7fc2f00a2410 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2e4001d00 con 0x7fc2f00a2410 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e4003a60 con 0x7fc2f00a2410 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 10 monclient: dump: 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.652 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2f75d6640 10 monclient: _renew_subs 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2f75d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2f0137cf0 con 0x7fc2f00a2410 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2ef7fe640 4 set_mon_vals no callback set 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.629+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc2e4003c80 con 0x7fc2f00a2410 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2f75d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2f009f040 con 0x7fc2f013c4a0 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fc2e40477c0 con 0x7fc2f00a2410 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2f5b73640 10 monclient: get_auth_request con 0x7fc2dc039210 auth_method 0 2024-09-16T04:24:23.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019308673 0 0) 0x7fc2f009f040 con 0x7fc2f013c4a0 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient(hunting): my global_id is 154109 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f009d8e0 con 0x7fc2f013c4a0 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 360826632 0 0) 0x7fc2f009d8e0 con 0x7fc2f013c4a0 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2f009f040 con 0x7fc2f013c4a0 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e0004d90 con 0x7fc2f013c4a0 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 10 monclient: dump: 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.654 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4126236430 0 0) 0x7fc2f009f040 con 0x7fc2f013c4a0 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.630+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2f013c880 con 0x7fc2f013c4a0 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.631+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc2e0006250 con 0x7fc2f013c4a0 2024-09-16T04:24:23.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.631+0000 7fc2edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.631+0000 7fc2edffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:23.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.631+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.148:6790/0 conn(0x7fc2f013c4a0 legacy=0x7fc2f013d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.631+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.148:6790/0 conn(0x7fc2f013c4a0 legacy=0x7fc2f013d820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2f75d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc2f0062630 con 0x7fc2f002d650 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158822395 0 0) 0x7fc2f0062630 con 0x7fc2f002d650 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient(hunting): my global_id is 154112 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f009f040 con 0x7fc2f002d650 2024-09-16T04:24:23.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1109889475 0 0) 0x7fc2f009f040 con 0x7fc2f002d650 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2f0062630 con 0x7fc2f002d650 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc2e00068c0 con 0x7fc2f002d650 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient: dump: 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.659 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2818461276 0 0) 0x7fc2f0062630 con 0x7fc2f002d650 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2f002da60 con 0x7fc2f002d650 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3974 (unknown 0 0 2188763911) 0x7fc2e0006790 con 0x7fc2f002d650 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.148:6790/0 conn(0x7fc2f002d650 legacy=0x7fc2f013df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.637+0000 7fc2edffb640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.148:6790/0 conn(0x7fc2f002d650 legacy=0x7fc2f013df30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 70, 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 544, 2024-09-16T04:24:23.661 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.662 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.663 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:23.664 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:23.665 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.666 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:23.667 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:23.668 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.669 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.670 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.671 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.672 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:23.673 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:23.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.171:6812/7327273 conn(0x7fc2dc039210 legacy=0x7fc2dc03b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 10 monclient: shutdown 2024-09-16T04:24:23.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 >> v1:172.21.15.148:6789/0 conn(0x7fc2f00a2410 legacy=0x7fc2f009d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f5b73640 1 -- 172.21.15.148:0/1261302800 reap_dead start 2024-09-16T04:24:23.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f5b73640 1 -- 172.21.15.148:0/1261302800 reap_dead start 2024-09-16T04:24:23.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 shutdown_connections 2024-09-16T04:24:23.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 >> 172.21.15.148:0/1261302800 conn(0x7fc2f00a97b0 msgr2=0x7fc2f00a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 shutdown_connections 2024-09-16T04:24:23.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.638+0000 7fc2f75d6640 1 -- 172.21.15.148:0/1261302800 wait complete. 2024-09-16T04:24:23.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:24:23.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:24:23.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:24:23.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.772+0000 7f1d43023640 10 monclient: get_monmap_and_config 2024-09-16T04:24:23.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.772+0000 7f1d43023640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.773 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.772+0000 7f1d43023640 10 monclient: monmap: 2024-09-16T04:24:23.773 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.773 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:23.772965+0000 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:23.772965+0000 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:23.774 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.772+0000 7f1d43023640 1 Processor -- start 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.772+0000 7f1d43023640 1 -- start start 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.772+0000 7f1d43023640 10 monclient: init 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c0a0e00 con 0x7f1d3c13d060 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c0a10e0 con 0x7f1d3c0a2410 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c0a4970 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 10 monclient(hunting): authenticate will time out at 1884.398350s 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d33fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f1d3c0a1b10 0x7f1d3c13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37530/0 (socket says 172.21.15.148:37530) 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d415c0640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1d3c13d060 0x7f1d3c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46594/0 (socket says 172.21.15.148:46594) 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d33fff640 1 -- 172.21.15.148:0/105305574 learned_addr learned my addr 172.21.15.148:0/105305574 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 535999511 0 0) 0x7f1d3c0a4970 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): my global_id is 154115 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d24003520 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3917024157 0 0) 0x7f1d3c0a0e00 con 0x7f1d3c13d060 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient(hunting): my global_id is 144216 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d3c0a4970 con 0x7f1d3c13d060 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3005320075 0 0) 0x7f1d3c0a4970 con 0x7f1d3c13d060 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 >> v1:172.21.15.148:6790/0 conn(0x7f1d3c0a1b10 legacy=0x7f1d3c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 >> v1:172.21.15.171:6789/0 conn(0x7f1d3c0a2410 legacy=0x7f1d3c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 monclient: found mon.noname-a 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c06dfd0 con 0x7f1d3c13d060 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d43023640 5 monclient: authenticate success, global_id 144216 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.773+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d34001990 con 0x7f1d3c13d060 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:23.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: dump: 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1d34001bf0 con 0x7f1d3c13d060 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 1 -- 172.21.15.148:0/105305574 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d34001fc0 con 0x7f1d3c13d060 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 4 set_mon_vals no callback set 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d337fe640 10 monclient: dump: 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.788 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 10 monclient: shutdown 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 1 -- 172.21.15.148:0/105305574 >> v1:172.21.15.148:6789/0 conn(0x7f1d3c13d060 legacy=0x7f1d3c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 1 -- 172.21.15.148:0/105305574 shutdown_connections 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 1 -- 172.21.15.148:0/105305574 >> 172.21.15.148:0/105305574 conn(0x7f1d3c0a97b0 msgr2=0x7f1d3c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 1 -- 172.21.15.148:0/105305574 shutdown_connections 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.774+0000 7f1d43023640 1 -- 172.21.15.148:0/105305574 wait complete. 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.775+0000 7f1d43023640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.775+0000 7f1d43023640 10 monclient: monmap: 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.775+0000 7f1d43023640 1 Processor -- start 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.775+0000 7f1d43023640 1 -- start start 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.775+0000 7f1d43023640 10 monclient: init 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c1403b0 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c13ccf0 con 0x7f1d3c13d060 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c0a0130 con 0x7f1d3c0a2410 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient(hunting): authenticate will time out at 1884.401200s 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d33fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f1d3c0a2410 0x7f1d3c1d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37546/0 (socket says 172.21.15.148:37546) 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d40dbf640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1d3c0a1b10 0x7f1d3c13bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46600/0 (socket says 172.21.15.148:46600) 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d33fff640 1 -- 172.21.15.148:0/1369173261 learned_addr learned my addr 172.21.15.148:0/1369173261 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622212660 0 0) 0x7f1d3c0a0130 con 0x7f1d3c0a2410 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): my global_id is 154118 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d28003550 con 0x7f1d3c0a2410 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770908838 0 0) 0x7f1d3c1403b0 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): my global_id is 144219 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d3c0a0130 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624453758 0 0) 0x7f1d3c13ccf0 con 0x7f1d3c13d060 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): my global_id is 7186 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d3c1403b0 con 0x7f1d3c13d060 2024-09-16T04:24:23.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 79071335 0 0) 0x7f1d28003550 con 0x7f1d3c0a2410 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d3c13ccf0 con 0x7f1d3c0a2410 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d2c001310 con 0x7f1d3c0a2410 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): dump: 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.794 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3583431188 0 0) 0x7f1d3c0a0130 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d28003550 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d380018f0 con 0x7f1d3c0a1b10 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): dump: 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2133510667 0 0) 0x7f1d3c1403b0 con 0x7f1d3c13d060 2024-09-16T04:24:23.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d3c0a0130 con 0x7f1d3c13d060 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d34001f40 con 0x7f1d3c13d060 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient(hunting): dump: 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.797 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3571624456 0 0) 0x7f1d3c13ccf0 con 0x7f1d3c0a2410 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.171:6789/0 conn(0x7f1d3c13d060 legacy=0x7f1d3c1d3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.148:6789/0 conn(0x7f1d3c0a1b10 legacy=0x7f1d3c13bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 monclient: found mon.c 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c09e5c0 con 0x7f1d3c0a2410 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 5 monclient: authenticate success, global_id 154118 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient: _renew_subs 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.776+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c137710 con 0x7f1d3c0a2410 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d43023640 10 monclient: _renew_subs 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d43023640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d3c09f2b0 con 0x7f1d3c0a2410 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1d2c001c50 con 0x7f1d3c0a2410 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d2c004a90 con 0x7f1d3c0a2410 2024-09-16T04:24:23.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d32ffd640 4 set_mon_vals no callback set 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 10 monclient: dump: 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.800 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f1d2c00ff40 con 0x7f1d3c0a2410 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d43023640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.777+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1d1c0065c0 con 0x7f1d1c002860 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f1d2c047ac0 con 0x7f1d3c0a2410 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d415c0640 10 monclient: get_auth_request con 0x7f1d28039190 auth_method 0 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1612570366 0 0) 0x7f1d1c0065c0 con 0x7f1d1c002860 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient(hunting): my global_id is 7189 2024-09-16T04:24:23.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d3c13ccf0 con 0x7f1d1c002860 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4125334650 0 0) 0x7f1d3c13ccf0 con 0x7f1d1c002860 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d1c0065c0 con 0x7f1d1c002860 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d380037b0 con 0x7f1d1c002860 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 10 monclient: dump: 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2047293149 0 0) 0x7f1d1c0065c0 con 0x7f1d1c002860 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.778+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d1c00a150 con 0x7f1d1c002860 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.779+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1d380057a0 con 0x7f1d1c002860 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.779+0000 7f1d317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.779+0000 7f1d317fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.779+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.171:6789/0 conn(0x7f1d1c002860 legacy=0x7f1d1c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.779+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.171:6789/0 conn(0x7f1d1c002860 legacy=0x7f1d1c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d43023640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1d1c009e20 con 0x7f1d1c0090e0 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560339353 0 0) 0x7f1d1c009e20 con 0x7f1d1c0090e0 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient(hunting): my global_id is 7192 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d1c0065c0 con 0x7f1d1c0090e0 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1351593708 0 0) 0x7f1d1c0065c0 con 0x7f1d1c0090e0 2024-09-16T04:24:23.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d1c009e20 con 0x7f1d1c0090e0 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f1d38003e50 con 0x7f1d1c0090e0 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.785+0000 7f1d317fa640 10 monclient: dump: 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.806 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2162929053 0 0) 0x7f1d1c009e20 con 0x7f1d1c0090e0 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d1c00c390 con 0x7f1d1c0090e0 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4312 (unknown 0 0 211312148) 0x7f1d380040f0 con 0x7f1d1c0090e0 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.171:6789/0 conn(0x7f1d1c0090e0 legacy=0x7f1d1c009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d317fa640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.171:6789/0 conn(0x7f1d1c0090e0 legacy=0x7f1d1c009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.807 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 70, 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 1087824, 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:23.808 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.809 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.810 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:23.811 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:23.812 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:23.813 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.814 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:23.815 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:23.816 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:23.817 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.818 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:23.819 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:23.820 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:23.820 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:23.820 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:23.820 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.171:6812/7327273 conn(0x7f1d28039190 legacy=0x7f1d2803b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 10 monclient: shutdown 2024-09-16T04:24:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 >> v1:172.21.15.148:6790/0 conn(0x7f1d3c0a2410 legacy=0x7f1d3c1d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d415c0640 1 -- 172.21.15.148:0/1369173261 reap_dead start 2024-09-16T04:24:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d415c0640 1 -- 172.21.15.148:0/1369173261 reap_dead start 2024-09-16T04:24:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 shutdown_connections 2024-09-16T04:24:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 >> 172.21.15.148:0/1369173261 conn(0x7f1d3c0a97b0 msgr2=0x7f1d3c135860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 shutdown_connections 2024-09-16T04:24:23.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.786+0000 7f1d43023640 1 -- 172.21.15.148:0/1369173261 wait complete. 2024-09-16T04:24:23.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:24:23.822 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:24:23.823 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:24:23.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.917+0000 7f8208080640 10 monclient: get_monmap_and_config 2024-09-16T04:24:23.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.917+0000 7f8208080640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.917+0000 7f8208080640 10 monclient: monmap: 2024-09-16T04:24:23.919 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.919 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.919 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:23.918519+0000 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:23.918519+0000 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:23.920 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 1 Processor -- start 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 1 -- start start 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient: init 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient(hunting): picked mon.noname-a con 0x7f82000a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient(hunting): picked mon.noname-c con 0x7f820009b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient(hunting): picked mon.noname-b con 0x7f820009b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820009e040 con 0x7f82000a0e90 2024-09-16T04:24:23.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820009d660 con 0x7f820009b170 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820009c0d0 con 0x7f820009b5c0 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.918+0000 7f8208080640 10 monclient(hunting): authenticate will time out at 1884.543956s 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f820661d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f820009b5c0 0x7f820013bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37556/0 (socket says 172.21.15.148:37556) 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8205e1c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f82000a0e90 0x7f820009aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46612/0 (socket says 172.21.15.148:46612) 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f820661d640 1 -- 172.21.15.148:0/3183369933 learned_addr learned my addr 172.21.15.148:0/3183369933 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917378398 0 0) 0x7f81f0001110 con 0x7f82000a0e90 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): my global_id is 144222 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81ec003440 con 0x7f82000a0e90 2024-09-16T04:24:23.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974422241 0 0) 0x7f820009c0d0 con 0x7f820009b5c0 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): my global_id is 154121 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81f0001110 con 0x7f820009b5c0 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3212475654 0 0) 0x7f81fc001110 con 0x7f820009b170 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient(hunting): my global_id is 7195 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f820009c0d0 con 0x7f820009b170 2024-09-16T04:24:23.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 526895674 0 0) 0x7f81ec003440 con 0x7f82000a0e90 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 >> v1:172.21.15.148:6790/0 conn(0x7f820009b5c0 legacy=0x7f820013bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 >> v1:172.21.15.171:6789/0 conn(0x7f820009b170 legacy=0x7f82001383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 monclient: found mon.noname-a 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820009d380 con 0x7f82000a0e90 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f81f00017e0 con 0x7f82000a0e90 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8208080640 5 monclient: authenticate success, global_id 144222 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: dump: 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8208080640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:23.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81f0001a90 con 0x7f82000a0e90 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 1 -- 172.21.15.148:0/3183369933 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f81f00047e0 con 0x7f82000a0e90 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8208080640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8208080640 4 set_mon_vals no callback set 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.919+0000 7f8204e1a640 10 monclient: dump: 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 10 monclient: shutdown 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 1 -- 172.21.15.148:0/3183369933 >> v1:172.21.15.148:6789/0 conn(0x7f82000a0e90 legacy=0x7f820009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 1 -- 172.21.15.148:0/3183369933 shutdown_connections 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 1 -- 172.21.15.148:0/3183369933 >> 172.21.15.148:0/3183369933 conn(0x7f82000a5640 msgr2=0x7f82000a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 1 -- 172.21.15.148:0/3183369933 shutdown_connections 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 1 -- 172.21.15.148:0/3183369933 wait complete. 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 10 monclient: build_initial_monmap 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.920+0000 7f8208080640 10 monclient: monmap: 2024-09-16T04:24:23.934 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 1 Processor -- start 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 1 -- start start 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient: init 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:23.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient(hunting): picked mon.noname-c con 0x7f820009b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient(hunting): picked mon.noname-a con 0x7f820009b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient(hunting): picked mon.noname-b con 0x7f82000a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82001c1a10 con 0x7f820009b5c0 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82001a61a0 con 0x7f82000a0e90 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82001a8c60 con 0x7f820009b170 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8208080640 10 monclient(hunting): authenticate will time out at 1884.546729s 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8205e1c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f820009b170 0x7f82001c1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37560/0 (socket says 172.21.15.148:37560) 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f8205e1c640 1 -- 172.21.15.148:0/3865122143 learned_addr learned my addr 172.21.15.148:0/3865122143 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f820561b640 1 -- 172.21.15.148:0/3865122143 >> v1:172.21.15.148:6789/0 conn(0x7f820009b5c0 legacy=0x7f82001cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:24:23.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f820561b640 1 -- 172.21.15.148:0/3865122143 >> v1:172.21.15.148:6789/0 conn(0x7f820009b5c0 legacy=0x7f82001cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f820561b640 1 --1- 172.21.15.148:0/3865122143 >> v1:172.21.15.148:6789/0 conn(0x7f820009b5c0 0x7f82001cc630 :-1 s=OPENED pgs=634 cs=1 l=1).handle_message read tag failed 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f820561b640 1 --1- 172.21.15.148:0/3865122143 >> v1:172.21.15.148:6789/0 conn(0x7f820009b5c0 0x7f82001cc630 :-1 s=OPENED pgs=634 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282364032 0 0) 0x7f82001a8c60 con 0x7f820009b170 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 10 monclient(hunting): my global_id is 154124 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.921+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81e4003520 con 0x7f820009b170 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3629269222 0 0) 0x7f82001a61a0 con 0x7f82000a0e90 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:23.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): my global_id is 7198 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82001a8c60 con 0x7f82000a0e90 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1154068816 0 0) 0x7f81e4003520 con 0x7f820009b170 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f82001a61a0 con 0x7f820009b170 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f81f0001600 con 0x7f820009b170 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): dump: 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.938 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 398745790 0 0) 0x7f82001a61a0 con 0x7f820009b170 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 >> v1:172.21.15.171:6789/0 conn(0x7f82000a0e90 legacy=0x7f82001cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 >> v1:172.21.15.148:6789/0 conn(0x7f820009b5c0 legacy=0x7f82001cc630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 monclient: found mon.c 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82001a6990 con 0x7f820009b170 2024-09-16T04:24:23.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: _finish_auth 0 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 5 monclient: authenticate success, global_id 154124 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 10 monclient: _renew_subs 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f82001a8980 con 0x7f820009b170 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 10 monclient: _renew_subs 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f82001c3d70 con 0x7f820009b170 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81f0001a90 con 0x7f820009b170 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:23.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f81f0004da0 con 0x7f820009b170 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f7fff640 4 set_mon_vals no callback set 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.922+0000 7f81f67fc640 10 monclient: dump: 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:23.941 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f81f00101d0 con 0x7f820009b170 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f81f0047910 con 0x7f820009b170 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f8208080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f8208080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f8208080640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e8005080 con 0x7f820009b170 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.923+0000 7f820561b640 10 monclient: get_auth_request con 0x7f81e4039000 auth_method 0 2024-09-16T04:24:23.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.928+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f820009e040 con 0x7f820009b170 2024-09-16T04:24:23.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.928+0000 7f81f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:23.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:23.928+0000 7f81f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:24.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.159+0000 7f8208080640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:24:24.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.159+0000 7f8208080640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:24:24.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.159+0000 7f8208080640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:24.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.159+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f81e8005b80 con 0x7f820009b170 2024-09-16T04:24:24.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f81f67fc640 1 -- 172.21.15.148:0/3865122143 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f81f0015d10 con 0x7f820009b170 2024-09-16T04:24:24.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f81f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:24:24.161 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f81f67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:24.162 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.162 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=1834993774,config=692619927,health=2679364835,kv=3730950374,logm=1304209993}) 2024-09-16T04:24:24.162 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=3082373482,config=692619927,health=4068505762,kv=2270513034,logm=3593779912}) 2024-09-16T04:24:24.162 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=1834993774,config=692619927,health=2679364835,kv=3730950374,logm=1304209993}) 2024-09-16T04:24:24.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.160+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=10,kv=9,logm=49} crc {auth=2117256348,config=692619927,health=2709829417,kv=3730950374,logm=227425927}) 2024-09-16T04:24:24.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.161+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.161+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592560242}) 2024-09-16T04:24:24.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.161+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2847281335}) 2024-09-16T04:24:24.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.161+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.161+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592560242}) 2024-09-16T04:24:24.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.161+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699047945}) 2024-09-16T04:24:24.164 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 >> v1:172.21.15.171:6812/7327273 conn(0x7f81e4039000 legacy=0x7f81e403b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 10 monclient: shutdown 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 >> v1:172.21.15.148:6790/0 conn(0x7f820009b170 legacy=0x7f82001c1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 shutdown_connections 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 >> 172.21.15.148:0/3865122143 conn(0x7f82000a5640 msgr2=0x7f8200071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 shutdown_connections 2024-09-16T04:24:24.165 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f8208080640 1 -- 172.21.15.148:0/3865122143 wait complete. 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.162+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579702845}) 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.163+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2619442232}) 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.163+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.163+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579702845}) 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.163+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250150458}) 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251634724}) 2024-09-16T04:24:24.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=226931797}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251634724}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775596737}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.164+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390425550}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494693102}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390425550}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955256489}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507875204}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678523889}) 2024-09-16T04:24:24.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507875204}) 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.165+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108370275}) 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.166+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.166+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160969248}) 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.166+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3948708818}) 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.166+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.166+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160969248}) 2024-09-16T04:24:24.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.166+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2992149881}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067021268}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091561306}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067021268}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562285968}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696082710}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3034575992}) 2024-09-16T04:24:24.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696082710}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.167+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168552937}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.168+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.168+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772539513}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.168+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557848915}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.168+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.168+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772539513}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.168+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803554485}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.169+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.169+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167136434}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.169+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2035857101}) 2024-09-16T04:24:24.176 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.169+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.169+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167136434}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.169+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1770303581}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038582829}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993456183}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038582829}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628060344}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158228599}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545096212}) 2024-09-16T04:24:24.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=158228599}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.170+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679755913}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.171+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.171+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139049958}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.171+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2964811373}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.171+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.171+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139049958}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.171+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74740369}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424018740}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195450925}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424018740}) 2024-09-16T04:24:24.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=95840427}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137203558}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4095813991}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137203558}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.172+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268706330}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.173+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.173+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814916330}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.173+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1990283385}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.173+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.173+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814916330}) 2024-09-16T04:24:24.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.173+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2829042485}) 2024-09-16T04:24:24.180 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441596743}) 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496400007}) 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441596743}) 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567569333}) 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438914068}) 2024-09-16T04:24:24.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1518611105}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438914068}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.174+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088494466}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.175+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.175+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836993039}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.175+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423251131}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.175+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.175+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836993039}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.175+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485160280}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192404722}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647913748}) 2024-09-16T04:24:24.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192404722}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824646339}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860111523}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3883041761}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860111523}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.176+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648816255}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317971448}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779561795}) 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317971448}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284549439}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743674638}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935787095}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743674638}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.177+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156676056}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.178+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.178+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434691664}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.178+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=425999871}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.178+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.178+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434691664}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.178+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174785312}) 2024-09-16T04:24:24.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.179+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.179+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108633323}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.179+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758488564}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.179+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.179+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3108633323}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.179+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352694664}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.180+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.180+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430200845}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.180+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781249335}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.180+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.180+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430200845}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.180+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285157233}) 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.197 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986512741}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1832727067}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986512741}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2101951677}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839283893}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=406932188}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839283893}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.181+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848875383}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.182+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.182+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484088355}) 2024-09-16T04:24:24.198 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.182+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589840661}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.182+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.182+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484088355}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.182+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1246423901}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165024776}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714256779}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4165024776}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3445581617}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28133415}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=155021564}) 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.199 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28133415}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.183+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197290696}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995806569}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4224910059}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995806569}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529683115}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028482969}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812788440}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028482969}) 2024-09-16T04:24:24.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.184+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152785663}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.185+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.185+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096723414}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.185+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058812556}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.185+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.185+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096723414}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.185+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=364282410}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356190285}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3778922110}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356190285}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357900921}) 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241059107}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=747681830}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241059107}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.186+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=884416366}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.187+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.187+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822968670}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.187+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606795732}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.187+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.187+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822968670}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.187+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1216784884}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839141851}) 2024-09-16T04:24:24.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1883772704}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839141851}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=231524834}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940217590}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196148176}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940217590}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.188+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841191978}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.189+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.189+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649329008}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.189+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170969491}) 2024-09-16T04:24:24.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.189+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.189+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649329008}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.189+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288686020}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147040221}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3888960835}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147040221}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630207101}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420242583}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292506095}) 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.204 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420242583}) 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.190+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697687942}) 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.191+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.191+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1741764556}) 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.191+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256845792}) 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.191+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.191+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1741764556}) 2024-09-16T04:24:24.205 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.191+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446578996}) 2024-09-16T04:24:24.212 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.212 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511998578}) 2024-09-16T04:24:24.212 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907941971}) 2024-09-16T04:24:24.212 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511998578}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187485760}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995284260}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984069652}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995284260}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.192+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686616242}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.193+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.193+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814375847}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.193+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170377138}) 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.193+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.213 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.193+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814375847}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.193+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767214999}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.194+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.194+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705041105}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.194+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614391864}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.194+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.194+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705041105}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.194+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2838367369}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621109167}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71476800}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621109167}) 2024-09-16T04:24:24.214 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2579558509}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179654527}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1929693556}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179654527}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.195+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416870108}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.196+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.196+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112465866}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.196+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654879017}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.196+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.196+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112465866}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.196+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901493112}) 2024-09-16T04:24:24.215 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881169348}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2518625752}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881169348}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450997898}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051985136}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320706469}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051985136}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.197+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263743634}) 2024-09-16T04:24:24.216 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.198+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.198+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960947317}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.198+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174298566}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.198+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.198+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960947317}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.198+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4159228540}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1550867159}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=527354187}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1550867159}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1449528297}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303877278}) 2024-09-16T04:24:24.217 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=747980188}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303877278}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.199+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42177911}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.200+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.200+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183718236}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.200+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275294733}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.200+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.200+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3183718236}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.200+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269533831}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476398263}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778560369}) 2024-09-16T04:24:24.218 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476398263}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=296250936}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209742764}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.201+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163550695}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.202+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.202+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209742764}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.202+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4196680698}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.203+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.203+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=44} crc {logm=474615255,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=1434592485,mon_config_key=167078543,monmap=3140514009,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=219976213}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.203+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=8,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=46} crc {logm=3125620407,mdsmap=1097804592,mgr=3460664738,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=1434592485,mon_config_key=167078543,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=2577231423}) 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.203+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.219 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.203+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=44} crc {logm=474615255,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=1434592485,mon_config_key=167078543,monmap=3140514009,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=219976213}) 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.203+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=43} crc {logm=474615255,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=1352170280,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=3880080259}) 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.204+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.205+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=4011866483}) 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.205+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=75} crc {osdmap=4121915824}) 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.205+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.205+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=4011866483}) 2024-09-16T04:24:24.220 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:24:24.205+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=74} crc {osdmap=700837165}) 2024-09-16T04:24:25.182 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:24:25.182 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:24:25.286 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.285+0000 7f2c70ff1640 10 monclient: get_monmap_and_config 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.285+0000 7f2c70ff1640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.285+0000 7f2c70ff1640 10 monclient: monmap: 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:25.286135+0000 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:25.286135+0000 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:25.287 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.285+0000 7f2c70ff1640 1 Processor -- start 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.285+0000 7f2c70ff1640 1 -- start start 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.285+0000 7f2c70ff1640 10 monclient: init 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c0a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c0a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c13d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0a0590 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0a1270 con 0x7f2c6c0a4580 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0a4fd0 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c70ff1640 10 monclient(hunting): authenticate will time out at 1885.911518s 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6a59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2c6c0a3a90 0x7f2c6c0a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37566/0 (socket says 172.21.15.148:37566) 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6a59c640 1 -- 172.21.15.148:0/621023508 learned_addr learned my addr 172.21.15.148:0/621023508 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2082880445 0 0) 0x7f2c6c0a4fd0 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): my global_id is 154127 2024-09-16T04:24:25.300 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c54003520 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666501703 0 0) 0x7f2c64001490 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 10 monclient(hunting): my global_id is 144228 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.286+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6c0a4fd0 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1650922163 0 0) 0x7f2c6c0a4fd0 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 >> v1:172.21.15.148:6790/0 conn(0x7f2c6c0a3a90 legacy=0x7f2c6c0a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 >> v1:172.21.15.171:6789/0 conn(0x7f2c6c0a4580 legacy=0x7f2c6c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 monclient: found mon.noname-a 2024-09-16T04:24:25.301 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c06e170 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 5 monclient: authenticate success, global_id 144228 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c64001b60 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: dump: 2024-09-16T04:24:25.302 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c64001dc0 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 1 -- 172.21.15.148:0/621023508 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c64003990 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 4 set_mon_vals no callback set 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c6959a640 10 monclient: dump: 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.304 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 10 monclient: shutdown 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 1 -- 172.21.15.148:0/621023508 >> v1:172.21.15.148:6789/0 conn(0x7f2c6c13d3c0 legacy=0x7f2c6c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 1 -- 172.21.15.148:0/621023508 shutdown_connections 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 1 -- 172.21.15.148:0/621023508 >> 172.21.15.148:0/621023508 conn(0x7f2c6c0a9a40 msgr2=0x7f2c6c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 1 -- 172.21.15.148:0/621023508 shutdown_connections 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.287+0000 7f2c70ff1640 1 -- 172.21.15.148:0/621023508 wait complete. 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.288+0000 7f2c70ff1640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.288+0000 7f2c70ff1640 10 monclient: monmap: 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:25.305 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.288+0000 7f2c70ff1640 1 Processor -- start 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 1 -- start start 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient: init 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:24:25.306 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c0a3a90 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c0a4580 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c13d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c13aa40 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c13c450 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c6c0a81c0 con 0x7f2c6c0a4580 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c70ff1640 10 monclient(hunting): authenticate will time out at 1885.914589s 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c6a59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f2c6c0a3a90 0x7f2c6c138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46626/0 (socket says 172.21.15.148:46626) 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c6a59c640 1 -- 172.21.15.148:0/1575534907 learned_addr learned my addr 172.21.15.148:0/1575534907 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c69d9b640 1 --1- 172.21.15.148:0/1575534907 >> v1:172.21.15.148:6790/0 conn(0x7f2c6c0a4580 0x7f2c6c138e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37578/0 (socket says 172.21.15.148:37578) 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190763783 0 0) 0x7f2c6c0a81c0 con 0x7f2c6c0a4580 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): my global_id is 154130 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c4c003520 con 0x7f2c6c0a4580 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826148972 0 0) 0x7f2c6c13aa40 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): my global_id is 144231 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6c0a81c0 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 468553290 0 0) 0x7f2c6c13c450 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.308 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 10 monclient(hunting): my global_id is 7204 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.289+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c6c13aa40 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 293355211 0 0) 0x7f2c4c003520 con 0x7f2c6c0a4580 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c6c13c450 con 0x7f2c6c0a4580 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c600018b0 con 0x7f2c6c0a4580 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): dump: 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.309 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 599347434 0 0) 0x7f2c6c0a81c0 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c4c003520 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c5c0011c0 con 0x7f2c6c0a3a90 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.310 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): dump: 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 54769934 0 0) 0x7f2c6c13aa40 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2c6c0a81c0 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.311 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c64001360 con 0x7f2c6c13d3c0 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient(hunting): dump: 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.312 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 706726573 0 0) 0x7f2c6c13c450 con 0x7f2c6c0a4580 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 >> v1:172.21.15.171:6789/0 conn(0x7f2c6c13d3c0 legacy=0x7f2c6c1d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 >> v1:172.21.15.148:6789/0 conn(0x7f2c6c0a3a90 legacy=0x7f2c6c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 monclient: found mon.c 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6c0a0280 con 0x7f2c6c0a4580 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 5 monclient: authenticate success, global_id 154130 2024-09-16T04:24:25.313 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 10 monclient: _renew_subs 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6c0a5510 con 0x7f2c6c0a4580 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 10 monclient: _renew_subs 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2c6c13c170 con 0x7f2c6c0a4580 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2c60003610 con 0x7f2c6c0a4580 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2c60003960 con 0x7f2c6c0a4580 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c68d99640 4 set_mon_vals no callback set 2024-09-16T04:24:25.314 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.290+0000 7f2c5affd640 10 monclient: dump: 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.315 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2c60003b80 con 0x7f2c6c0a4580 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c70ff1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c70ff1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c70ff1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f2c60047800 con 0x7f2c6c0a4580 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c50005080 con 0x7f2c6c0a4580 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.291+0000 7f2c6a59c640 10 monclient: get_auth_request con 0x7f2c4c039210 auth_method 0 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.295+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2c60015aa0 con 0x7f2c6c0a4580 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.295+0000 7f2c5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.316 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.295+0000 7f2c5affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:25.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c70ff1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:24:25.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c70ff1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:25.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c70ff1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2c50005370 con 0x7f2c6c0a4580 2024-09-16T04:24:25.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c5affd640 1 -- 172.21.15.148:0/1575534907 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2077293608) 0x7f2c600153a0 con 0x7f2c6c0a4580 2024-09-16T04:24:25.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:25.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.529+0000 7f2c5affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:25.531 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:24:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 >> v1:172.21.15.171:6812/7327273 conn(0x7f2c4c039210 legacy=0x7f2c4c03b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 10 monclient: shutdown 2024-09-16T04:24:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 >> v1:172.21.15.148:6790/0 conn(0x7f2c6c0a4580 legacy=0x7f2c6c138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 shutdown_connections 2024-09-16T04:24:25.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 >> 172.21.15.148:0/1575534907 conn(0x7f2c6c0a9a40 msgr2=0x7f2c6c140550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 shutdown_connections 2024-09-16T04:24:25.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.531+0000 7f2c70ff1640 1 -- 172.21.15.148:0/1575534907 wait complete. 2024-09-16T04:24:25.543 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:24:25.543 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:24:25.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:24:25.544 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.641+0000 7f398aaac640 10 monclient: get_monmap_and_config 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.641+0000 7f398aaac640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.641+0000 7f398aaac640 10 monclient: monmap: 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:25.642348+0000 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:25.642348+0000 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:25.644 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:25.644 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 1 Processor -- start 2024-09-16T04:24:25.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 1 -- start start 2024-09-16T04:24:25.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient: init 2024-09-16T04:24:25.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:25.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient(hunting): picked mon.noname-b con 0x7f3984134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient(hunting): picked mon.noname-c con 0x7f398409e240 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient(hunting): picked mon.noname-a con 0x7f398413ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3984135360 con 0x7f398413ae80 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3984134440 con 0x7f398409e240 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398409e650 con 0x7f3984134870 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f398aaac640 10 monclient(hunting): authenticate will time out at 1886.267777s 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f3988848640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f3984134870 0x7f3984134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37594/0 (socket says 172.21.15.148:37594) 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f3988848640 1 -- 172.21.15.148:0/3974666135 learned_addr learned my addr 172.21.15.148:0/3974666135 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.642+0000 7f3989049640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f398413ae80 0x7f398413d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46632/0 (socket says 172.21.15.148:46632) 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728300795 0 0) 0x7f3984135360 con 0x7f398413ae80 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): my global_id is 144234 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f396c003520 con 0x7f398413ae80 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 509315831 0 0) 0x7f398409e650 con 0x7f3984134870 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): my global_id is 154133 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3984135360 con 0x7f3984134870 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2193207715 0 0) 0x7f3984134440 con 0x7f398409e240 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient(hunting): my global_id is 7207 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f398409e650 con 0x7f398409e240 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 280888901 0 0) 0x7f396c003520 con 0x7f398413ae80 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 >> v1:172.21.15.148:6790/0 conn(0x7f3984134870 legacy=0x7f3984134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 >> v1:172.21.15.171:6789/0 conn(0x7f398409e240 legacy=0x7f3984139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3984073230 con 0x7f398413ae80 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f397c0018c0 con 0x7f398413ae80 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 5 monclient: authenticate success, global_id 144234 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: dump: 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.663 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f397c001be0 con 0x7f398413ae80 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 1 -- 172.21.15.148:0/3974666135 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f397c001fb0 con 0x7f398413ae80 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 4 set_mon_vals no callback set 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f397b7fe640 10 monclient: dump: 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.664 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 10 monclient: shutdown 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.643+0000 7f398aaac640 1 -- 172.21.15.148:0/3974666135 >> v1:172.21.15.148:6789/0 conn(0x7f398413ae80 legacy=0x7f398413d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 1 -- 172.21.15.148:0/3974666135 shutdown_connections 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 1 -- 172.21.15.148:0/3974666135 >> 172.21.15.148:0/3974666135 conn(0x7f39840a5640 msgr2=0x7f39840a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 1 -- 172.21.15.148:0/3974666135 shutdown_connections 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 1 -- 172.21.15.148:0/3974666135 wait complete. 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 10 monclient: monmap: 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:25.666 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 1 Processor -- start 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 1 -- start start 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.644+0000 7f398aaac640 10 monclient: init 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient(hunting): picked mon.noname-a con 0x7f398409e240 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient(hunting): picked mon.noname-b con 0x7f3984134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient(hunting): picked mon.noname-c con 0x7f398413ae80 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39840a4190 con 0x7f398409e240 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398411dba0 con 0x7f3984134870 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398411f330 con 0x7f398413ae80 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f398aaac640 10 monclient(hunting): authenticate will time out at 1886.270599s 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f3989049640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f398413ae80 0x7f39841d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37602/0 (socket says 172.21.15.148:37602) 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f3988848640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f398409e240 0x7f3984133970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46638/0 (socket says 172.21.15.148:46638) 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f3989049640 1 -- 172.21.15.148:0/2716835638 learned_addr learned my addr 172.21.15.148:0/2716835638 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627791851 0 0) 0x7f398411f330 con 0x7f398413ae80 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): my global_id is 154136 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3970003520 con 0x7f398413ae80 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148981867 0 0) 0x7f3984135360 con 0x7f398409e240 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 10 monclient(hunting): my global_id is 144237 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.645+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f398411f330 con 0x7f398409e240 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675238797 0 0) 0x7f398411dba0 con 0x7f3984134870 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): my global_id is 7210 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3984135360 con 0x7f3984134870 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1273119630 0 0) 0x7f3970003520 con 0x7f398413ae80 2024-09-16T04:24:25.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f398411dba0 con 0x7f398413ae80 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f397c001880 con 0x7f398413ae80 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): dump: 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.670 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3083130792 0 0) 0x7f398411f330 con 0x7f398409e240 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3970003520 con 0x7f398409e240 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f39800017b0 con 0x7f398409e240 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient(hunting): dump: 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.671 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1683505058 0 0) 0x7f3984135360 con 0x7f3984134870 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f398411f330 con 0x7f3984134870 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1429131050 0 0) 0x7f398411dba0 con 0x7f398413ae80 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.171:6789/0 conn(0x7f3984134870 legacy=0x7f39841ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.148:6789/0 conn(0x7f398409e240 legacy=0x7f3984133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 monclient: found mon.c 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398411d570 con 0x7f398413ae80 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 5 monclient: authenticate success, global_id 154136 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 10 monclient: _renew_subs 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f398411f050 con 0x7f398413ae80 2024-09-16T04:24:25.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 10 monclient: _renew_subs 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3984134080 con 0x7f398413ae80 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f397c001be0 con 0x7f398413ae80 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f397c005050 con 0x7f398413ae80 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f397affd640 4 set_mon_vals no callback set 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.646+0000 7f39797fa640 10 monclient: dump: 2024-09-16T04:24:25.674 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f397c0102e0 con 0x7f398413ae80 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f398aaac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f39640065c0 con 0x7f3964002860 2024-09-16T04:24:25.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f397c047a40 con 0x7f398413ae80 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f3988848640 10 monclient: get_auth_request con 0x7f3970039190 auth_method 0 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103234353 0 0) 0x7f39640065c0 con 0x7f3964002860 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient(hunting): my global_id is 144240 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f398411dba0 con 0x7f3964002860 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2150495661 0 0) 0x7f398411dba0 con 0x7f3964002860 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f39640065c0 con 0x7f3964002860 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3974003790 con 0x7f3964002860 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 10 monclient: dump: 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3951229555 0 0) 0x7f39640065c0 con 0x7f3964002860 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.647+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f396400a150 con 0x7f3964002860 2024-09-16T04:24:25.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.648+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3974005a50 con 0x7f3964002860 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.648+0000 7f39797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.648+0000 7f39797fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.648+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.148:6789/0 conn(0x7f3964002860 legacy=0x7f3964004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.648+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.148:6789/0 conn(0x7f3964002860 legacy=0x7f3964004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f398aaac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3964009e20 con 0x7f39640090e0 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735428685 0 0) 0x7f3964009e20 con 0x7f39640090e0 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient(hunting): my global_id is 144243 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39640065c0 con 0x7f39640090e0 2024-09-16T04:24:25.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 482233838 0 0) 0x7f39640065c0 con 0x7f39640090e0 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3964009e20 con 0x7f39640090e0 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f39740051b0 con 0x7f39640090e0 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 10 monclient: dump: 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.679 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3973648934 0 0) 0x7f3964009e20 con 0x7f39640090e0 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.654+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f396400c390 con 0x7f39640090e0 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3979 (unknown 0 0 2524854973) 0x7f3974004fd0 con 0x7f39640090e0 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f39797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f39797fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.148:6789/0 conn(0x7f39640090e0 legacy=0x7f3964009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f39797fa640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.148:6789/0 conn(0x7f39640090e0 legacy=0x7f3964009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 70, 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 110355, 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:24:25.681 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.682 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.683 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:25.684 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.685 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:25.686 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:25.687 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:25.688 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:25.689 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:25.690 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:25.691 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:25.692 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.171:6812/7327273 conn(0x7f3970039190 legacy=0x7f397003b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 10 monclient: shutdown 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 >> v1:172.21.15.148:6790/0 conn(0x7f398413ae80 legacy=0x7f39841d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f3989049640 1 -- 172.21.15.148:0/2716835638 reap_dead start 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f3989049640 1 -- 172.21.15.148:0/2716835638 reap_dead start 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 shutdown_connections 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 >> 172.21.15.148:0/2716835638 conn(0x7f39840a5640 msgr2=0x7f39840a32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 shutdown_connections 2024-09-16T04:24:25.695 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.655+0000 7f398aaac640 1 -- 172.21.15.148:0/2716835638 wait complete. 2024-09-16T04:24:25.695 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:24:25.695 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:24:25.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:24:25.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.791+0000 7f26d710a640 10 monclient: get_monmap_and_config 2024-09-16T04:24:25.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.791+0000 7f26d710a640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.791+0000 7f26d710a640 10 monclient: monmap: 2024-09-16T04:24:25.794 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.794 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.794 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:25.792817+0000 2024-09-16T04:24:25.794 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:25.792817+0000 2024-09-16T04:24:25.795 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.795 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.795 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.795 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:25.795 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:25.795 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.792+0000 7f26d710a640 1 Processor -- start 2024-09-16T04:24:25.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.792+0000 7f26d710a640 1 -- start start 2024-09-16T04:24:25.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.792+0000 7f26d710a640 10 monclient: init 2024-09-16T04:24:25.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient(hunting): picked mon.noname-c con 0x7f26d009bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient(hunting): picked mon.noname-b con 0x7f26d009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient(hunting): picked mon.noname-a con 0x7f26d009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d00a1010 con 0x7f26d009cc30 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d009f030 con 0x7f26d009bb70 2024-09-16T04:24:25.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d009dc60 con 0x7f26d009c660 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient(hunting): authenticate will time out at 1886.418099s 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d56a7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f26d009cc30 0x7f26d013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46676/0 (socket says 172.21.15.148:46676) 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f26d009c660 0x7f26d0138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37608/0 (socket says 172.21.15.148:37608) 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d56a7640 1 -- 172.21.15.148:0/3675327355 learned_addr learned my addr 172.21.15.148:0/3675327355 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1017210023 0 0) 0x7f26d009dc60 con 0x7f26d009c660 2024-09-16T04:24:25.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): my global_id is 154139 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26c0003520 con 0x7f26d009c660 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2797700430 0 0) 0x7f26d00a1010 con 0x7f26d009cc30 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): my global_id is 144246 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d009dc60 con 0x7f26d009cc30 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 87594414 0 0) 0x7f26d009f030 con 0x7f26d009bb70 2024-09-16T04:24:25.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient(hunting): my global_id is 7213 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d00a1010 con 0x7f26d009bb70 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 455691674 0 0) 0x7f26d009dc60 con 0x7f26d009cc30 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 >> v1:172.21.15.148:6790/0 conn(0x7f26d009c660 legacy=0x7f26d0138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 >> v1:172.21.15.171:6789/0 conn(0x7f26d009bb70 legacy=0x7f26d009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 monclient: found mon.noname-a 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d009df40 con 0x7f26d009cc30 2024-09-16T04:24:25.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26c8001970 con 0x7f26d009cc30 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 5 monclient: authenticate success, global_id 144246 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: dump: 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.810 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26c8001be0 con 0x7f26d009cc30 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 1 -- 172.21.15.148:0/3675327355 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26c8001ff0 con 0x7f26d009cc30 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26d710a640 4 set_mon_vals no callback set 2024-09-16T04:24:25.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.793+0000 7f26cf7fe640 10 monclient: dump: 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.812 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 10 monclient: shutdown 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 1 -- 172.21.15.148:0/3675327355 >> v1:172.21.15.148:6789/0 conn(0x7f26d009cc30 legacy=0x7f26d013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 1 -- 172.21.15.148:0/3675327355 shutdown_connections 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 1 -- 172.21.15.148:0/3675327355 >> 172.21.15.148:0/3675327355 conn(0x7f26d00a5640 msgr2=0x7f26d00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 1 -- 172.21.15.148:0/3675327355 shutdown_connections 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 1 -- 172.21.15.148:0/3675327355 wait complete. 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.794+0000 7f26d710a640 10 monclient: monmap: 2024-09-16T04:24:25.813 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 1 Processor -- start 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 1 -- start start 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 10 monclient: init 2024-09-16T04:24:25.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 10 monclient(hunting): picked mon.noname-c con 0x7f26d009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 10 monclient(hunting): picked mon.noname-b con 0x7f26d009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 10 monclient(hunting): picked mon.noname-a con 0x7f26d009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d0125860 con 0x7f26d009cc30 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.795+0000 7f26d710a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d0125b40 con 0x7f26d009c660 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d01323b0 con 0x7f26d009bb70 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 10 monclient(hunting): authenticate will time out at 1886.421040s 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d56a7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f26d009cc30 0x7f26d01d4230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46678/0 (socket says 172.21.15.148:46678) 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d56a7640 1 -- 172.21.15.148:0/4107952468 learned_addr learned my addr 172.21.15.148:0/4107952468 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d4ea6640 1 --1- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6790/0 conn(0x7f26d009bb70 0x7f26d01cbac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37616/0 (socket says 172.21.15.148:37616) 2024-09-16T04:24:25.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006427430 0 0) 0x7f26d0125860 con 0x7f26d009cc30 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): my global_id is 144249 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26b4003520 con 0x7f26d009cc30 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718068853 0 0) 0x7f26d01323b0 con 0x7f26d009bb70 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): my global_id is 154142 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d0125860 con 0x7f26d009bb70 2024-09-16T04:24:25.816 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239752064 0 0) 0x7f26d0125b40 con 0x7f26d009c660 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): my global_id is 7216 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d01323b0 con 0x7f26d009c660 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1779003520 0 0) 0x7f26b4003520 con 0x7f26d009cc30 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26d0125b40 con 0x7f26d009cc30 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26c8001870 con 0x7f26d009cc30 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): dump: 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.818 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 877328680 0 0) 0x7f26d0125860 con 0x7f26d009bb70 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26b4003520 con 0x7f26d009bb70 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26c40018b0 con 0x7f26d009bb70 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): dump: 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.819 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2790036779 0 0) 0x7f26d01323b0 con 0x7f26d009c660 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26d0125860 con 0x7f26d009c660 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26bc001190 con 0x7f26d009c660 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient(hunting): dump: 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.820 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1937348981 0 0) 0x7f26d0125b40 con 0x7f26d009cc30 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6790/0 conn(0x7f26d009bb70 legacy=0x7f26d01cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.171:6789/0 conn(0x7f26d009c660 legacy=0x7f26d01d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.821 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 monclient: found mon.a 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d00a3f80 con 0x7f26d009cc30 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 5 monclient: authenticate success, global_id 144249 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 10 monclient: _renew_subs 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f26d0125580 con 0x7f26d009cc30 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 10 monclient: _renew_subs 2024-09-16T04:24:25.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f26d0122bc0 con 0x7f26d009cc30 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26c8001be0 con 0x7f26d009cc30 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26c8005110 con 0x7f26d009cc30 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26ceffd640 4 set_mon_vals no callback set 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.796+0000 7f26cd7fa640 10 monclient: dump: 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.823 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.797+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f26c8005110 con 0x7f26d009cc30 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.797+0000 7f26d710a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.797+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26b00065c0 con 0x7f26b0002860 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.797+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f26c8047ad0 con 0x7f26d009cc30 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26d4ea6640 10 monclient: get_auth_request con 0x7f26b4039120 auth_method 0 2024-09-16T04:24:25.824 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193001288 0 0) 0x7f26b00065c0 con 0x7f26b0002860 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient(hunting): my global_id is 154145 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26d0125b40 con 0x7f26b0002860 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1726944955 0 0) 0x7f26d0125b40 con 0x7f26b0002860 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26b00065c0 con 0x7f26b0002860 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26bc004980 con 0x7f26b0002860 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient: dump: 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.825 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4085385814 0 0) 0x7f26b00065c0 con 0x7f26b0002860 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26b000a150 con 0x7f26b0002860 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f26bc005ee0 con 0x7f26b0002860 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6790/0 conn(0x7f26b0002860 legacy=0x7f26b0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.826 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.798+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6790/0 conn(0x7f26b0002860 legacy=0x7f26b0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:25.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26d710a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:25.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26b0009e40 con 0x7f26b0009100 2024-09-16T04:24:25.828 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249396770 0 0) 0x7f26b0009e40 con 0x7f26b0009100 2024-09-16T04:24:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26cd7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26cd7fa640 10 monclient(hunting): my global_id is 154148 2024-09-16T04:24:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.804+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26b00065c0 con 0x7f26b0009100 2024-09-16T04:24:25.829 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1543368941 0 0) 0x7f26b00065c0 con 0x7f26b0009100 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26b0009e40 con 0x7f26b0009100 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26bc006520 con 0x7f26b0009100 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 10 monclient: dump: 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.830 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2743140304 0 0) 0x7f26b0009e40 con 0x7f26b0009100 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26b000c3d0 con 0x7f26b0009100 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3975 (unknown 0 0 2902154917) 0x7f26bc0058b0 con 0x7f26b0009100 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6790/0 conn(0x7f26b0009100 legacy=0x7f26b0009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26cd7fa640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6790/0 conn(0x7f26b0009100 legacy=0x7f26b0009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:25.831 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 70, 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 2712, 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:25.832 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:25.833 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.834 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:25.835 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.836 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.837 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:25.838 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.839 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:25.840 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:25.840 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:25.840 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:25.840 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:25.840 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.841 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:25.841 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:25.841 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:25.841 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:25.841 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.842 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:25.842 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:25.842 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:25.842 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.842 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:25.843 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.844 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.844 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.844 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:25.844 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:25.844 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.844 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:25.845 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:25.846 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:25.846 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:25.846 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:25.846 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.846 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.846 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:25.847 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.171:6812/7327273 conn(0x7f26b4039120 legacy=0x7f26b403b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d710a640 10 monclient: shutdown 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d710a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 >> v1:172.21.15.148:6789/0 conn(0x7f26d009cc30 legacy=0x7f26d01d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d56a7640 1 -- 172.21.15.148:0/4107952468 reap_dead start 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d56a7640 1 -- 172.21.15.148:0/4107952468 reap_dead start 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 shutdown_connections 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.805+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 >> 172.21.15.148:0/4107952468 conn(0x7f26d00a5640 msgr2=0x7f26d01325e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.806+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 shutdown_connections 2024-09-16T04:24:25.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.806+0000 7f26d710a640 1 -- 172.21.15.148:0/4107952468 wait complete. 2024-09-16T04:24:25.850 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:24:25.850 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:24:25.850 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:24:25.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.947+0000 7f9cd32c9640 10 monclient: get_monmap_and_config 2024-09-16T04:24:25.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.947+0000 7f9cd32c9640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.947+0000 7f9cd32c9640 10 monclient: monmap: 2024-09-16T04:24:25.949 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.949 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:25.948748+0000 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:25.948748+0000 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:25.950 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.948+0000 7f9cd32c9640 1 Processor -- start 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.948+0000 7f9cd32c9640 1 -- start start 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.948+0000 7f9cd32c9640 10 monclient: init 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.948+0000 7f9cd32c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.948+0000 7f9cd32c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.948+0000 7f9cd32c9640 10 monclient(hunting): picked mon.noname-b con 0x7f9ccc0a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 10 monclient(hunting): picked mon.noname-a con 0x7f9ccc09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 10 monclient(hunting): picked mon.noname-c con 0x7f9ccc09b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc09e040 con 0x7f9ccc09b170 2024-09-16T04:24:25.960 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc09d660 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc09c0d0 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 10 monclient(hunting): authenticate will time out at 1886.574059s 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd1065640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f9ccc0a0e90 0x7f9ccc09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37632/0 (socket says 172.21.15.148:37632) 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd1065640 1 -- 172.21.15.148:0/2165996778 learned_addr learned my addr 172.21.15.148:0/2165996778 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd0864640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9ccc09b170 0x7f9ccc1383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46688/0 (socket says 172.21.15.148:46688) 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030269152 0 0) 0x7f9ccc09c0d0 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): my global_id is 154151 2024-09-16T04:24:25.961 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cb4003520 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417803151 0 0) 0x7f9cc00013c0 con 0x7f9ccc09b170 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): my global_id is 144252 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ccc09c0d0 con 0x7f9ccc09b170 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3316305939 0 0) 0x7f9cbc001110 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient(hunting): my global_id is 7219 2024-09-16T04:24:25.962 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cc00013c0 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 968729362 0 0) 0x7f9ccc09c0d0 con 0x7f9ccc09b170 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 >> v1:172.21.15.148:6790/0 conn(0x7f9ccc0a0e90 legacy=0x7f9ccc09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 >> v1:172.21.15.171:6789/0 conn(0x7f9ccc09b5c0 legacy=0x7f9ccc13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 monclient: found mon.noname-a 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc09d380 con 0x7f9ccc09b170 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 5 monclient: authenticate success, global_id 144252 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cc0001a70 con 0x7f9ccc09b170 2024-09-16T04:24:25.963 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: dump: 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.964 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.965 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.965 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.965 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.965 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.965 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.965 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:25.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9cc0001ce0 con 0x7f9ccc09b170 2024-09-16T04:24:25.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 1 -- 172.21.15.148:0/2165996778 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cc0004b60 con 0x7f9ccc09b170 2024-09-16T04:24:25.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:25.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9cd32c9640 4 set_mon_vals no callback set 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.949+0000 7f9ccbfff640 10 monclient: dump: 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.967 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.968 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.968 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.968 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.968 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.968 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.968 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 10 monclient: shutdown 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 1 -- 172.21.15.148:0/2165996778 >> v1:172.21.15.148:6789/0 conn(0x7f9ccc09b170 legacy=0x7f9ccc1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 1 -- 172.21.15.148:0/2165996778 shutdown_connections 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 1 -- 172.21.15.148:0/2165996778 >> 172.21.15.148:0/2165996778 conn(0x7f9ccc0a5640 msgr2=0x7f9ccc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 1 -- 172.21.15.148:0/2165996778 shutdown_connections 2024-09-16T04:24:25.969 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 1 -- 172.21.15.148:0/2165996778 wait complete. 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 10 monclient: build_initial_monmap 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.950+0000 7f9cd32c9640 10 monclient: monmap: 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:25.970 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:25.971 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.971 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:25.971 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:25.971 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:25.971 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.971 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 1 Processor -- start 2024-09-16T04:24:25.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 1 -- start start 2024-09-16T04:24:25.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient: init 2024-09-16T04:24:25.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:25.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:24:25.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient(hunting): picked mon.noname-b con 0x7f9ccc09b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:25.972 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient(hunting): picked mon.noname-c con 0x7f9ccc09b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:25.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient(hunting): picked mon.noname-a con 0x7f9ccc0a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:25.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc1c5dd0 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc1aa560 con 0x7f9ccc09b170 2024-09-16T04:24:25.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc1ad020 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:25.973 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd32c9640 10 monclient(hunting): authenticate will time out at 1886.576829s 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd0864640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f9ccc09b5c0 0x7f9ccc1d09f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37636/0 (socket says 172.21.15.148:37636) 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd1866640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9ccc0a0e90 0x7f9ccc1d4120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46702/0 (socket says 172.21.15.148:46702) 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.951+0000 7f9cd0864640 1 -- 172.21.15.148:0/1579730528 learned_addr learned my addr 172.21.15.148:0/1579730528 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2198550418 0 0) 0x7f9ccc1ad020 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.974 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): my global_id is 154154 2024-09-16T04:24:25.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cb8003520 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2528825801 0 0) 0x7f9ccc1c5dd0 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.975 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): my global_id is 144255 2024-09-16T04:24:25.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ccc1ad020 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1515587178 0 0) 0x7f9cb8003520 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ccc1c5dd0 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cc0001a30 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.976 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:25.977 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): dump: 2024-09-16T04:24:25.977 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.977 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.977 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.977 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.977 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.978 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.978 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.978 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.978 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.978 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:25.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641430595 0 0) 0x7f9ccc1aa560 con 0x7f9ccc09b170 2024-09-16T04:24:25.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.979 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): my global_id is 7222 2024-09-16T04:24:25.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cb8003520 con 0x7f9ccc09b170 2024-09-16T04:24:25.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1979730776 0 0) 0x7f9ccc1ad020 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ccc1aa560 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cbc001a00 con 0x7f9ccc0a0e90 2024-09-16T04:24:25.980 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:25.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:25.981 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient(hunting): dump: 2024-09-16T04:24:25.981 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.981 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.981 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.981 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2839578431 0 0) 0x7f9ccc1c5dd0 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.982 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.171:6789/0 conn(0x7f9ccc09b170 legacy=0x7f9ccc1c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.148:6789/0 conn(0x7f9ccc0a0e90 legacy=0x7f9ccc1d4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 monclient: found mon.c 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc072cb0 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:24:25.983 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 5 monclient: authenticate success, global_id 154154 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 10 monclient: _renew_subs 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ccc1aad50 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9cc0005120 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.984 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cc00054f0 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cc9ffb640 10 monclient: dump: 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.985 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.986 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.986 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.986 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.986 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.986 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.986 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 10 monclient: _renew_subs 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9ccb7fe640 4 set_mon_vals no callback set 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.952+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ccc1abad0 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.953+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9cc0005690 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.953+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f9cc0048030 con 0x7f9ccc09b5c0 2024-09-16T04:24:25.987 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.953+0000 7f9cd32c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.953+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9ccc0008d0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.953+0000 7f9cd1065640 10 monclient: get_auth_request con 0x7f9cb80391d0 auth_method 0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155140049 0 0) 0x7f9ccc0008d0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient(hunting): my global_id is 7225 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ccc1c5dd0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 819186784 0 0) 0x7f9ccc1c5dd0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ccc0008d0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cbc003790 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.988 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient: dump: 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.989 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1368943190 0 0) 0x7f9ccc0008d0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ccc0a42c0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9cbc0036f0 con 0x7f9ccc02d7c0 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.171:6789/0 conn(0x7f9ccc02d7c0 legacy=0x7f9ccc068180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.990 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.954+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.171:6789/0 conn(0x7f9ccc02d7c0 legacy=0x7f9ccc068180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.960+0000 7f9cd32c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.960+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9ccc1a1670 con 0x7f9ccc1c8130 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377160798 0 0) 0x7f9ccc1a1670 con 0x7f9ccc1c8130 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient(hunting): my global_id is 7228 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ccc0008d0 con 0x7f9ccc1c8130 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 63187627 0 0) 0x7f9ccc0008d0 con 0x7f9ccc1c8130 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ccc1a1670 con 0x7f9ccc1c8130 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9cbc003ec0 con 0x7f9ccc1c8130 2024-09-16T04:24:25.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 10 monclient: dump: 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:25.993 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4203306578 0 0) 0x7f9ccc1a1670 con 0x7f9ccc1c8130 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.961+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9ccc02dba0 con 0x7f9ccc1c8130 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4312 (unknown 0 0 1345507323) 0x7f9cbc004260 con 0x7f9ccc1c8130 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cc9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cc9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.171:6789/0 conn(0x7f9ccc1c8130 legacy=0x7f9ccc1a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cc9ffb640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.171:6789/0 conn(0x7f9ccc1c8130 legacy=0x7f9ccc1a0f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:25.994 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 70, 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 1090000, 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.995 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:25.996 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:25.997 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:25.998 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:25.998 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:25.998 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:25.998 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:25.998 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:25.998 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:25.999 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:26.000 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:26.001 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:26.002 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:26.003 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.004 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.005 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:26.006 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 6 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:26.007 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:26.008 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:26.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.171:6812/7327273 conn(0x7f9cb80391d0 legacy=0x7f9cb803b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 10 monclient: shutdown 2024-09-16T04:24:26.009 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 >> v1:172.21.15.148:6790/0 conn(0x7f9ccc09b5c0 legacy=0x7f9ccc1d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd1866640 1 -- 172.21.15.148:0/1579730528 reap_dead start 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd1866640 1 -- 172.21.15.148:0/1579730528 reap_dead start 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 shutdown_connections 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 >> 172.21.15.148:0/1579730528 conn(0x7f9ccc0a5640 msgr2=0x7f9ccc0a35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 shutdown_connections 2024-09-16T04:24:26.010 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:25.962+0000 7f9cd32c9640 1 -- 172.21.15.148:0/1579730528 wait complete. 2024-09-16T04:24:26.010 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:24:26.011 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T04:24:26.011 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:24:26.011 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T04:24:26.011 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:24:26.011 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:24:26.011 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:24:26.034 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:24:26.035 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:24:26.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:24:26.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.131+0000 7fcfe1b5f640 10 monclient: get_monmap_and_config 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.131+0000 7fcfe1b5f640 10 monclient: build_initial_monmap 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.131+0000 7fcfe1b5f640 10 monclient: monmap: 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:26.132048+0000 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:26.132048+0000 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:26.133 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:26.134 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.131+0000 7fcfe1b5f640 1 Processor -- start 2024-09-16T04:24:26.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.131+0000 7fcfe1b5f640 1 -- start start 2024-09-16T04:24:26.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.131+0000 7fcfe1b5f640 10 monclient: init 2024-09-16T04:24:26.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:26.142 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfdc0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfdc0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfdc13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc0a0e00 con 0x7fcfdc13d060 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc0a10e0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc0a4970 con 0x7fcfdc0a2410 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfe1b5f640 10 monclient(hunting): authenticate will time out at 1886.757418s 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdb7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fcfdc0a2410 0x7fcfdc0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37674/0 (socket says 172.21.15.148:37674) 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdbfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcfdc13d060 0x7fcfdc13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46758/0 (socket says 172.21.15.148:46758) 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdb7fe640 1 -- 172.21.15.148:0/4055784464 learned_addr learned my addr 172.21.15.148:0/4055784464 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfd2ffd640 1 -- 172.21.15.148:0/4055784464 >> v1:172.21.15.171:6789/0 conn(0x7fcfdc0a1b10 legacy=0x7fcfdc13b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:26.143 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664180622 0 0) 0x7fcfdc0a4970 con 0x7fcfdc0a2410 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): my global_id is 154157 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfc8003520 con 0x7fcfdc0a2410 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404784067 0 0) 0x7fcfdc0a0e00 con 0x7fcfdc13d060 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 10 monclient(hunting): my global_id is 144258 2024-09-16T04:24:26.144 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfdc0a4970 con 0x7fcfdc13d060 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1330935653 0 0) 0x7fcfdc0a4970 con 0x7fcfdc13d060 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.132+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 >> v1:172.21.15.148:6790/0 conn(0x7fcfdc0a2410 legacy=0x7fcfdc0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 >> v1:172.21.15.171:6789/0 conn(0x7fcfdc0a1b10 legacy=0x7fcfdc13b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 1 monclient: found mon.noname-a 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc06dfd0 con 0x7fcfdc13d060 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: _finish_auth 0 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcfd40018f0 con 0x7fcfdc13d060 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 5 monclient: authenticate success, global_id 144258 2024-09-16T04:24:26.145 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: dump: 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:26.146 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfd4001bd0 con 0x7fcfdc13d060 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 1 -- 172.21.15.148:0/4055784464 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcfd4001fe0 con 0x7fcfdc13d060 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 4 set_mon_vals no callback set 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfdaffd640 10 monclient: dump: 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:26.147 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 10 monclient: shutdown 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/4055784464 >> v1:172.21.15.148:6789/0 conn(0x7fcfdc13d060 legacy=0x7fcfdc13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/4055784464 shutdown_connections 2024-09-16T04:24:26.148 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/4055784464 >> 172.21.15.148:0/4055784464 conn(0x7fcfdc0a97b0 msgr2=0x7fcfdc0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/4055784464 shutdown_connections 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.133+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/4055784464 wait complete. 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.134+0000 7fcfe1b5f640 10 monclient: build_initial_monmap 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.134+0000 7fcfe1b5f640 10 monclient: monmap: 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.149 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.134+0000 7fcfe1b5f640 1 Processor -- start 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.134+0000 7fcfe1b5f640 1 -- start start 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.134+0000 7fcfe1b5f640 10 monclient: init 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient(hunting): picked mon.noname-c con 0x7fcfdc0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient(hunting): picked mon.noname-a con 0x7fcfdc0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient(hunting): picked mon.noname-b con 0x7fcfdc13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc13ba70 con 0x7fcfdc0a2410 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc137e10 con 0x7fcfdc13d060 2024-09-16T04:24:26.150 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfdc1404d0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient(hunting): authenticate will time out at 1886.760240s 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfdb7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fcfdc0a1b10 0x7fcfdc09ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37688/0 (socket says 172.21.15.148:37688) 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd2ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcfdc0a2410 0x7fcfdc09d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46766/0 (socket says 172.21.15.148:46766) 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfdb7fe640 1 -- 172.21.15.148:0/2926314973 learned_addr learned my addr 172.21.15.148:0/2926314973 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfdbfff640 1 -- 172.21.15.148:0/2926314973 >> v1:172.21.15.171:6789/0 conn(0x7fcfdc13d060 legacy=0x7fcfdc1d3b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031900151 0 0) 0x7fcfdc1404d0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:24:26.151 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:26.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:26.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): my global_id is 154160 2024-09-16T04:24:26.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfbc003550 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513501318 0 0) 0x7fcfdc13ba70 con 0x7fcfdc0a2410 2024-09-16T04:24:26.152 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): my global_id is 144261 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfdc1404d0 con 0x7fcfdc0a2410 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 553861273 0 0) 0x7fcfbc003550 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfdc13ba70 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcfcc001a30 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.153 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): dump: 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:26.154 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2924845958 0 0) 0x7fcfdc1404d0 con 0x7fcfdc0a2410 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcfbc003550 con 0x7fcfdc0a2410 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcfc4004070 con 0x7fcfdc0a2410 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient(hunting): dump: 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.155 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3542650046 0 0) 0x7fcfdc13ba70 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 >> v1:172.21.15.171:6789/0 conn(0x7fcfdc13d060 legacy=0x7fcfdc1d3b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 >> v1:172.21.15.148:6789/0 conn(0x7fcfdc0a2410 legacy=0x7fcfdc09d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 monclient: found mon.c 2024-09-16T04:24:26.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc0a0130 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 5 monclient: authenticate success, global_id 154160 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient: _renew_subs 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc09e840 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient: _renew_subs 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.135+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcfdc09f120 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.157 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcfcc001ce0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcfcc004c30 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfda7fc640 4 set_mon_vals no callback set 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 10 monclient: dump: 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:26.158 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcfcc0100e0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.136+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7fcfcc047b70 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.137+0000 7fcfe1b5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.137+0000 7fcfe1b5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.137+0000 7fcfe1b5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.137+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfc0005080 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.159 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.137+0000 7fcfd2ffd640 10 monclient: get_auth_request con 0x7fcfbc038eb0 auth_method 0 2024-09-16T04:24:26.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.141+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fcfcc015dd0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.141+0000 7fcfd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:26.160 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.141+0000 7fcfd8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:26.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfe1b5f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:24:26.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfe1b5f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:26.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfe1b5f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:26.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcfc0005370 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfd8ff9640 1 -- 172.21.15.148:0/2926314973 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 4074510744) 0x7fcfcc0156d0 con 0x7fcfdc0a1b10 2024-09-16T04:24:26.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfd8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:26.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.373+0000 7fcfd8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:26.377 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:24:26.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 >> v1:172.21.15.171:6812/7327273 conn(0x7fcfbc038eb0 legacy=0x7fcfbc03b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 10 monclient: shutdown 2024-09-16T04:24:26.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:26.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 >> v1:172.21.15.148:6790/0 conn(0x7fcfdc0a1b10 legacy=0x7fcfdc09ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:26.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 shutdown_connections 2024-09-16T04:24:26.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 >> 172.21.15.148:0/2926314973 conn(0x7fcfdc0a97b0 msgr2=0x7fcfdc135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:26.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 shutdown_connections 2024-09-16T04:24:26.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:26.375+0000 7fcfe1b5f640 1 -- 172.21.15.148:0/2926314973 wait complete. 2024-09-16T04:24:29.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:24:29.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.499+0000 7f65dbf4b640 10 monclient: get_monmap_and_config 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.499+0000 7f65dbf4b640 10 monclient: build_initial_monmap 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.499+0000 7f65dbf4b640 10 monclient: monmap: 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:29.500273+0000 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:29.500273+0000 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:29.501 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:29.502 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:29.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.499+0000 7f65dbf4b640 1 Processor -- start 2024-09-16T04:24:29.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 1 -- start start 2024-09-16T04:24:29.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient: init 2024-09-16T04:24:29.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient(hunting): picked mon.noname-a con 0x7f65d40a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient(hunting): picked mon.noname-b con 0x7f65d409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient(hunting): picked mon.noname-c con 0x7f65d409b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d409e040 con 0x7f65d40a0e90 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d409d660 con 0x7f65d409b5c0 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d409c0d0 con 0x7f65d409b170 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:29.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65dbf4b640 10 monclient(hunting): authenticate will time out at 1890.125677s 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65d94e6640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f65d409b170 0x7f65d41383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:37690/0 (socket says 172.21.15.148:37690) 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65d94e6640 1 -- 172.21.15.148:0/3976496470 learned_addr learned my addr 172.21.15.148:0/3976496470 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65d9ce7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f65d40a0e90 0x7f65d409aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46782/0 (socket says 172.21.15.148:46782) 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.500+0000 7f65da4e8640 1 -- 172.21.15.148:0/3976496470 >> v1:172.21.15.171:6789/0 conn(0x7f65d409b5c0 legacy=0x7f65d413bb20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150713646 0 0) 0x7f65cc001110 con 0x7f65d40a0e90 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient(hunting): _negotiate 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient(hunting): my global_id is 144264 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65bc003520 con 0x7f65d40a0e90 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3327800355 0 0) 0x7f65bc003520 con 0x7f65d40a0e90 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 >> v1:172.21.15.148:6790/0 conn(0x7f65d409b170 legacy=0x7f65d41383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:29.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 >> v1:172.21.15.171:6789/0 conn(0x7f65d409b5c0 legacy=0x7f65d413bb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: _finish_hunting 0 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 monclient: found mon.noname-a 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d409d380 con 0x7f65d40a0e90 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: _finish_auth 0 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: _check_auth_tickets 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f65cc0017e0 con 0x7f65d40a0e90 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 5 monclient: authenticate success, global_id 144264 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:29.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: dump: 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:29.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65cc001ad0 con 0x7f65d40a0e90 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 1 -- 172.21.15.148:0/3976496470 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f65cc0047e0 con 0x7f65d40a0e90 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 4 set_mon_vals no callback set 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65d8ce5640 10 monclient: dump: 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:29.513 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 10 monclient: shutdown 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 1 -- 172.21.15.148:0/3976496470 >> v1:172.21.15.148:6789/0 conn(0x7f65d40a0e90 legacy=0x7f65d409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.501+0000 7f65dbf4b640 1 -- 172.21.15.148:0/3976496470 shutdown_connections 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.502+0000 7f65dbf4b640 1 -- 172.21.15.148:0/3976496470 >> 172.21.15.148:0/3976496470 conn(0x7f65d40a5640 msgr2=0x7f65d40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.502+0000 7f65dbf4b640 1 -- 172.21.15.148:0/3976496470 shutdown_connections 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.502+0000 7f65dbf4b640 1 -- 172.21.15.148:0/3976496470 wait complete. 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.502+0000 7f65dbf4b640 10 monclient: build_initial_monmap 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.502+0000 7f65dbf4b640 10 monclient: monmap: 2024-09-16T04:24:29.514 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:29.515 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.502+0000 7f65dbf4b640 1 Processor -- start 2024-09-16T04:24:29.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 1 -- start start 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient: init 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient(hunting): picked mon.noname-a con 0x7f65d409b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient(hunting): picked mon.noname-c con 0x7f65d409b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient(hunting): picked mon.noname-b con 0x7f65d40a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d4126600 con 0x7f65d409b170 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d41268e0 con 0x7f65d40a0e90 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d40a4430 con 0x7f65d409b5c0 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65dbf4b640 10 monclient(hunting): authenticate will time out at 1890.128513s 2024-09-16T04:24:29.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d9ce7640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f65d409b170 0x7f65d411f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:46788/0 (socket says 172.21.15.148:46788) 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d9ce7640 1 -- 172.21.15.148:0/2432078984 learned_addr learned my addr 172.21.15.148:0/2432078984 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65da4e8640 1 -- 172.21.15.148:0/2432078984 >> v1:172.21.15.171:6789/0 conn(0x7f65d40a0e90 legacy=0x7f65d41d4140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3769108536 0 0) 0x7f65d4126600 con 0x7f65d409b170 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): my global_id is 144267 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65c0003520 con 0x7f65d409b170 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3986076543 0 0) 0x7f65c0003520 con 0x7f65d409b170 2024-09-16T04:24:29.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65d4126600 con 0x7f65d409b170 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f65cc000fe0 con 0x7f65d409b170 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.503+0000 7f65d27fc640 10 monclient(hunting): dump: 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:29.519 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1495695240 0 0) 0x7f65d4126600 con 0x7f65d409b170 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 >> v1:172.21.15.148:6790/0 conn(0x7f65d409b5c0 legacy=0x7f65d411fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 >> v1:172.21.15.171:6789/0 conn(0x7f65d40a0e90 legacy=0x7f65d41d4140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 monclient: found mon.a 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:29.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d4072ca0 con 0x7f65d409b170 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: _finish_auth 0 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 5 monclient: authenticate success, global_id 144267 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 10 monclient: _renew_subs 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65d40740a0 con 0x7f65d409b170 2024-09-16T04:24:29.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 10 monclient: _renew_subs 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65d4120420 con 0x7f65d409b170 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65cc004d50 con 0x7f65d409b170 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f65cc005120 con 0x7f65d409b170 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 10 monclient: dump: 2024-09-16T04:24:29.522 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d3fff640 4 set_mon_vals no callback set 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.504+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f65cc0105d0 con 0x7f65d409b170 2024-09-16T04:24:29.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.505+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f65cc047c30 con 0x7f65d409b170 2024-09-16T04:24:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.505+0000 7f65dbf4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.505+0000 7f65dbf4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.505+0000 7f65dbf4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.505+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b4005080 con 0x7f65d409b170 2024-09-16T04:24:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.505+0000 7f65d94e6640 10 monclient: get_auth_request con 0x7f65c0038ec0 auth_method 0 2024-09-16T04:24:29.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.510+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f65d409e040 con 0x7f65d409b170 2024-09-16T04:24:29.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.510+0000 7f65d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:29.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.510+0000 7f65d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:29.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65dbf4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:24:29.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65dbf4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:29.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65dbf4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:24:29.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65b4005370 con 0x7f65d409b170 2024-09-16T04:24:29.743 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65d27fc640 1 -- 172.21.15.148:0/2432078984 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 1413187338) 0x7f65cc015ef0 con 0x7f65d409b170 2024-09-16T04:24:29.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:29.744 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.740+0000 7f65d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:29.744 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:24:29.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.742+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 >> v1:172.21.15.171:6812/7327273 conn(0x7f65c0038ec0 legacy=0x7f65c003b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:29.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.742+0000 7f65dbf4b640 10 monclient: shutdown 2024-09-16T04:24:29.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.742+0000 7f65dbf4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:29.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.742+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 >> v1:172.21.15.148:6789/0 conn(0x7f65d409b170 legacy=0x7f65d411f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:29.746 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.742+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 shutdown_connections 2024-09-16T04:24:29.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.742+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 >> 172.21.15.148:0/2432078984 conn(0x7f65d40a5640 msgr2=0x7f65d40a2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:29.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.743+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 shutdown_connections 2024-09-16T04:24:29.747 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:29.743+0000 7f65dbf4b640 1 -- 172.21.15.148:0/2432078984 wait complete. 2024-09-16T04:24:32.756 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.863+0000 7f933592c640 10 monclient: get_monmap_and_config 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.863+0000 7f933592c640 10 monclient: build_initial_monmap 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.863+0000 7f933592c640 10 monclient: monmap: 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:32.864551+0000 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:32.864551+0000 2024-09-16T04:24:32.865 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:32.866 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:32.866 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:32.866 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:32.866 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:32.866 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:32.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 1 Processor -- start 2024-09-16T04:24:32.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 1 -- start start 2024-09-16T04:24:32.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient: init 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient(hunting): picked mon.noname-c con 0x7f9330134b40 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient(hunting): picked mon.noname-a con 0x7f9330134610 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient(hunting): picked mon.noname-b con 0x7f933013b050 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93300a1910 con 0x7f9330134610 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330099690 con 0x7f9330134b40 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330099970 con 0x7f933013b050 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f933592c640 10 monclient(hunting): authenticate will time out at 1893.489766s 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f932effd640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7f9330134b40 legacy=0x7f9330134f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f932e7fc640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9330134610 0x7f9330139930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53718/0 (socket says 172.21.15.148:53718) 2024-09-16T04:24:32.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f932f7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f933013b050 0x7f933013d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59400/0 (socket says 172.21.15.148:59400) 2024-09-16T04:24:32.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:32.864+0000 7f932e7fc640 1 -- 172.21.15.148:0/4140485508 learned_addr learned my addr 172.21.15.148:0/4140485508 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:33.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:33.065+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f9330134b40 legacy=0x7f9330134f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:33.467 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:33.466+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f9330134b40 legacy=0x7f9330134f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:34.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:34.267+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f9330134b40 legacy=0x7f9330134f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:35.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient(hunting): tick 2024-09-16T04:24:35.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.148:6790/0 conn(0x7f933013b050 legacy=0x7f933013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f9330134b40 legacy=0x7f9330134f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.148:6789/0 conn(0x7f9330134610 legacy=0x7f9330139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f931c0036c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f931c007380 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f933013b050 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:35.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933009f160 con 0x7f931c007380 2024-09-16T04:24:35.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.864+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330099690 con 0x7f931c0036c0 2024-09-16T04:24:35.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.865+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931c0108d0 con 0x7f933013b050 2024-09-16T04:24:35.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.865+0000 7f932cff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:35.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:35.865+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c0036c0 legacy=0x7f931c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:36.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:36.066+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c0036c0 legacy=0x7f931c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:36.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:36.467+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c0036c0 legacy=0x7f931c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:37.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:37.268+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c0036c0 legacy=0x7f931c005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:38.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): tick 2024-09-16T04:24:38.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:24:38.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:24:38.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.148:6790/0 conn(0x7f933013b050 legacy=0x7f931c00d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c0036c0 legacy=0x7f931c005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.148:6789/0 conn(0x7f931c007380 legacy=0x7f931c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f931c010bd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f931c003ae0 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f933013b050 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931c011b20 con 0x7f931c003ae0 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330099690 con 0x7f931c010bd0 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931c005ed0 con 0x7f933013b050 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932cff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:38.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:38.866+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f931c010fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:39.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:39.067+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f931c010fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:39.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:39.468+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f931c010fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.268+0000 7f932effd640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f931c010fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.336+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148167220 0 0) 0x7f931c005ed0 con 0x7f933013b050 2024-09-16T04:24:40.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.336+0000 7f932dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.336+0000 7f932dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.336+0000 7f932dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.336+0000 7f932dffb640 10 monclient(hunting): my global_id is 154163 2024-09-16T04:24:40.339 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.336+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9318003520 con 0x7f933013b050 2024-09-16T04:24:40.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.338+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571904485 0 0) 0x7f931c011b20 con 0x7f931c003ae0 2024-09-16T04:24:40.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.338+0000 7f932dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.338+0000 7f932dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.338+0000 7f932dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.338+0000 7f932dffb640 10 monclient(hunting): my global_id is 144270 2024-09-16T04:24:40.341 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.338+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931c005ed0 con 0x7f931c003ae0 2024-09-16T04:24:40.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1084838209 0 0) 0x7f931c005ed0 con 0x7f931c003ae0 2024-09-16T04:24:40.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.148:6790/0 conn(0x7f933013b050 legacy=0x7f931c00d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f931c010fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:40.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:24:40.342 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 1 monclient: found mon.noname-a 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931c009820 con 0x7f931c003ae0 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: _finish_auth 0 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9324001590 con 0x7f931c003ae0 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f932dffb640 10 monclient: dump: 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.343 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f933592c640 5 monclient: authenticate success, global_id 144270 2024-09-16T04:24:40.344 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.339+0000 7f933592c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93240017f0 con 0x7f931c003ae0 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f932dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f932dffb640 1 -- 172.21.15.148:0/4140485508 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9324001c00 con 0x7f931c003ae0 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f933592c640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f933592c640 4 set_mon_vals no callback set 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f932dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f932dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.341+0000 7f932dffb640 10 monclient: dump: 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 10 monclient: shutdown 2024-09-16T04:24:40.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:40.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 1 -- 172.21.15.148:0/4140485508 >> v1:172.21.15.148:6789/0 conn(0x7f931c003ae0 legacy=0x7f931c011dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 1 -- 172.21.15.148:0/4140485508 shutdown_connections 2024-09-16T04:24:40.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 1 -- 172.21.15.148:0/4140485508 >> 172.21.15.148:0/4140485508 conn(0x7f93300a5860 msgr2=0x7f93300a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:40.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 1 -- 172.21.15.148:0/4140485508 shutdown_connections 2024-09-16T04:24:40.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.342+0000 7f933592c640 1 -- 172.21.15.148:0/4140485508 wait complete. 2024-09-16T04:24:40.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.343+0000 7f933592c640 10 monclient: build_initial_monmap 2024-09-16T04:24:40.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.343+0000 7f933592c640 10 monclient: monmap: 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.343+0000 7f933592c640 1 Processor -- start 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 1 -- start start 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient: init 2024-09-16T04:24:40.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient(hunting): picked mon.noname-c con 0x7f931c003ae0 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient(hunting): picked mon.noname-b con 0x7f931c010bd0 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient(hunting): picked mon.noname-a con 0x7f933013b050 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933012c200 con 0x7f933013b050 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933012c4e0 con 0x7f931c010bd0 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9330139a90 con 0x7f931c003ae0 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f933592c640 10 monclient(hunting): authenticate will time out at 1900.970790s 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f932f7fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f933013b050 0x7f93301d2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53740/0 (socket says 172.21.15.148:53740) 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f932effd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f931c003ae0 0x7f9330138c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59428/0 (socket says 172.21.15.148:59428) 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f932e7fc640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f9330139380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.344+0000 7f932f7fe640 1 -- 172.21.15.148:0/2157248872 learned_addr learned my addr 172.21.15.148:0/2157248872 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978476920 0 0) 0x7f9330139a90 con 0x7f931c003ae0 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): my global_id is 154166 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931c011b20 con 0x7f931c003ae0 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367324210 0 0) 0x7f933012c200 con 0x7f933013b050 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): my global_id is 144273 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9330139a90 con 0x7f933013b050 2024-09-16T04:24:40.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1060736578 0 0) 0x7f931c011b20 con 0x7f931c003ae0 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f933012c200 con 0x7f931c003ae0 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9320004330 con 0x7f931c003ae0 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): dump: 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.354 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 823747603 0 0) 0x7f9330139a90 con 0x7f933013b050 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f931c011b20 con 0x7f933013b050 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2906355229 0 0) 0x7f933012c200 con 0x7f931c003ae0 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 >> v1:172.21.15.171:6789/0 conn(0x7f931c010bd0 legacy=0x7f9330139380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 >> v1:172.21.15.148:6789/0 conn(0x7f933013b050 legacy=0x7f93301d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: _finish_hunting 0 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 monclient: found mon.c 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933009c630 con 0x7f931c003ae0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: _finish_auth 0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: _check_auth_tickets 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 5 monclient: authenticate success, global_id 154166 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 10 monclient: _renew_subs 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f933012c7c0 con 0x7f931c003ae0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 10 monclient: _renew_subs 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f933012b1e0 con 0x7f931c003ae0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93200048c0 con 0x7f931c003ae0 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:40.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9320004c10 con 0x7f931c003ae0 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f932d7fa640 4 set_mon_vals no callback set 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.345+0000 7f9317fff640 10 monclient: dump: 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.357 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.346+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9320004db0 con 0x7f931c003ae0 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.346+0000 7f933592c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.346+0000 7f933592c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.346+0000 7f933592c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.347+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9310005080 con 0x7f931c003ae0 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.347+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f93200479c0 con 0x7f931c003ae0 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.347+0000 7f932e7fc640 10 monclient: get_auth_request con 0x7f931c03a980 auth_method 0 2024-09-16T04:24:40.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.351+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9320015c20 con 0x7f931c003ae0 2024-09-16T04:24:40.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.351+0000 7f9317fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.351+0000 7f9317fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f933592c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f933592c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f933592c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9310005370 con 0x7f931c003ae0 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f9317fff640 1 -- 172.21.15.148:0/2157248872 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1252251582) 0x7f9320015520 con 0x7f931c003ae0 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f9317fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.589+0000 7f9317fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:40.592 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":74,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 >> v1:172.21.15.171:6812/7327273 conn(0x7f931c03a980 legacy=0x7f931c03ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 10 monclient: shutdown 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 >> v1:172.21.15.148:6790/0 conn(0x7f931c003ae0 legacy=0x7f9330138c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 shutdown_connections 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 >> 172.21.15.148:0/2157248872 conn(0x7f93300a5860 msgr2=0x7f93300a0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:40.594 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 shutdown_connections 2024-09-16T04:24:40.595 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.591+0000 7f933592c640 1 -- 172.21.15.148:0/2157248872 wait complete. 2024-09-16T04:24:40.605 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:24:40.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:24:40.605 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:24:40.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.703+0000 7efedf8b7640 10 monclient: get_monmap_and_config 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.703+0000 7efedf8b7640 10 monclient: build_initial_monmap 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.703+0000 7efedf8b7640 10 monclient: monmap: 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:40.705070+0000 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:40.705070+0000 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:40.706 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.703+0000 7efedf8b7640 1 Processor -- start 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.703+0000 7efedf8b7640 1 -- start start 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.703+0000 7efedf8b7640 10 monclient: init 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient(hunting): picked mon.noname-c con 0x7efed80a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient(hunting): picked mon.noname-a con 0x7efed80a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient(hunting): picked mon.noname-b con 0x7efed813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed80a0e00 con 0x7efed80a1b10 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed80a10e0 con 0x7efed80a2410 2024-09-16T04:24:40.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed80a4970 con 0x7efed813d060 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedf8b7640 10 monclient(hunting): authenticate will time out at 1901.330507s 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedd653640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7efed80a2410 legacy=0x7efed80a27f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedde54640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efed813d060 0x7efed813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59436/0 (socket says 172.21.15.148:59436) 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedde54640 1 -- 172.21.15.148:0/3426850695 learned_addr learned my addr 172.21.15.148:0/3426850695 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efedce52640 1 --1- 172.21.15.148:0/3426850695 >> v1:172.21.15.148:6789/0 conn(0x7efed80a1b10 0x7efed813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53750/0 (socket says 172.21.15.148:53750) 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384422391 0 0) 0x7efed80a0e00 con 0x7efed80a1b10 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): my global_id is 144276 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efec4003520 con 0x7efed80a1b10 2024-09-16T04:24:40.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994152388 0 0) 0x7efed80a4970 con 0x7efed813d060 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 10 monclient(hunting): my global_id is 154169 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.704+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed80a0e00 con 0x7efed813d060 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 852552067 0 0) 0x7efec4003520 con 0x7efed80a1b10 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 >> v1:172.21.15.148:6790/0 conn(0x7efed813d060 legacy=0x7efed813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 >> v1:172.21.15.171:6789/0 conn(0x7efed80a2410 legacy=0x7efed80a27f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: _finish_hunting 0 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 monclient: found mon.noname-a 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:24:40.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed806dfd0 con 0x7efed80a1b10 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: _finish_auth 0 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: _check_auth_tickets 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efec8001610 con 0x7efed80a1b10 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 5 monclient: authenticate success, global_id 144276 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: dump: 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.719 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efec80018b0 con 0x7efed80a1b10 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 1 -- 172.21.15.148:0/3426850695 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efec8001cc0 con 0x7efed80a1b10 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 4 set_mon_vals no callback set 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efecffff640 10 monclient: dump: 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 10 monclient: shutdown 2024-09-16T04:24:40.721 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 1 -- 172.21.15.148:0/3426850695 >> v1:172.21.15.148:6789/0 conn(0x7efed80a1b10 legacy=0x7efed813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 1 -- 172.21.15.148:0/3426850695 shutdown_connections 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 1 -- 172.21.15.148:0/3426850695 >> 172.21.15.148:0/3426850695 conn(0x7efed80a97b0 msgr2=0x7efed80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 1 -- 172.21.15.148:0/3426850695 shutdown_connections 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.705+0000 7efedf8b7640 1 -- 172.21.15.148:0/3426850695 wait complete. 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.706+0000 7efedf8b7640 10 monclient: build_initial_monmap 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.706+0000 7efedf8b7640 10 monclient: monmap: 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:40.722 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.706+0000 7efedf8b7640 1 Processor -- start 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.706+0000 7efedf8b7640 1 -- start start 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.706+0000 7efedf8b7640 10 monclient: init 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient(hunting): picked mon.noname-c con 0x7efed80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient(hunting): picked mon.noname-b con 0x7efed80a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient(hunting): picked mon.noname-a con 0x7efed813d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:40.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed809e530 con 0x7efed813d060 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed81372c0 con 0x7efed80a2410 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed80a0130 con 0x7efed80a1b10 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedf8b7640 10 monclient(hunting): authenticate will time out at 1901.333364s 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedce52640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7efed80a2410 legacy=0x7efed81d0310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedd653640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efed80a1b10 0x7efed813c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59448/0 (socket says 172.21.15.148:59448) 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efedd653640 1 -- 172.21.15.148:0/1046527317 learned_addr learned my addr 172.21.15.148:0/1046527317 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257440055 0 0) 0x7efed80a0130 con 0x7efed80a1b10 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): my global_id is 154172 2024-09-16T04:24:40.724 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efebc003520 con 0x7efed80a1b10 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1629203149 0 0) 0x7efed809e530 con 0x7efed813d060 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): my global_id is 144279 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed80a0130 con 0x7efed813d060 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1638382110 0 0) 0x7efebc003520 con 0x7efed80a1b10 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efed809e530 con 0x7efed80a1b10 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efed40041d0 con 0x7efed80a1b10 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): dump: 2024-09-16T04:24:40.725 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 915684788 0 0) 0x7efed80a0130 con 0x7efed813d060 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efebc003520 con 0x7efed813d060 2024-09-16T04:24:40.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efed0001b20 con 0x7efed813d060 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 10 monclient(hunting): dump: 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.727 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2355663848 0 0) 0x7efed809e530 con 0x7efed80a1b10 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.707+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.171:6789/0 conn(0x7efed80a2410 legacy=0x7efed81d0310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.148:6789/0 conn(0x7efed813d060 legacy=0x7efed81d3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: _finish_hunting 0 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 1 monclient: found mon.c 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed8137540 con 0x7efed80a1b10 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: _finish_auth 0 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 5 monclient: authenticate success, global_id 154172 2024-09-16T04:24:40.728 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 10 monclient: _renew_subs 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efed8137820 con 0x7efed80a1b10 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efed4004480 con 0x7efed80a1b10 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efed4004850 con 0x7efed80a1b10 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 10 monclient: dump: 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.729 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecf7fe640 4 set_mon_vals no callback set 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 10 monclient: _renew_subs 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efed809f2d0 con 0x7efed80a1b10 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7efed40049f0 con 0x7efed80a1b10 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.708+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efed80008d0 con 0x7efed81cb220 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7efed4047a60 con 0x7efed80a1b10 2024-09-16T04:24:40.730 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087936705 0 0) 0x7efed80008d0 con 0x7efed81cb220 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient(hunting): my global_id is 144282 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed809e530 con 0x7efed81cb220 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efedde54640 10 monclient: get_auth_request con 0x7efebc038fb0 auth_method 0 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2821989569 0 0) 0x7efed809e530 con 0x7efed81cb220 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efed80008d0 con 0x7efed81cb220 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efec80019e0 con 0x7efed81cb220 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient: dump: 2024-09-16T04:24:40.731 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2451825706 0 0) 0x7efed80008d0 con 0x7efed81cb220 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efed81cb600 con 0x7efed81cb220 2024-09-16T04:24:40.732 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efec8001840 con 0x7efed81cb220 2024-09-16T04:24:40.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:40.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.148:6789/0 conn(0x7efed81cb220 legacy=0x7efed813d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.733 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.709+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.148:6789/0 conn(0x7efed81cb220 legacy=0x7efed813d820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:40.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.715+0000 7efedf8b7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:40.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.715+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efed802da30 con 0x7efed802d650 2024-09-16T04:24:40.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2798313997 0 0) 0x7efed802da30 con 0x7efed802d650 2024-09-16T04:24:40.734 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient(hunting): my global_id is 144285 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed80008d0 con 0x7efed802d650 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4028602714 0 0) 0x7efed80008d0 con 0x7efed802d650 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efed802da30 con 0x7efed802d650 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efec8004c10 con 0x7efed802d650 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient: dump: 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.735 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3002499115 0 0) 0x7efed802da30 con 0x7efed802d650 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efed813e640 con 0x7efed802d650 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4302 (unknown 0 0 1110965686) 0x7efec80060e0 con 0x7efed802d650 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:40.736 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.148:6789/0 conn(0x7efed802d650 legacy=0x7efed813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.716+0000 7efecdffb640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.148:6789/0 conn(0x7efed802d650 legacy=0x7efed813df30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 74, 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 125416, 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:40.737 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:40.738 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:40.739 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:40.740 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:40.741 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.742 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:40.743 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:40.744 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:40.745 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:40.746 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:24:40.747 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.748 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:40.749 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:40.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.171:6812/7327273 conn(0x7efebc038fb0 legacy=0x7efebc03b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 10 monclient: shutdown 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 >> v1:172.21.15.148:6790/0 conn(0x7efed80a1b10 legacy=0x7efed813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedde54640 1 -- 172.21.15.148:0/1046527317 reap_dead start 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedde54640 1 -- 172.21.15.148:0/1046527317 reap_dead start 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 shutdown_connections 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 >> 172.21.15.148:0/1046527317 conn(0x7efed80a97b0 msgr2=0x7efed806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 shutdown_connections 2024-09-16T04:24:40.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.717+0000 7efedf8b7640 1 -- 172.21.15.148:0/1046527317 wait complete. 2024-09-16T04:24:40.751 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:24:40.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:24:40.752 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.848+0000 7f67745e8640 10 monclient: get_monmap_and_config 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.848+0000 7f67745e8640 10 monclient: build_initial_monmap 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.848+0000 7f67745e8640 10 monclient: monmap: 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:24:40.850151+0000 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:24:40.850151+0000 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.851 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:24:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:24:40.852 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.848+0000 7f67745e8640 1 Processor -- start 2024-09-16T04:24:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.848+0000 7f67745e8640 1 -- start start 2024-09-16T04:24:40.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.848+0000 7f67745e8640 10 monclient: init 2024-09-16T04:24:40.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:40.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:24:40.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient(hunting): picked mon.noname-a con 0x7f676c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient(hunting): picked mon.noname-c con 0x7f676c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient(hunting): picked mon.noname-b con 0x7f676c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676c0a0e00 con 0x7f676c0a2410 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676c0a10e0 con 0x7f676c0a1b10 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676c0a4970 con 0x7f676c13d060 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f67745e8640 10 monclient(hunting): authenticate will time out at 1901.475524s 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6772384640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f676c0a2410 0x7f676c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:53772/0 (socket says 172.21.15.148:53772) 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771b83640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7f676c0a1b10 legacy=0x7f676c13b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6772b85640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f676c13d060 0x7f676c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59454/0 (socket says 172.21.15.148:59454) 2024-09-16T04:24:40.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6772384640 1 -- 172.21.15.148:0/3502682675 learned_addr learned my addr 172.21.15.148:0/3502682675 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750623339 0 0) 0x7f676c0a4970 con 0x7f676c13d060 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): my global_id is 154175 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6758003520 con 0x7f676c13d060 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2889967769 0 0) 0x7f676c0a0e00 con 0x7f676c0a2410 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 10 monclient(hunting): my global_id is 144288 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.849+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f676c0a4970 con 0x7f676c0a2410 2024-09-16T04:24:40.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2760801217 0 0) 0x7f6758003520 con 0x7f676c13d060 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 >> v1:172.21.15.171:6789/0 conn(0x7f676c0a1b10 legacy=0x7f676c13b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 >> v1:172.21.15.148:6789/0 conn(0x7f676c0a2410 legacy=0x7f676c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: _finish_hunting 0 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 monclient: found mon.noname-b 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c06dfd0 con 0x7f676c13d060 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: _finish_auth 0 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: _check_auth_tickets 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 5 monclient: authenticate success, global_id 154175 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6764001990 con 0x7f676c13d060 2024-09-16T04:24:40.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: dump: 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.864 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.865 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.865 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.865 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:24:40.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6764001c30 con 0x7f676c13d060 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 1 -- 172.21.15.148:0/3502682675 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6764002040 con 0x7f676c13d060 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 4 set_mon_vals no callback set 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f6771382640 10 monclient: dump: 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.866 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 10 monclient: shutdown 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 1 -- 172.21.15.148:0/3502682675 >> v1:172.21.15.148:6790/0 conn(0x7f676c13d060 legacy=0x7f676c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 1 -- 172.21.15.148:0/3502682675 shutdown_connections 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 1 -- 172.21.15.148:0/3502682675 >> 172.21.15.148:0/3502682675 conn(0x7f676c0a97b0 msgr2=0x7f676c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 1 -- 172.21.15.148:0/3502682675 shutdown_connections 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.850+0000 7f67745e8640 1 -- 172.21.15.148:0/3502682675 wait complete. 2024-09-16T04:24:40.867 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.851+0000 7f67745e8640 10 monclient: build_initial_monmap 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.851+0000 7f67745e8640 10 monclient: monmap: 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:24:40.868 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.851+0000 7f67745e8640 1 Processor -- start 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.851+0000 7f67745e8640 1 -- start start 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.851+0000 7f67745e8640 10 monclient: init 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient(hunting): picked mon.noname-b con 0x7f676c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient(hunting): picked mon.noname-c con 0x7f676c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient(hunting): picked mon.noname-a con 0x7f676c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676c1378c0 con 0x7f676c13d060 2024-09-16T04:24:40.869 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676c13c810 con 0x7f676c0a1b10 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676c1403b0 con 0x7f676c0a2410 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f67745e8640 10 monclient(hunting): authenticate will time out at 1901.478325s 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f6771b83640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f676c0a2410 0x7f676c1d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:59456/0 (socket says 172.21.15.148:59456) 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f6771b83640 1 -- 172.21.15.148:0/216145739 learned_addr learned my addr 172.21.15.148:0/216145739 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f6772384640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.171:6789/0 conn(0x7f676c0a1b10 legacy=0x7f676c13c100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268606411 0 0) 0x7f676c1403b0 con 0x7f676c0a2410 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): my global_id is 154178 2024-09-16T04:24:40.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6750003520 con 0x7f676c0a2410 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2537522165 0 0) 0x7f676c1378c0 con 0x7f676c13d060 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): my global_id is 144291 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f676c1403b0 con 0x7f676c13d060 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 223965624 0 0) 0x7f6750003520 con 0x7f676c0a2410 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f676c1378c0 con 0x7f676c0a2410 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6768004180 con 0x7f676c0a2410 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): dump: 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.871 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1938011232 0 0) 0x7f676c1403b0 con 0x7f676c13d060 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6750003520 con 0x7f676c13d060 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6764001890 con 0x7f676c13d060 2024-09-16T04:24:40.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 10 monclient(hunting): dump: 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3536933746 0 0) 0x7f676c1378c0 con 0x7f676c0a2410 2024-09-16T04:24:40.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.852+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.171:6789/0 conn(0x7f676c0a1b10 legacy=0x7f676c13c100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.148:6789/0 conn(0x7f676c13d060 legacy=0x7f676c1d3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: _finish_hunting 0 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 1 monclient: found mon.c 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c13ccf0 con 0x7f676c0a2410 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: _finish_auth 0 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: _check_auth_tickets 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 5 monclient: authenticate success, global_id 154178 2024-09-16T04:24:40.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 10 monclient: _renew_subs 2024-09-16T04:24:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f676c0a0130 con 0x7f676c0a2410 2024-09-16T04:24:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 10 monclient: _renew_subs 2024-09-16T04:24:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:24:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f676c136f80 con 0x7f676c0a2410 2024-09-16T04:24:40.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6768004430 con 0x7f676c0a2410 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6768004800 con 0x7f676c0a2410 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f6770b81640 4 set_mon_vals no callback set 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 10 monclient: dump: 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.876 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67540065c0 con 0x7f6754002860 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.853+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f67680049a0 con 0x7f676c0a2410 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 188..257) ==== 6817+0+0 (unknown 2509300937 0 0) 0x7f67680479f0 con 0x7f676c0a2410 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152344430 0 0) 0x7f67540065c0 con 0x7f6754002860 2024-09-16T04:24:40.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient(hunting): my global_id is 154181 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f676c1378c0 con 0x7f6754002860 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f6772b85640 10 monclient: get_auth_request con 0x7f6750038ed0 auth_method 0 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3044218958 0 0) 0x7f676c1378c0 con 0x7f6754002860 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67540065c0 con 0x7f6754002860 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6760001940 con 0x7f6754002860 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 10 monclient: dump: 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.878 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 20556854 0 0) 0x7f67540065c0 con 0x7f6754002860 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.854+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f675400a150 con 0x7f6754002860 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.855+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6760001ca0 con 0x7f6754002860 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.855+0000 7f675effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.855+0000 7f675effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:24:40.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.855+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.148:6790/0 conn(0x7f6754002860 legacy=0x7f6754004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.855+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.148:6790/0 conn(0x7f6754002860 legacy=0x7f6754004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:40.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.860+0000 7f67745e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:24:40.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6754009e20 con 0x7f67540090e0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211981440 0 0) 0x7f6754009e20 con 0x7f67540090e0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient(hunting): _negotiate 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient(hunting): my global_id is 154184 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67540065c0 con 0x7f67540090e0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4257276736 0 0) 0x7f67540065c0 con 0x7f67540090e0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6754009e20 con 0x7f67540090e0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6760004a40 con 0x7f67540090e0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:24:40.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient: dump: 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:24:40.883 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3524737055 0 0) 0x7f6754009e20 con 0x7f67540090e0 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f675400c390 con 0x7f67540090e0 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4132 (unknown 0 0 3645793736) 0x7f6760006030 con 0x7f67540090e0 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.148:6790/0 conn(0x7f67540090e0 legacy=0x7f6754009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.861+0000 7f675effd640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.148:6790/0 conn(0x7f67540090e0 legacy=0x7f6754009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:24:40.884 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 74, 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 17769, 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:24:40.885 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:40.886 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:40.887 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:24:40.888 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:24:40.889 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:24:40.890 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.891 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:24:40.892 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:24:40.893 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:24:40.894 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.895 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:24:40.896 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.171:6812/7327273 conn(0x7f6750038ed0 legacy=0x7f675003b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 10 monclient: shutdown 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 >> v1:172.21.15.148:6790/0 conn(0x7f676c0a2410 legacy=0x7f676c1d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f6772b85640 1 -- 172.21.15.148:0/216145739 reap_dead start 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f6772b85640 1 -- 172.21.15.148:0/216145739 reap_dead start 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 shutdown_connections 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 >> 172.21.15.148:0/216145739 conn(0x7f676c0a97b0 msgr2=0x7f676c0a7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 shutdown_connections 2024-09-16T04:24:40.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:24:40.862+0000 7f67745e8640 1 -- 172.21.15.148:0/216145739 wait complete. 2024-09-16T04:24:40.899 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:24:40.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:25:50.030 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=110 2024-09-16T04:25:50.031 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 110 -ne 110 ]] 2024-09-16T04:25:50.031 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:25:50.031 INFO:tasks.workunit.client.0.smithi171.stderr:+ export CEPH_ARGS= 2024-09-16T04:25:50.031 INFO:tasks.workunit.client.0.smithi171.stderr:+ CEPH_ARGS= 2024-09-16T04:25:50.031 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph auth del client.bar 2024-09-16T04:25:50.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.109+0000 7f3f4eba6640 10 monclient: get_monmap_and_config 2024-09-16T04:25:50.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.109+0000 7f3f4eba6640 10 monclient: build_initial_monmap 2024-09-16T04:25:50.110 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.109+0000 7f3f4eba6640 10 monclient: monmap: 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:25:50.109949+0000 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:25:50.109949+0000 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:25:50.111 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.109+0000 7f3f4eba6640 1 Processor -- start 2024-09-16T04:25:50.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.109+0000 7f3f4eba6640 1 -- start start 2024-09-16T04:25:50.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.109+0000 7f3f4eba6640 10 monclient: init 2024-09-16T04:25:50.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient: _reopen_session rank -1 2024-09-16T04:25:50.118 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4809ba20 addr v1:172.21.15.148:6790/0 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient(hunting): picked mon.noname-a con 0x7f3f48072e40 addr v1:172.21.15.148:6789/0 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4809c510 addr v1:172.21.15.171:6789/0 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f480a4900 con 0x7f3f48072e40 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f480a4e40 con 0x7f3f4809c510 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f480a5010 con 0x7f3f4809ba20 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient(hunting): _renew_subs 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4eba6640 10 monclient(hunting): authenticate will time out at 1925.615479s 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4d143640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7f3f4809c510 legacy=0x7f3f4813f640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4c942640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f3f4809ba20 0x7f3f4809be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.171:59138/0 (socket says 172.21.15.171:59138) 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f4c942640 1 -- 172.21.15.171:0/3410201264 learned_addr learned my addr 172.21.15.171:0/3410201264 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:25:50.119 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.110+0000 7f3f3ffff640 1 --1- 172.21.15.171:0/3410201264 >> v1:172.21.15.148:6789/0 conn(0x7f3f48072e40 0x7f3f4813bf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.171:49948/0 (socket says 172.21.15.171:49948) 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2710388968 0 0) 0x7f3f480a5010 con 0x7f3f4809ba20 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): my global_id is 154187 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f30003520 con 0x7f3f4809ba20 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568253658 0 0) 0x7f3f480a4900 con 0x7f3f48072e40 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient(hunting): my global_id is 144294 2024-09-16T04:25:50.120 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f480a5010 con 0x7f3f48072e40 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2762975215 0 0) 0x7f3f480a5010 con 0x7f3f48072e40 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 >> v1:172.21.15.148:6790/0 conn(0x7f3f4809ba20 legacy=0x7f3f4809be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 >> v1:172.21.15.171:6789/0 conn(0x7f3f4809c510 legacy=0x7f3f4813f640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f480a51e0 con 0x7f3f48072e40 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:25:50.121 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f380015f0 con 0x7f3f48072e40 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: dump: 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:25:50.122 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f4eba6640 5 monclient: authenticate success, global_id 144294 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f4eba6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f38001860 con 0x7f3f48072e40 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f4eba6640 10 monclient: get_monmap_and_config success 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f4eba6640 4 set_mon_vals no callback set 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 1 -- 172.21.15.171:0/3410201264 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f38001c70 con 0x7f3f48072e40 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.111+0000 7f3f3f7fe640 10 monclient: dump: 2024-09-16T04:25:50.123 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:25:50.124 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 10 monclient: shutdown 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 1 -- 172.21.15.171:0/3410201264 >> v1:172.21.15.148:6789/0 conn(0x7f3f48072e40 legacy=0x7f3f4813bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 1 -- 172.21.15.171:0/3410201264 shutdown_connections 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 1 -- 172.21.15.171:0/3410201264 >> 172.21.15.171:0/3410201264 conn(0x7f3f480a94e0 msgr2=0x7f3f480a98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 1 -- 172.21.15.171:0/3410201264 shutdown_connections 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 1 -- 172.21.15.171:0/3410201264 wait complete. 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 10 monclient: build_initial_monmap 2024-09-16T04:25:50.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.112+0000 7f3f4eba6640 10 monclient: monmap: 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 1 Processor -- start 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 1 -- start start 2024-09-16T04:25:50.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient: init 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient: _reopen_session rank -1 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient(hunting): picked mon.noname-a con 0x7f3f48072e40 addr v1:172.21.15.148:6789/0 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4809ba20 addr v1:172.21.15.171:6789/0 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4809c510 addr v1:172.21.15.148:6790/0 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f480a0b10 con 0x7f3f48072e40 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f48136c00 con 0x7f3f4809ba20 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f3ffff640 1 -- >> v1:172.21.15.171:6789/0 conn(0x7f3f4809ba20 legacy=0x7f3f481d02b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.171:6789/0 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f481391d0 con 0x7f3f4809c510 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient(hunting): _renew_subs 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.113+0000 7f3f4eba6640 10 monclient(hunting): authenticate will time out at 1925.618793s 2024-09-16T04:25:50.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f4c942640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f3f48072e40 0x7f3f4813f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.171:49958/0 (socket says 172.21.15.171:49958) 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f4c942640 1 -- 172.21.15.171:0/2745721802 learned_addr learned my addr 172.21.15.171:0/2745721802 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577825784 0 0) 0x7f3f480a0b10 con 0x7f3f48072e40 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): my global_id is 144297 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f34003550 con 0x7f3f48072e40 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766660698 0 0) 0x7f3f481391d0 con 0x7f3f4809c510 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): my global_id is 154190 2024-09-16T04:25:50.128 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f480a0b10 con 0x7f3f4809c510 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4221191133 0 0) 0x7f3f34003550 con 0x7f3f48072e40 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f481391d0 con 0x7f3f48072e40 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f440013b0 con 0x7f3f48072e40 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): dump: 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:25:50.129 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3581558289 0 0) 0x7f3f480a0b10 con 0x7f3f4809c510 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f34003550 con 0x7f3f4809c510 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f400042f0 con 0x7f3f4809c510 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient(hunting): dump: 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.130 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 223921589 0 0) 0x7f3f481391d0 con 0x7f3f48072e40 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 >> v1:172.21.15.148:6790/0 conn(0x7f3f4809c510 legacy=0x7f3f481d39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 >> v1:172.21.15.171:6789/0 conn(0x7f3f4809ba20 legacy=0x7f3f481d02b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 monclient: found mon.a 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f48138930 con 0x7f3f48072e40 2024-09-16T04:25:50.131 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f4eba6640 5 monclient: authenticate success, global_id 144297 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f4eba6640 10 monclient: _renew_subs 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f4eba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.114+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4813fe60 con 0x7f3f48072e40 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f4eba6640 10 monclient: _renew_subs 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f4eba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f480a06d0 con 0x7f3f48072e40 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f44004a90 con 0x7f3f48072e40 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f3f44004de0 con 0x7f3f48072e40 2024-09-16T04:25:50.132 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3effd640 4 set_mon_vals no callback set 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f3d7fa640 10 monclient: dump: 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:25:50.133 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.115+0000 7f3f4eba6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.116+0000 7f3f4eba6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.116+0000 7f3f4eba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.116+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f3f44010210 con 0x7f3f48072e40 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.116+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f28005080 con 0x7f3f48072e40 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.116+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f3f44047bc0 con 0x7f3f48072e40 2024-09-16T04:25:50.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.116+0000 7f3f3ffff640 10 monclient: get_auth_request con 0x7f3f34038ef0 auth_method 0 2024-09-16T04:25:50.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.120+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f3f44015d60 con 0x7f3f48072e40 2024-09-16T04:25:50.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.120+0000 7f3f3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:25:50.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.120+0000 7f3f3d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:25:50.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.348+0000 7f3f4eba6640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T04:25:50.349 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.348+0000 7f3f4eba6640 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T04:25:50.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.348+0000 7f3f4eba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.350 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.348+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) -- 0x7f3f28005370 con 0x7f3f48072e40 2024-09-16T04:25:50.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.351+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v26) ==== 80+0+0 (unknown 2801643084 0 0) 0x7f3f44019a10 con 0x7f3f48072e40 2024-09-16T04:25:50.352 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.351+0000 7f3f3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T04:25:50.353 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.351+0000 7f3f3d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:25:50.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.511+0000 7f3f4eba6640 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T04:25:50.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.511+0000 7f3f4eba6640 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T04:25:50.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.511+0000 7f3f4eba6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:25:50.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.511+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) -- 0x7f3f28002210 con 0x7f3f48072e40 2024-09-16T04:25:50.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.512+0000 7f3f3d7fa640 1 -- 172.21.15.171:0/2745721802 <== mon.0 v1:172.21.15.148:6789/0 11 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v26) ==== 80+0+0 (unknown 2801643084 0 0) 0x7f3f4401e2d0 con 0x7f3f48072e40 2024-09-16T04:25:50.513 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.512+0000 7f3f3d7fa640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-16T04:25:50.514 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.512+0000 7f3f3d7fa640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:25:50.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 >> v1:172.21.15.171:6812/7327273 conn(0x7f3f34038ef0 legacy=0x7f3f3403b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:25:50.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 10 monclient: shutdown 2024-09-16T04:25:50.515 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:25:50.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 >> v1:172.21.15.148:6789/0 conn(0x7f3f48072e40 legacy=0x7f3f4813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:25:50.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 shutdown_connections 2024-09-16T04:25:50.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 >> 172.21.15.171:0/2745721802 conn(0x7f3f480a94e0 msgr2=0x7f3f480a8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:25:50.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 shutdown_connections 2024-09-16T04:25:50.516 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:25:50.514+0000 7f3f4eba6640 1 -- 172.21.15.171:0/2745721802 wait complete. 2024-09-16T04:25:50.524 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-16T04:25:50.525 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-16T04:25:50.525 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=13 2024-09-16T04:25:50.525 INFO:tasks.workunit.client.0.smithi171.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T04:25:50.525 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T04:25:50.525 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-16T04:26:10.899 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:26:10.900 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T04:26:10.900 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:26:10.900 DEBUG:teuthology.orchestra.run.smithi171:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:26:10.904 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:26:10.904 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:26:10.905 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:26:11.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.012+0000 7fa4cd0d9640 10 monclient: get_monmap_and_config 2024-09-16T04:26:11.013 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.012+0000 7fa4cd0d9640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.012+0000 7fa4cd0d9640 10 monclient: monmap: 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:11.013077+0000 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:11.013077+0000 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:11.014 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.012+0000 7fa4cd0d9640 1 Processor -- start 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.012+0000 7fa4cd0d9640 1 -- start start 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.012+0000 7fa4cd0d9640 10 monclient: init 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c80a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.022 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c80a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient(hunting): picked mon.noname-b con 0x7fa4c813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c80a0e00 con 0x7fa4c80a1b10 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c80a10e0 con 0x7fa4c80a2410 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c80a4970 con 0x7fa4c813d060 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 10 monclient(hunting): authenticate will time out at 1991.638342s 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c659c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa4c80a1b10 0x7fa4c813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34568/0 (socket says 172.21.15.148:34568) 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c659c640 1 -- 172.21.15.148:0/1521148700 learned_addr learned my addr 172.21.15.148:0/1521148700 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884704596 0 0) 0x7fa4c80a0e00 con 0x7fa4c80a1b10 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.023 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): my global_id is 144321 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4b0003520 con 0x7fa4c80a1b10 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129034253 0 0) 0x7fa4c80a4970 con 0x7fa4c813d060 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient(hunting): my global_id is 154214 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4c80a0e00 con 0x7fa4c813d060 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3854765544 0 0) 0x7fa4c80a0e00 con 0x7fa4c813d060 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 >> v1:172.21.15.171:6789/0 conn(0x7fa4c80a2410 legacy=0x7fa4c80a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.024 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 >> v1:172.21.15.148:6789/0 conn(0x7fa4c80a1b10 legacy=0x7fa4c813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 monclient: found mon.noname-b 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c806dfd0 con 0x7fa4c813d060 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4c00018f0 con 0x7fa4c813d060 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 5 monclient: authenticate success, global_id 154214 2024-09-16T04:26:11.025 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4c5d9b640 10 monclient: dump: 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.026 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.013+0000 7fa4cd0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4c0001be0 con 0x7fa4c813d060 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4c5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4c5d9b640 1 -- 172.21.15.148:0/1521148700 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4c0001ff0 con 0x7fa4c813d060 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 4 set_mon_vals no callback set 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4c5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4c5d9b640 10 monclient: dump: 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.027 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 10 monclient: shutdown 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/1521148700 >> v1:172.21.15.148:6790/0 conn(0x7fa4c813d060 legacy=0x7fa4c813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.028 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/1521148700 shutdown_connections 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/1521148700 >> 172.21.15.148:0/1521148700 conn(0x7fa4c80a97b0 msgr2=0x7fa4c80a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/1521148700 shutdown_connections 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.014+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/1521148700 wait complete. 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.015+0000 7fa4cd0d9640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.015+0000 7fa4cd0d9640 10 monclient: monmap: 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:11.029 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.016+0000 7fa4cd0d9640 1 Processor -- start 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.016+0000 7fa4cd0d9640 1 -- start start 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.016+0000 7fa4cd0d9640 10 monclient: init 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c80a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c80a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient(hunting): picked mon.noname-b con 0x7fa4c813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c81379d0 con 0x7fa4c80a2410 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c81392f0 con 0x7fa4c813d060 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c813cb20 con 0x7fa4c80a1b10 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient(hunting): authenticate will time out at 1991.642209s 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4c6d9d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa4c80a1b10 0x7fa4c8136bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39394/0 (socket says 172.21.15.148:39394) 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4c6d9d640 1 -- 172.21.15.148:0/126369311 learned_addr learned my addr 172.21.15.148:0/126369311 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 197918336 0 0) 0x7fa4c81379d0 con 0x7fa4c80a2410 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): my global_id is 144324 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4b4003520 con 0x7fa4c80a2410 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 130641624 0 0) 0x7fa4b4003520 con 0x7fa4c80a2410 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4c81379d0 con 0x7fa4c80a2410 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4bc001a30 con 0x7fa4c80a2410 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): dump: 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:11.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3559106351 0 0) 0x7fa4c81379d0 con 0x7fa4c80a2410 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 >> v1:172.21.15.148:6790/0 conn(0x7fa4c80a1b10 legacy=0x7fa4c8136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 >> v1:172.21.15.171:6789/0 conn(0x7fa4c813d060 legacy=0x7fa4c81d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 monclient: found mon.a 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c80a0130 con 0x7fa4c80a2410 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 5 monclient: authenticate success, global_id 144324 2024-09-16T04:26:11.033 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.017+0000 7fa4cd0d9640 10 monclient: _renew_subs 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4cd0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c813a4c0 con 0x7fa4c80a2410 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4cd0d9640 10 monclient: _renew_subs 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4cd0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa4c8139070 con 0x7fa4c80a2410 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4bc001d20 con 0x7fa4c80a2410 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa4bc004c30 con 0x7fa4c80a2410 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4c559a640 4 set_mon_vals no callback set 2024-09-16T04:26:11.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 10 monclient: dump: 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.018+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa4bc010060 con 0x7fa4c80a2410 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.019+0000 7fa4cd0d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.019+0000 7fa4cd0d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.019+0000 7fa4cd0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.019+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a8005080 con 0x7fa4c80a2410 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.019+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fa4bc047aa0 con 0x7fa4c80a2410 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.019+0000 7fa4c6d9d640 10 monclient: get_auth_request con 0x7fa4b4038f70 auth_method 0 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.023+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa4bc015ca0 con 0x7fa4c80a2410 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.023+0000 7fa4af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.023+0000 7fa4af7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4cd0d9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4cd0d9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4cd0d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa4a8005370 con 0x7fa4c80a2410 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4af7fe640 1 -- 172.21.15.148:0/126369311 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2658307241) 0x7fa4bc0155a0 con 0x7fa4c80a2410 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.256+0000 7fa4af7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:11.258 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 >> v1:172.21.15.171:6812/7327273 conn(0x7fa4b4038f70 legacy=0x7fa4b403b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 10 monclient: shutdown 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 >> v1:172.21.15.148:6789/0 conn(0x7fa4c80a2410 legacy=0x7fa4c81372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 shutdown_connections 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 >> 172.21.15.148:0/126369311 conn(0x7fa4c80a97b0 msgr2=0x7fa4c813fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 shutdown_connections 2024-09-16T04:26:11.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.258+0000 7fa4cd0d9640 1 -- 172.21.15.148:0/126369311 wait complete. 2024-09-16T04:26:11.270 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:26:11.271 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:26:11.271 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.369+0000 7fcb8c22e640 10 monclient: get_monmap_and_config 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.369+0000 7fcb8c22e640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.369+0000 7fcb8c22e640 10 monclient: monmap: 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:11.370079+0000 2024-09-16T04:26:11.371 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:11.370079+0000 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.369+0000 7fcb8c22e640 1 Processor -- start 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.369+0000 7fcb8c22e640 1 -- start start 2024-09-16T04:26:11.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.369+0000 7fcb8c22e640 10 monclient: init 2024-09-16T04:26:11.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:11.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient(hunting): picked mon.noname-a con 0x7fcb840a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient(hunting): picked mon.noname-c con 0x7fcb840a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient(hunting): picked mon.noname-b con 0x7fcb8413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb840a0e00 con 0x7fcb840a2410 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb840a10e0 con 0x7fcb840a1b10 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb840a4970 con 0x7fcb8413d060 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8c22e640 10 monclient(hunting): authenticate will time out at 1991.995468s 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb89fca640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcb840a2410 0x7fcb840a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34586/0 (socket says 172.21.15.148:34586) 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb8a7cb640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fcb8413d060 0x7fcb8413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39396/0 (socket says 172.21.15.148:39396) 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb89fca640 1 -- 172.21.15.148:0/438298826 learned_addr learned my addr 172.21.15.148:0/438298826 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4116590765 0 0) 0x7fcb840a0e00 con 0x7fcb840a2410 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): my global_id is 144327 2024-09-16T04:26:11.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb70003520 con 0x7fcb840a2410 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444869420 0 0) 0x7fcb840a4970 con 0x7fcb8413d060 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 10 monclient(hunting): my global_id is 154217 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.370+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb840a0e00 con 0x7fcb8413d060 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3420711308 0 0) 0x7fcb840a0e00 con 0x7fcb8413d060 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 >> v1:172.21.15.171:6789/0 conn(0x7fcb840a1b10 legacy=0x7fcb8413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 >> v1:172.21.15.148:6789/0 conn(0x7fcb840a2410 legacy=0x7fcb840a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 monclient: found mon.noname-b 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8406dfd0 con 0x7fcb8413d060 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcb7c001970 con 0x7fcb8413d060 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: dump: 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.383 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 5 monclient: authenticate success, global_id 154217 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb7c001be0 con 0x7fcb8413d060 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 1 -- 172.21.15.148:0/438298826 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcb7c001ff0 con 0x7fcb8413d060 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:11.384 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 4 set_mon_vals no callback set 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb88fc8640 10 monclient: dump: 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.385 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 10 monclient: shutdown 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 1 -- 172.21.15.148:0/438298826 >> v1:172.21.15.148:6790/0 conn(0x7fcb8413d060 legacy=0x7fcb8413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 1 -- 172.21.15.148:0/438298826 shutdown_connections 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 1 -- 172.21.15.148:0/438298826 >> 172.21.15.148:0/438298826 conn(0x7fcb840a97b0 msgr2=0x7fcb840a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 1 -- 172.21.15.148:0/438298826 shutdown_connections 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.371+0000 7fcb8c22e640 1 -- 172.21.15.148:0/438298826 wait complete. 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.372+0000 7fcb8c22e640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.372+0000 7fcb8c22e640 10 monclient: monmap: 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:11.386 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.372+0000 7fcb8c22e640 1 Processor -- start 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.372+0000 7fcb8c22e640 1 -- start start 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.372+0000 7fcb8c22e640 10 monclient: init 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient(hunting): picked mon.noname-b con 0x7fcb840a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient(hunting): picked mon.noname-c con 0x7fcb840a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.387 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient(hunting): picked mon.noname-a con 0x7fcb8413d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb8409ee50 con 0x7fcb8413d060 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb8409d6f0 con 0x7fcb840a1b10 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb8409f0d0 con 0x7fcb840a2410 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8c22e640 10 monclient(hunting): authenticate will time out at 1991.998305s 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8a7cb640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fcb8413d060 0x7fcb841d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34596/0 (socket says 172.21.15.148:34596) 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb897c9640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fcb840a2410 0x7fcb8409cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39398/0 (socket says 172.21.15.148:39398) 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb8a7cb640 1 -- 172.21.15.148:0/699599206 learned_addr learned my addr 172.21.15.148:0/699599206 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384647454 0 0) 0x7fcb8409ee50 con 0x7fcb8413d060 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.388 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): my global_id is 144330 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb68003520 con 0x7fcb8413d060 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 873486817 0 0) 0x7fcb68003520 con 0x7fcb8413d060 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb8409ee50 con 0x7fcb8413d060 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcb7c001f70 con 0x7fcb8413d060 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): dump: 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.389 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.373+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2851265183 0 0) 0x7fcb8409ee50 con 0x7fcb8413d060 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.148:6790/0 conn(0x7fcb840a2410 legacy=0x7fcb8409cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.171:6789/0 conn(0x7fcb840a1b10 legacy=0x7fcb8409c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 monclient: found mon.a 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.390 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8413cc40 con 0x7fcb8413d060 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 5 monclient: authenticate success, global_id 144330 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 10 monclient: _renew_subs 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb840a0130 con 0x7fcb8413d060 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 10 monclient: _renew_subs 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb841379f0 con 0x7fcb8413d060 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb7c004f70 con 0x7fcb8413d060 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcb7c005120 con 0x7fcb8413d060 2024-09-16T04:26:11.391 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb77fff640 4 set_mon_vals no callback set 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 10 monclient: dump: 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.392 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.374+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fcb7c0052c0 con 0x7fcb8413d060 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.375+0000 7fcb8c22e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.375+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcb6c0065c0 con 0x7fcb6c002860 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.375+0000 7fcb89fca640 10 monclient: get_auth_request con 0x7fcb680390f0 auth_method 0 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fcb7c045fe0 con 0x7fcb8413d060 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3107274480 0 0) 0x7fcb6c0065c0 con 0x7fcb6c002860 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient(hunting): my global_id is 144333 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb8409ee50 con 0x7fcb6c002860 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3423196166 0 0) 0x7fcb8409ee50 con 0x7fcb6c002860 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb6c0065c0 con 0x7fcb6c002860 2024-09-16T04:26:11.393 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcb80001640 con 0x7fcb6c002860 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient: dump: 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.394 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3206496593 0 0) 0x7fcb6c0065c0 con 0x7fcb6c002860 2024-09-16T04:26:11.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb6c00a150 con 0x7fcb6c002860 2024-09-16T04:26:11.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcb80001a10 con 0x7fcb6c002860 2024-09-16T04:26:11.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:11.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.148:6789/0 conn(0x7fcb6c002860 legacy=0x7fcb6c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.395 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.376+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.148:6789/0 conn(0x7fcb6c002860 legacy=0x7fcb6c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:11.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.382+0000 7fcb8c22e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:11.396 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.382+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcb6c009e40 con 0x7fcb6c009100 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967534008 0 0) 0x7fcb6c009e40 con 0x7fcb6c009100 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient(hunting): my global_id is 144336 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb6c0065c0 con 0x7fcb6c009100 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3614581405 0 0) 0x7fcb6c0065c0 con 0x7fcb6c009100 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb6c009e40 con 0x7fcb6c009100 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fcb800037b0 con 0x7fcb6c009100 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient: dump: 2024-09-16T04:26:11.397 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 367328350 0 0) 0x7fcb6c009e40 con 0x7fcb6c009100 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb6c00c3d0 con 0x7fcb6c009100 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4313 (unknown 0 0 3189533673) 0x7fcb800057c0 con 0x7fcb6c009100 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:11.398 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:11.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.148:6789/0 conn(0x7fcb6c009100 legacy=0x7fcb6c009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.399 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.383+0000 7fcb767fc640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.148:6789/0 conn(0x7fcb6c009100 legacy=0x7fcb6c009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:11.399 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:11.399 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:11.399 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 76, 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 216083, 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:11.400 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.401 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.402 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:11.403 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.404 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.405 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:11.406 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.407 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.408 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.409 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.410 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.411 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.412 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:11.412 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.412 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.412 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.412 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:11.412 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:11.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.171:6812/7327273 conn(0x7fcb680390f0 legacy=0x7fcb6803b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 10 monclient: shutdown 2024-09-16T04:26:11.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 >> v1:172.21.15.148:6789/0 conn(0x7fcb8413d060 legacy=0x7fcb841d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8a7cb640 1 -- 172.21.15.148:0/699599206 reap_dead start 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8a7cb640 1 -- 172.21.15.148:0/699599206 reap_dead start 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 shutdown_connections 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 >> 172.21.15.148:0/699599206 conn(0x7fcb840a97b0 msgr2=0x7fcb840a6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 shutdown_connections 2024-09-16T04:26:11.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.384+0000 7fcb8c22e640 1 -- 172.21.15.148:0/699599206 wait complete. 2024-09-16T04:26:11.414 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:26:11.414 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:26:11.415 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:26:11.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.512+0000 7f6f8ef86640 10 monclient: get_monmap_and_config 2024-09-16T04:26:11.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.512+0000 7f6f8ef86640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.512+0000 7f6f8ef86640 10 monclient: monmap: 2024-09-16T04:26:11.514 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.514 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.514 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:11.513723+0000 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:11.513723+0000 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:11.515 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.513+0000 7f6f8ef86640 1 Processor -- start 2024-09-16T04:26:11.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.513+0000 7f6f8ef86640 1 -- start start 2024-09-16T04:26:11.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.513+0000 7f6f8ef86640 10 monclient: init 2024-09-16T04:26:11.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient(hunting): picked mon.noname-b con 0x7f6f88134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient(hunting): picked mon.noname-c con 0x7f6f8809e240 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8813ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f88135360 con 0x7f6f8813ae80 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f88134440 con 0x7f6f8809e240 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8809e650 con 0x7f6f88134870 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient(hunting): authenticate will time out at 1992.139148s 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8cd22640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6f88134870 0x7f6f88134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39404/0 (socket says 172.21.15.148:39404) 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8cd22640 1 -- 172.21.15.148:0/3820100127 learned_addr learned my addr 172.21.15.148:0/3820100127 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105252271 0 0) 0x7f6f8809e650 con 0x7f6f88134870 2024-09-16T04:26:11.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): my global_id is 154220 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f70003520 con 0x7f6f88134870 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855761997 0 0) 0x7f6f88135360 con 0x7f6f8813ae80 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient(hunting): my global_id is 144339 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8809e650 con 0x7f6f8813ae80 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 424722997 0 0) 0x7f6f70003520 con 0x7f6f88134870 2024-09-16T04:26:11.526 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 >> v1:172.21.15.171:6789/0 conn(0x7f6f8809e240 legacy=0x7f6f88139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 >> v1:172.21.15.148:6789/0 conn(0x7f6f8813ae80 legacy=0x7f6f8813d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 monclient: found mon.noname-b 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f88073230 con 0x7f6f88134870 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f78001540 con 0x7f6f88134870 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 5 monclient: authenticate success, global_id 154220 2024-09-16T04:26:11.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: dump: 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.528 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f780017a0 con 0x7f6f88134870 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f8ef86640 4 set_mon_vals no callback set 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 1 -- 172.21.15.148:0/3820100127 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f78001bb0 con 0x7f6f88134870 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.514+0000 7f6f877fe640 10 monclient: dump: 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.529 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 10 monclient: shutdown 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 1 -- 172.21.15.148:0/3820100127 >> v1:172.21.15.148:6790/0 conn(0x7f6f88134870 legacy=0x7f6f88134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 1 -- 172.21.15.148:0/3820100127 shutdown_connections 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 1 -- 172.21.15.148:0/3820100127 >> 172.21.15.148:0/3820100127 conn(0x7f6f880a5640 msgr2=0x7f6f880a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.530 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 1 -- 172.21.15.148:0/3820100127 shutdown_connections 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 1 -- 172.21.15.148:0/3820100127 wait complete. 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.515+0000 7f6f8ef86640 10 monclient: monmap: 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:11.531 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 1 Processor -- start 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 1 -- start start 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient: init 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient(hunting): picked mon.noname-b con 0x7f6f8809e240 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient(hunting): picked mon.noname-c con 0x7f6f88134870 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8813ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8811eaf0 con 0x7f6f8813ae80 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8811edd0 con 0x7f6f8809e240 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8811e810 con 0x7f6f88134870 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.516+0000 7f6f8ef86640 10 monclient(hunting): authenticate will time out at 1992.141988s 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f87fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f6f88134870 0x7f6f881ce450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39408/0 (socket says 172.21.15.148:39408) 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f87fff640 1 -- 172.21.15.148:0/551261341 learned_addr learned my addr 172.21.15.148:0/551261341 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193336591 0 0) 0x7f6f8811eaf0 con 0x7f6f8813ae80 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): my global_id is 144342 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f74003520 con 0x7f6f8813ae80 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344138647 0 0) 0x7f6f8811e810 con 0x7f6f88134870 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): my global_id is 154223 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8811eaf0 con 0x7f6f88134870 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722815313 0 0) 0x7f6f8811edd0 con 0x7f6f8809e240 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): my global_id is 164101 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8811e810 con 0x7f6f8809e240 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3409197964 0 0) 0x7f6f8811eaf0 con 0x7f6f88134870 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f8811edd0 con 0x7f6f88134870 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f7c0018b0 con 0x7f6f88134870 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): dump: 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:11.535 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1002661286 0 0) 0x7f6f74003520 con 0x7f6f8813ae80 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f8811eaf0 con 0x7f6f8813ae80 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f80001610 con 0x7f6f8813ae80 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): dump: 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.536 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 421579276 0 0) 0x7f6f8811e810 con 0x7f6f8809e240 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f74003520 con 0x7f6f8809e240 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f78001af0 con 0x7f6f8809e240 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient(hunting): dump: 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.537 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3902604999 0 0) 0x7f6f8811edd0 con 0x7f6f88134870 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.171:6789/0 conn(0x7f6f8809e240 legacy=0x7f6f88132e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.148:6789/0 conn(0x7f6f8813ae80 legacy=0x7f6f881d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 monclient: found mon.c 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8811d310 con 0x7f6f88134870 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 5 monclient: authenticate success, global_id 154223 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 10 monclient: _renew_subs 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8811c250 con 0x7f6f88134870 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 10 monclient: _renew_subs 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f88133560 con 0x7f6f88134870 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f7c001ca0 con 0x7f6f88134870 2024-09-16T04:26:11.539 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f7c003a60 con 0x7f6f88134870 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f857fa640 10 monclient: dump: 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.540 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.517+0000 7f6f86ffd640 4 set_mon_vals no callback set 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f8ef86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f680065c0 con 0x7f6f68002860 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6f7c003c80 con 0x7f6f88134870 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f6f7c0478c0 con 0x7f6f88134870 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510989159 0 0) 0x7f6f680065c0 con 0x7f6f68002860 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 10 monclient(hunting): my global_id is 154226 2024-09-16T04:26:11.541 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.518+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f8811edd0 con 0x7f6f68002860 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f8d523640 10 monclient: get_auth_request con 0x7f6f74039180 auth_method 0 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1959184992 0 0) 0x7f6f8811edd0 con 0x7f6f68002860 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f680065c0 con 0x7f6f68002860 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f78004940 con 0x7f6f68002860 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 10 monclient: dump: 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.542 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3193755996 0 0) 0x7f6f680065c0 con 0x7f6f68002860 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f6800a150 con 0x7f6f68002860 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6f78006ba0 con 0x7f6f68002860 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.148:6790/0 conn(0x7f6f68002860 legacy=0x7f6f68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.543 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.519+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.148:6790/0 conn(0x7f6f68002860 legacy=0x7f6f68004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.525+0000 7f6f8ef86640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.525+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6f68009e20 con 0x7f6f680090e0 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3535379963 0 0) 0x7f6f68009e20 con 0x7f6f680090e0 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient(hunting): my global_id is 154229 2024-09-16T04:26:11.545 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f680065c0 con 0x7f6f680090e0 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1492293641 0 0) 0x7f6f680065c0 con 0x7f6f680090e0 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f68009e20 con 0x7f6f680090e0 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6f78006280 con 0x7f6f680090e0 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient: dump: 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.546 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1952836996 0 0) 0x7f6f68009e20 con 0x7f6f680090e0 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6f6800c390 con 0x7f6f680090e0 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 3374449666) 0x7f6f78006150 con 0x7f6f680090e0 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.148:6790/0 conn(0x7f6f680090e0 legacy=0x7f6f68009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.526+0000 7f6f857fa640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.148:6790/0 conn(0x7f6f680090e0 legacy=0x7f6f68009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:11.547 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 76, 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 108434, 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.548 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.549 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.550 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:11.551 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.552 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:11.553 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.554 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.555 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:11.557 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.558 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 3 2024-09-16T04:26:11.559 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.560 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.560 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.560 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:11.560 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:11.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.171:6812/7327273 conn(0x7f6f74039180 legacy=0x7f6f7403b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 10 monclient: shutdown 2024-09-16T04:26:11.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 >> v1:172.21.15.148:6790/0 conn(0x7f6f88134870 legacy=0x7f6f881ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8d523640 1 -- 172.21.15.148:0/551261341 reap_dead start 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8d523640 1 -- 172.21.15.148:0/551261341 reap_dead start 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 shutdown_connections 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 >> 172.21.15.148:0/551261341 conn(0x7f6f880a5640 msgr2=0x7f6f880a32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 shutdown_connections 2024-09-16T04:26:11.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.527+0000 7f6f8ef86640 1 -- 172.21.15.148:0/551261341 wait complete. 2024-09-16T04:26:11.562 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:26:11.562 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:26:11.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:26:11.602 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=13 2024-09-16T04:26:11.602 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 13 -ne 13 ]] 2024-09-16T04:26:11.602 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:26:11.602 INFO:tasks.workunit.client.0.smithi171.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-16T04:26:11.603 INFO:tasks.workunit.client.0.smithi171.stderr:+ cat 2024-09-16T04:26:11.603 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph auth import -i - 2024-09-16T04:26:11.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.661+0000 7f4a8a6dd640 10 monclient: get_monmap_and_config 2024-09-16T04:26:11.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.661+0000 7f4a8a6dd640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.661+0000 7f4a8a6dd640 10 monclient: monmap: 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:11.662036+0000 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:11.662036+0000 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:11.663 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.661+0000 7f4a8a6dd640 1 Processor -- start 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.661+0000 7f4a8a6dd640 1 -- start start 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.661+0000 7f4a8a6dd640 10 monclient: init 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f4a840a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f4a840a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a840a0e00 con 0x7f4a840a2410 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a840a10e0 con 0x7f4a840a1b10 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a840a4970 con 0x7f4a8413d060 2024-09-16T04:26:11.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a8a6dd640 10 monclient(hunting): authenticate will time out at 1992.287431s 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a83fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4a840a2410 0x7f4a840a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34648/0 (socket says 172.21.15.148:34648) 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a83fff640 1 -- 172.21.15.148:0/3231467894 learned_addr learned my addr 172.21.15.148:0/3231467894 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a88c7a640 1 --1- 172.21.15.148:0/3231467894 >> v1:172.21.15.148:6790/0 conn(0x7f4a8413d060 0x7f4a8413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39440/0 (socket says 172.21.15.148:39440) 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2869926369 0 0) 0x7f4a840a4970 con 0x7f4a8413d060 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): my global_id is 154235 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a70003520 con 0x7f4a8413d060 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344381062 0 0) 0x7f4a840a0e00 con 0x7f4a840a2410 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): my global_id is 144348 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a840a4970 con 0x7f4a840a2410 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923561822 0 0) 0x7f4a840a10e0 con 0x7f4a840a1b10 2024-09-16T04:26:11.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 10 monclient(hunting): my global_id is 164107 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.662+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a840a0e00 con 0x7f4a840a1b10 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 342158739 0 0) 0x7f4a70003520 con 0x7f4a8413d060 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 >> v1:172.21.15.171:6789/0 conn(0x7f4a840a1b10 legacy=0x7f4a8413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 >> v1:172.21.15.148:6789/0 conn(0x7f4a840a2410 legacy=0x7f4a840a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 monclient: found mon.noname-b 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8406dfd0 con 0x7f4a8413d060 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a7c0018c0 con 0x7f4a8413d060 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 5 monclient: authenticate success, global_id 154235 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: dump: 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.677 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a7c001be0 con 0x7f4a8413d060 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:11.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 1 -- 172.21.15.148:0/3231467894 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a7c001ff0 con 0x7f4a8413d060 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 4 set_mon_vals no callback set 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a82ffd640 10 monclient: dump: 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.679 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 10 monclient: shutdown 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/3231467894 >> v1:172.21.15.148:6790/0 conn(0x7f4a8413d060 legacy=0x7f4a8413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/3231467894 shutdown_connections 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/3231467894 >> 172.21.15.148:0/3231467894 conn(0x7f4a840a97b0 msgr2=0x7f4a840a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/3231467894 shutdown_connections 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.663+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/3231467894 wait complete. 2024-09-16T04:26:11.680 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.664+0000 7f4a8a6dd640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.664+0000 7f4a8a6dd640 10 monclient: monmap: 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.664+0000 7f4a8a6dd640 1 Processor -- start 2024-09-16T04:26:11.681 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.664+0000 7f4a8a6dd640 1 -- start start 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.664+0000 7f4a8a6dd640 10 monclient: init 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f4a840a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f4a840a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8409ee80 con 0x7f4a840a1b10 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8409d720 con 0x7f4a840a2410 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8409ea40 con 0x7f4a8413d060 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a8a6dd640 10 monclient(hunting): authenticate will time out at 1992.290368s 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a83fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f4a840a1b10 0x7f4a8409c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34662/0 (socket says 172.21.15.148:34662) 2024-09-16T04:26:11.682 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a83fff640 1 -- 172.21.15.148:0/1401710388 learned_addr learned my addr 172.21.15.148:0/1401710388 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a88c7a640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f4a8413d060 0x7f4a841d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39446/0 (socket says 172.21.15.148:39446) 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3473612286 0 0) 0x7f4a8409ea40 con 0x7f4a8413d060 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): my global_id is 154238 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a64003520 con 0x7f4a8413d060 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4239996253 0 0) 0x7f4a8409ee80 con 0x7f4a840a1b10 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): my global_id is 144351 2024-09-16T04:26:11.683 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a8409ea40 con 0x7f4a840a1b10 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632622940 0 0) 0x7f4a8409d720 con 0x7f4a840a2410 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): my global_id is 164110 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a8409ee80 con 0x7f4a840a2410 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3708907998 0 0) 0x7f4a64003520 con 0x7f4a8413d060 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a8409d720 con 0x7f4a8413d060 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a7c001f30 con 0x7f4a8413d060 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): dump: 2024-09-16T04:26:11.684 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:11.685 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 851551773 0 0) 0x7f4a8409ea40 con 0x7f4a840a1b10 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a64003520 con 0x7f4a840a1b10 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a740017f0 con 0x7f4a840a1b10 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 10 monclient(hunting): dump: 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.686 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.665+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3412113339 0 0) 0x7f4a8409ee80 con 0x7f4a840a2410 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a8409ea40 con 0x7f4a840a2410 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a6c001870 con 0x7f4a840a2410 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient(hunting): dump: 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.687 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2491044404 0 0) 0x7f4a8409d720 con 0x7f4a8413d060 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.171:6789/0 conn(0x7f4a840a2410 legacy=0x7f4a8409d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.148:6789/0 conn(0x7f4a840a1b10 legacy=0x7f4a8409c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 monclient: found mon.c 2024-09-16T04:26:11.688 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a84137480 con 0x7f4a8413d060 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 5 monclient: authenticate success, global_id 154238 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 10 monclient: _renew_subs 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4a84137760 con 0x7f4a8413d060 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 10 monclient: _renew_subs 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.689 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4a8413c440 con 0x7f4a8413d060 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4a7c001be0 con 0x7f4a8413d060 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a7c005050 con 0x7f4a8413d060 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a827fc640 4 set_mon_vals no callback set 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 10 monclient: dump: 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.690 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.666+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4a7c0102e0 con 0x7f4a8413d060 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f4a7c047b00 con 0x7f4a8413d060 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a8a6dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a680065c0 con 0x7f4a68002860 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a83fff640 10 monclient: get_auth_request con 0x7f4a640392b0 auth_method 0 2024-09-16T04:26:11.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1892188513 0 0) 0x7f4a680065c0 con 0x7f4a68002860 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient(hunting): my global_id is 164113 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a8409d720 con 0x7f4a68002860 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2662406030 0 0) 0x7f4a8409d720 con 0x7f4a68002860 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a680065c0 con 0x7f4a68002860 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a6c003790 con 0x7f4a68002860 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 10 monclient: dump: 2024-09-16T04:26:11.692 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3149542880 0 0) 0x7f4a680065c0 con 0x7f4a68002860 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.667+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a6800a150 con 0x7f4a68002860 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.668+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4a6c005a50 con 0x7f4a68002860 2024-09-16T04:26:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.668+0000 7f4a80ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.668+0000 7f4a80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:11.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.668+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.171:6789/0 conn(0x7f4a68002860 legacy=0x7f4a68004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.668+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.171:6789/0 conn(0x7f4a68002860 legacy=0x7f4a68004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:11.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a8a6dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:11.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4a68009e40 con 0x7f4a68009100 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802660999 0 0) 0x7f4a68009e40 con 0x7f4a68009100 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient(hunting): my global_id is 164116 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a680065c0 con 0x7f4a68009100 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 8319259 0 0) 0x7f4a680065c0 con 0x7f4a68009100 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4a68009e40 con 0x7f4a68009100 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4a6c005060 con 0x7f4a68009100 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 10 monclient: dump: 2024-09-16T04:26:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1530620744 0 0) 0x7f4a68009e40 con 0x7f4a68009100 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.674+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a6800c3d0 con 0x7f4a68009100 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3974 (unknown 0 0 3368699930) 0x7f4a6c004ec0 con 0x7f4a68009100 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a80ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:11.699 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a80ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:11.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.171:6789/0 conn(0x7f4a68009100 legacy=0x7f4a68009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.700 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a80ff9640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.171:6789/0 conn(0x7f4a68009100 legacy=0x7f4a68009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:11.700 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:11.700 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 76, 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 682, 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:11.701 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.702 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.703 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:11.704 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:11.705 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:11.706 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.707 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.708 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.709 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.710 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.711 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:11.712 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:11.713 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.171:6812/7327273 conn(0x7f4a640392b0 legacy=0x7f4a6403b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 10 monclient: shutdown 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 >> v1:172.21.15.148:6790/0 conn(0x7f4a8413d060 legacy=0x7f4a841d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a88c7a640 1 -- 172.21.15.148:0/1401710388 reap_dead start 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a88c7a640 1 -- 172.21.15.148:0/1401710388 reap_dead start 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 shutdown_connections 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 >> 172.21.15.148:0/1401710388 conn(0x7f4a840a97b0 msgr2=0x7f4a840a83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 shutdown_connections 2024-09-16T04:26:11.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.675+0000 7f4a8a6dd640 1 -- 172.21.15.148:0/1401710388 wait complete. 2024-09-16T04:26:11.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:26:11.718 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:26:11.718 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:26:11.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.683+0000 7fd3a2a1e640 10 monclient: get_monmap_and_config 2024-09-16T04:26:11.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.683+0000 7fd3a2a1e640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.725 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.683+0000 7fd3a2a1e640 10 monclient: monmap: 2024-09-16T04:26:11.725 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:26:11.684307+0000 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:26:11.684307+0000 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 1 Processor -- start 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 1 -- start start 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 10 monclient: init 2024-09-16T04:26:11.726 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fd39c067e40 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fd39c0a4360 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fd39c0a4810 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c09ed60 con 0x7fd39c067e40 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c09f040 con 0x7fd39c0a4360 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.684+0000 7fd3a2a1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c09f320 con 0x7fd39c0a4810 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 10 monclient(hunting): authenticate will time out at 1947.190018s 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39b7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd39c0a4360 0x7fd39c139960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39066/0 (socket says 172.21.15.171:39066) 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39b7fe640 1 -- 172.21.15.171:0/4204553591 learned_addr learned my addr 172.21.15.171:0/4204553591 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:11.727 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609824311 0 0) 0x7fd39c09f040 con 0x7fd39c0a4360 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): my global_id is 164119 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd384003520 con 0x7fd39c0a4360 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141815164 0 0) 0x7fd39c09ed60 con 0x7fd39c067e40 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): my global_id is 144354 2024-09-16T04:26:11.728 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd39c09f040 con 0x7fd39c067e40 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626834959 0 0) 0x7fd39c09f320 con 0x7fd39c0a4810 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient(hunting): my global_id is 154241 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd39c09ed60 con 0x7fd39c0a4810 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2758119638 0 0) 0x7fd384003520 con 0x7fd39c0a4360 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 >> v1:172.21.15.148:6790/0 conn(0x7fd39c0a4810 legacy=0x7fd39c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 >> v1:172.21.15.148:6789/0 conn(0x7fd39c067e40 legacy=0x7fd39c0a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 monclient: found mon.noname-c 2024-09-16T04:26:11.729 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c09f600 con 0x7fd39c0a4360 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 5 monclient: authenticate success, global_id 164119 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd390001610 con 0x7fd39c0a4360 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.730 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: dump: 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.731 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3900018c0 con 0x7fd39c0a4360 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 1 -- 172.21.15.171:0/4204553591 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd390001c90 con 0x7fd39c0a4360 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd3a2a1e640 4 set_mon_vals no callback set 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.732 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.685+0000 7fd39affd640 10 monclient: dump: 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.733 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 10 monclient: shutdown 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/4204553591 >> v1:172.21.15.171:6789/0 conn(0x7fd39c0a4360 legacy=0x7fd39c139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/4204553591 shutdown_connections 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/4204553591 >> 172.21.15.171:0/4204553591 conn(0x7fd39c0a5d20 msgr2=0x7fd39c0a6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/4204553591 shutdown_connections 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/4204553591 wait complete. 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.734 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.686+0000 7fd3a2a1e640 10 monclient: monmap: 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.735 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 1 Processor -- start 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 1 -- start start 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient: init 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fd39c067e40 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fd39c0a4360 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fd39c0a4810 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c13e030 con 0x7fd39c0a4810 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c134a10 con 0x7fd39c067e40 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c132ea0 con 0x7fd39c0a4360 2024-09-16T04:26:11.736 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd3a2a1e640 10 monclient(hunting): authenticate will time out at 1947.192924s 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd39bfff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fd39c067e40 0x7fd39c131160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39072/0 (socket says 172.21.15.171:39072) 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.687+0000 7fd39bfff640 1 -- 172.21.15.171:0/620881783 learned_addr learned my addr 172.21.15.171:0/620881783 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414873145 0 0) 0x7fd39c134a10 con 0x7fd39c067e40 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): my global_id is 164122 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd388003520 con 0x7fd39c067e40 2024-09-16T04:26:11.737 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134863319 0 0) 0x7fd39c13e030 con 0x7fd39c0a4810 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): my global_id is 144357 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd39c134a10 con 0x7fd39c0a4810 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 426631602 0 0) 0x7fd39c132ea0 con 0x7fd39c0a4360 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): my global_id is 154244 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd39c13e030 con 0x7fd39c0a4360 2024-09-16T04:26:11.738 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3221659881 0 0) 0x7fd388003520 con 0x7fd39c067e40 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd39c132ea0 con 0x7fd39c067e40 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd38c0018f0 con 0x7fd39c067e40 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): dump: 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.739 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3357969947 0 0) 0x7fd39c134a10 con 0x7fd39c0a4810 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd388003520 con 0x7fd39c0a4810 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd394001c30 con 0x7fd39c0a4810 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.740 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): dump: 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.741 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.742 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4055250353 0 0) 0x7fd39c13e030 con 0x7fd39c0a4360 2024-09-16T04:26:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd39c134a10 con 0x7fd39c0a4360 2024-09-16T04:26:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd390001a70 con 0x7fd39c0a4360 2024-09-16T04:26:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.742 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient(hunting): dump: 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.743 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3260937059 0 0) 0x7fd39c132ea0 con 0x7fd39c067e40 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 >> v1:172.21.15.148:6790/0 conn(0x7fd39c0a4360 legacy=0x7fd39c1d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 >> v1:172.21.15.148:6789/0 conn(0x7fd39c0a4810 legacy=0x7fd39c1d4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 monclient: found mon.b 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c13dd50 con 0x7fd39c067e40 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.744 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 5 monclient: authenticate success, global_id 164122 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 10 monclient: _renew_subs 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd39c131870 con 0x7fd39c067e40 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 10 monclient: _renew_subs 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd39c13c800 con 0x7fd39c067e40 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd38c001cc0 con 0x7fd39c067e40 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.745 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd38c003a60 con 0x7fd39c067e40 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd398ff9640 10 monclient: dump: 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.746 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.688+0000 7fd39a7fc640 4 set_mon_vals no callback set 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd38c003cc0 con 0x7fd39c067e40 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd3a2a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd3a2a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd3a2a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fd38c0478f0 con 0x7fd39c067e40 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd39c09fe40 con 0x7fd39c067e40 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.689+0000 7fd39b7fe640 10 monclient: get_auth_request con 0x7fd388039200 auth_method 0 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.694+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fd38c015af0 con 0x7fd39c067e40 2024-09-16T04:26:11.747 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.694+0000 7fd398ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.748 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.694+0000 7fd398ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:11.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.817+0000 7f59759ed640 10 monclient: get_monmap_and_config 2024-09-16T04:26:11.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.817+0000 7f59759ed640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.817+0000 7f59759ed640 10 monclient: monmap: 2024-09-16T04:26:11.835 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:11.818381+0000 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:11.818381+0000 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 1 Processor -- start 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 1 -- start start 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient: init 2024-09-16T04:26:11.836 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient(hunting): picked mon.noname-a con 0x7f59700a22b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient(hunting): picked mon.noname-c con 0x7f59700a1b70 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient(hunting): picked mon.noname-b con 0x7f597013d0d0 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59700a0440 con 0x7f59700a22b0 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59700a4d90 con 0x7f59700a1b70 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59700a08f0 con 0x7f597013d0d0 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f59759ed640 10 monclient(hunting): authenticate will time out at 1992.443571s 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596effd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f59700a22b0 0x7f59700a2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:34678/0 (socket says 172.21.15.148:34678) 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596effd640 1 -- 172.21.15.148:0/3839247445 learned_addr learned my addr 172.21.15.148:0/3839247445 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596f7fe640 1 --1- 172.21.15.148:0/3839247445 >> v1:172.21.15.148:6790/0 conn(0x7f597013d0d0 0x7f597013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39448/0 (socket says 172.21.15.148:39448) 2024-09-16T04:26:11.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743375255 0 0) 0x7f59700a0440 con 0x7f59700a22b0 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): my global_id is 144360 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f595c003520 con 0x7f59700a22b0 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843863365 0 0) 0x7f59700a08f0 con 0x7f597013d0d0 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 10 monclient(hunting): my global_id is 154247 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.818+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59700a0440 con 0x7f597013d0d0 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735181335 0 0) 0x7f59700a4d90 con 0x7f59700a1b70 2024-09-16T04:26:11.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient(hunting): my global_id is 164125 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59700a08f0 con 0x7f59700a1b70 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2640976311 0 0) 0x7f595c003520 con 0x7f59700a22b0 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 >> v1:172.21.15.148:6790/0 conn(0x7f597013d0d0 legacy=0x7f597013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 >> v1:172.21.15.171:6789/0 conn(0x7f59700a1b70 legacy=0x7f597013b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 monclient: found mon.noname-a 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59700a2da0 con 0x7f59700a22b0 2024-09-16T04:26:11.839 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 5 monclient: authenticate success, global_id 144360 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5968001990 con 0x7f59700a22b0 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: dump: 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.840 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5968001c30 con 0x7f59700a22b0 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 4 set_mon_vals no callback set 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 1 -- 172.21.15.148:0/3839247445 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5968002040 con 0x7f59700a22b0 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.841 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f596dffb640 10 monclient: dump: 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 10 monclient: shutdown 2024-09-16T04:26:11.842 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 1 -- 172.21.15.148:0/3839247445 >> v1:172.21.15.148:6789/0 conn(0x7f59700a22b0 legacy=0x7f59700a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 1 -- 172.21.15.148:0/3839247445 shutdown_connections 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 1 -- 172.21.15.148:0/3839247445 >> 172.21.15.148:0/3839247445 conn(0x7f59700732c0 msgr2=0x7f59700736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 1 -- 172.21.15.148:0/3839247445 shutdown_connections 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.819+0000 7f59759ed640 1 -- 172.21.15.148:0/3839247445 wait complete. 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.820+0000 7f59759ed640 10 monclient: build_initial_monmap 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.820+0000 7f59759ed640 10 monclient: monmap: 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:11.843 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.820+0000 7f59759ed640 1 Processor -- start 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.820+0000 7f59759ed640 1 -- start start 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.820+0000 7f59759ed640 10 monclient: init 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient(hunting): picked mon.noname-c con 0x7f59700a1b70 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient(hunting): picked mon.noname-b con 0x7f59700a22b0 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient(hunting): picked mon.noname-a con 0x7f597013d0d0 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:11.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597009f2a0 con 0x7f597013d0d0 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5970137840 con 0x7f59700a22b0 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597013c9f0 con 0x7f59700a1b70 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f59759ed640 10 monclient(hunting): authenticate will time out at 1992.446408s 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f596effd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f59700a1b70 0x7f597009caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39460/0 (socket says 172.21.15.148:39460) 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f596effd640 1 -- 172.21.15.148:0/612050949 learned_addr learned my addr 172.21.15.148:0/612050949 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2777195760 0 0) 0x7f597013c9f0 con 0x7f59700a1b70 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): my global_id is 154250 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5950003520 con 0x7f59700a1b70 2024-09-16T04:26:11.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955713439 0 0) 0x7f597009f2a0 con 0x7f597013d0d0 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): my global_id is 144363 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f597013c9f0 con 0x7f597013d0d0 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010612487 0 0) 0x7f5970137840 con 0x7f59700a22b0 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): my global_id is 164128 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f597009f2a0 con 0x7f59700a22b0 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3356294373 0 0) 0x7f5950003520 con 0x7f59700a1b70 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5970137840 con 0x7f59700a1b70 2024-09-16T04:26:11.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3030611890 0 0) 0x7f597013c9f0 con 0x7f597013d0d0 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5950003520 con 0x7f597013d0d0 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5968001890 con 0x7f59700a1b70 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): dump: 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.847 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59600017a0 con 0x7f597013d0d0 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): dump: 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.848 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2451224245 0 0) 0x7f597009f2a0 con 0x7f59700a22b0 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f597013c9f0 con 0x7f59700a22b0 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59580017f0 con 0x7f59700a22b0 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.821+0000 7f5967fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient(hunting): dump: 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.849 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1568589350 0 0) 0x7f5970137840 con 0x7f59700a1b70 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 >> v1:172.21.15.171:6789/0 conn(0x7f59700a22b0 legacy=0x7f597009d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 >> v1:172.21.15.148:6789/0 conn(0x7f597013d0d0 legacy=0x7f597009d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: _finish_hunting 0 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 monclient: found mon.c 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59701408b0 con 0x7f59700a1b70 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: _finish_auth 0 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: _check_auth_tickets 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 5 monclient: authenticate success, global_id 154250 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 10 monclient: _renew_subs 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5970137560 con 0x7f59700a1b70 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 10 monclient: _renew_subs 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5970137280 con 0x7f59700a1b70 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5968004f70 con 0x7f59700a1b70 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f59680052c0 con 0x7f59700a1b70 2024-09-16T04:26:11.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f5967fff640 10 monclient: dump: 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:11.852 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.822+0000 7f596d7fa640 4 set_mon_vals no callback set 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5968010550 con 0x7f59700a1b70 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f59759ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f59759ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f59759ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5970137a80 con 0x7f59700a1b70 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f5968047bd0 con 0x7f59700a1b70 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.823+0000 7f596e7fc640 10 monclient: get_auth_request con 0x7f5950039180 auth_method 0 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.828+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5968015f50 con 0x7f59700a1b70 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.828+0000 7f5967fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:11.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:11.828+0000 7f5967fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:11.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.923+0000 7fd3a2a1e640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T04:26:11.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.923+0000 7fd3a2a1e640 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-16T04:26:11.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.923+0000 7fd3a2a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:11.925 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.923+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth import"} v 0) -- 0x7fd39c06bc10 con 0x7fd39c067e40 2024-09-16T04:26:11.926 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:11.924+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-16T04:26:11.928 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.927+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "auth import"}]=0 v28) ==== 59+0+0 (unknown 126837407 0 0) 0x7fd38c0197a0 con 0x7fd39c067e40 2024-09-16T04:26:11.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.927+0000 7fd398ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-16T04:26:11.929 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:11.927+0000 7fd398ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.057+0000 7f59759ed640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:26:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.057+0000 7f59759ed640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:26:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.057+0000 7f59759ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.057+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5970134ff0 con 0x7f59700a1b70 2024-09-16T04:26:12.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f5967fff640 1 -- 172.21.15.148:0/612050949 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f5968015850 con 0x7f59700a1b70 2024-09-16T04:26:12.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f5967fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:26:12.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f5967fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=378235851,config=692619927,health=3943369176,kv=2270513034,logm=892481479}) 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=3723257892,config=692619927,health=771768438,kv=2360976541,logm=3412709310}) 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=378235851,config=692619927,health=3943369176,kv=2270513034,logm=892481479}) 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.058+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=1,health=11,kv=10,logm=52} crc {auth=3449109966,config=3678701417,health=3943369176,kv=2270513034,logm=1662903016}) 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.059+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.059+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736625934}) 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.059+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609826136}) 2024-09-16T04:26:12.061 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.059+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.062 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.059+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736625934}) 2024-09-16T04:26:12.062 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.059+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652111734}) 2024-09-16T04:26:12.063 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.060+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.063 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.060+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269846935}) 2024-09-16T04:26:12.063 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.060+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=512628439}) 2024-09-16T04:26:12.063 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.060+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.063 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.060+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269846935}) 2024-09-16T04:26:12.063 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.060+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876066751}) 2024-09-16T04:26:12.064 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.064 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275637984}) 2024-09-16T04:26:12.064 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3164932926}) 2024-09-16T04:26:12.064 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.064 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275637984}) 2024-09-16T04:26:12.064 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252991455}) 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 >> v1:172.21.15.171:6812/7327273 conn(0x7f5950039180 legacy=0x7f595003b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 10 monclient: shutdown 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 >> v1:172.21.15.148:6790/0 conn(0x7f59700a1b70 legacy=0x7f597009caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 shutdown_connections 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 >> 172.21.15.148:0/612050949 conn(0x7f59700732c0 msgr2=0x7f597013fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 shutdown_connections 2024-09-16T04:26:12.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f59759ed640 1 -- 172.21.15.148:0/612050949 wait complete. 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927668757}) 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909752330}) 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2927668757}) 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.061+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2528716906}) 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.062+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.062+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160023737}) 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.062+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2028855093}) 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.062+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.068 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.062+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160023737}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.062+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768773480}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.063+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.063+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920245471}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.063+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12122351}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.063+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.063+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920245471}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.063+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3994282396}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.064+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.064+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254592639}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.064+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219318701}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.064+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.064+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254592639}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.064+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325604058}) 2024-09-16T04:26:12.069 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.065+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.065+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405463728}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.065+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1477428788}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.065+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.065+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405463728}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.065+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=218545555}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.066+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.066+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780202261}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.066+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088152734}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.066+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.066+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780202261}) 2024-09-16T04:26:12.070 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.066+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433140470}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.067+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.067+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711572848}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.067+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356690532}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.067+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.067+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711572848}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.067+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360635986}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93838288}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952994304}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93838288}) 2024-09-16T04:26:12.073 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051572616}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.068+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311528051}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3697723622}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311528051}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927672102}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750149329}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848656938}) 2024-09-16T04:26:12.074 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.075 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750149329}) 2024-09-16T04:26:12.075 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.069+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245099032}) 2024-09-16T04:26:12.075 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.070+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.075 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.070+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404346556}) 2024-09-16T04:26:12.075 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.070+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070380224}) 2024-09-16T04:26:12.075 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.070+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.070+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404346556}) 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.070+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675184526}) 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.071+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.071+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284518547}) 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.071+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366402546}) 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.071+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.071+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284518547}) 2024-09-16T04:26:12.076 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.071+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574316593}) 2024-09-16T04:26:12.077 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:26:12.081 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.072+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.081 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.072+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10431084}) 2024-09-16T04:26:12.081 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.072+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1684059282}) 2024-09-16T04:26:12.081 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.072+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.072+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10431084}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.072+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2809873040}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.073+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.073+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511522637}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.073+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367167104}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.073+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.073+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511522637}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.073+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188764938}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.074+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.074+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090635053}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.074+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3290467398}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.074+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.074+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090635053}) 2024-09-16T04:26:12.082 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.074+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=30708456}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.075+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.075+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936246900}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.075+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4038441193}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.075+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.075+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936246900}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.075+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308270311}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.076+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.076+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266870697}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.076+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678182890}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.076+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.076+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266870697}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.076+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295464432}) 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.077+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.083 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.077+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989384203}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.077+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3179292088}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.077+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.077+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989384203}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.077+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040753949}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867142228}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639518608}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1867142228}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395646000}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494226553}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872602415}) 2024-09-16T04:26:12.084 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.078+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494226553}) 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353614115}) 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667082554}) 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128296329}) 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667082554}) 2024-09-16T04:26:12.085 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.079+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2600891559}) 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.080+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.080+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488320229}) 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.080+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3896183123}) 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.080+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.080+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488320229}) 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.080+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015830197}) 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.081+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.081+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135916997}) 2024-09-16T04:26:12.090 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.081+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021207444}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.081+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.081+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135916997}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.081+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562697463}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.082+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.082+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366676466}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.082+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573215227}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.082+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.082+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366676466}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.082+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2325539687}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.083+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.083+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389648019}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.083+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787026402}) 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.083+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.091 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.083+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389648019}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.083+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1120832003}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.084+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.084+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434732625}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.084+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=416656638}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.084+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.084+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434732625}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.084+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967706124}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.085+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.085+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193948973}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.085+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277915498}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.085+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.085+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193948973}) 2024-09-16T04:26:12.092 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.085+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2276564924}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612677951}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1100207134}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612677951}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749889795}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707443558}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447264754}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707443558}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.086+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=601893098}) 2024-09-16T04:26:12.093 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.087+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.087+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885357962}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.087+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=391945122}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.087+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.087+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885357962}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.087+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538994381}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.088+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.088+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=351466405}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.088+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1510808907}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.088+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.088+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=351466405}) 2024-09-16T04:26:12.094 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.088+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582251910}) 2024-09-16T04:26:12.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.098+0000 7fd3a2a1e640 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-16T04:26:12.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.098+0000 7fd3a2a1e640 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-16T04:26:12.102 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.098+0000 7fd3a2a1e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.103 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.098+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth import"} v 0) -- 0x7fd39c09db90 con 0x7fd39c067e40 2024-09-16T04:26:12.105 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.089+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.089+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505701117}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.089+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384849076}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.089+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.089+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505701117}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.089+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952724817}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.090+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.090+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276307275}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.090+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=391605727}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.090+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.090+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276307275}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.090+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3341414330}) 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.091+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.106 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.091+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034719331}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.091+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343761054}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.091+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.091+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034719331}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.091+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3433373865}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688434399}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3870309322}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688434399}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153829589}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1483627619}) 2024-09-16T04:26:12.107 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10967437}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1483627619}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.092+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968033178}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.093+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.093+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720766097}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.093+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3831109739}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.093+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.093+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720766097}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.093+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1132072831}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.094+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.094+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024605575}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.094+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550377564}) 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.094+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.108 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.094+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024605575}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.094+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=569787843}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.095+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.095+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333140071}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.095+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58867189}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.095+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.095+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333140071}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.095+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201150344}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.096+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.096+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86717747}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.096+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1213612491}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.096+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.096+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86717747}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.096+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975400868}) 2024-09-16T04:26:12.109 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.097+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.097+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331041157}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.097+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002433552}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.097+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.097+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331041157}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.097+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3480551271}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387114308}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221302405}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387114308}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322347889}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821383620}) 2024-09-16T04:26:12.110 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.098+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191910765}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821383620}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=29367157}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800119482}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279419083}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800119482}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.099+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742428180}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.100+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.100+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227050826}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.100+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966454878}) 2024-09-16T04:26:12.111 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.100+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.100+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227050826}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.100+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274094596}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.101+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.101+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240320603}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.101+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820193817}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.101+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.101+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240320603}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.101+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320942728}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.102+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.102+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484777062}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.102+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197248581}) 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.102+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.112 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.102+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484777062}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.102+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1525898483}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756026746}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2131035034}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756026746}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439219487}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1953628521}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=564105762}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1953628521}) 2024-09-16T04:26:12.113 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.103+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918224257}) 2024-09-16T04:26:12.114 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.104+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.114 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.104+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066598877}) 2024-09-16T04:26:12.114 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.104+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90124847}) 2024-09-16T04:26:12.114 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.104+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.114 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.104+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066598877}) 2024-09-16T04:26:12.114 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.104+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329939858}) 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244566665}) 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107851406}) 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244566665}) 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226415577}) 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.120 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942782116}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167826744}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942782116}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.105+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194952860}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619850853}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101429989}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619850853}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374787444}) 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.121 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436391152}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160199583}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436391152}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.106+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766559038}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683875225}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=653822334}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683875225}) 2024-09-16T04:26:12.122 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213338944}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=197174943}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843158364}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.107+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=197174943}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508495908}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101615164}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4291531163}) 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.123 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101615164}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.108+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311878326}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410534013}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938155265}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410534013}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=10676380}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354007584}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2087922831}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354007584}) 2024-09-16T04:26:12.124 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.109+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264342944}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.110+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.110+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1496705852,mdsmap=1097804592,mgr=3429773186}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.110+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=3867939395,mdsmap=1097804592,mgr=1586861337}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.110+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.110+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1496705852,mdsmap=1097804592,mgr=3429773186}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.110+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1652414651,mdsmap=1097804592,mgr=3829632668}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.111+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.111+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=1747058506,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2170579518,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=269952690}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.111+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=1747058506,mgr_metadata=582288965,mgrstat=694991710,mon_config_key=4229131226,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=1957783866}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.111+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.111+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgr=1747058506,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=2170579518,mon_config_key=2589496441,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=269952690}) 2024-09-16T04:26:12.125 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:26:12.111+0000 7f3bfbd52640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgr=1747058506,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3891080661,mon_config_key=2170886992,monmap=2633035915,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=841190635,osdmap=4104974758}) 2024-09-16T04:26:12.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.119+0000 7fd398ff9640 1 -- 172.21.15.171:0/620881783 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "auth import"}]=0 v29) ==== 59+0+0 (unknown 869199659 0 0) 0x7fd38c01e060 con 0x7fd39c067e40 2024-09-16T04:26:12.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.119+0000 7fd398ff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-16T04:26:12.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.119+0000 7fd398ff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:26:12.126 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 >> v1:172.21.15.171:6812/7327273 conn(0x7fd388039200 legacy=0x7fd38803b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 10 monclient: shutdown 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 >> v1:172.21.15.171:6789/0 conn(0x7fd39c067e40 legacy=0x7fd39c131160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 shutdown_connections 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 >> 172.21.15.171:0/620881783 conn(0x7fd39c0a5d20 msgr2=0x7fd39c0a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 shutdown_connections 2024-09-16T04:26:12.127 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.121+0000 7fd3a2a1e640 1 -- 172.21.15.171:0/620881783 wait complete. 2024-09-16T04:26:12.132 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph health 2024-09-16T04:26:12.133 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep AUTH_BAD_CAP 2024-09-16T04:26:12.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.203+0000 7ff4263e8640 10 monclient: get_monmap_and_config 2024-09-16T04:26:12.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.203+0000 7ff4263e8640 10 monclient: build_initial_monmap 2024-09-16T04:26:12.204 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.203+0000 7ff4263e8640 10 monclient: monmap: 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:26:12.203964+0000 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:26:12.203964+0000 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:12.205 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.203+0000 7ff4263e8640 1 Processor -- start 2024-09-16T04:26:12.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 1 -- start start 2024-09-16T04:26:12.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient: init 2024-09-16T04:26:12.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:12.212 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff4200a4c80 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff42009faa0 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff4200a5770 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4200a10a0 con 0x7ff4200a5770 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4200a0220 con 0x7ff42009faa0 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff420074130 con 0x7ff4200a4c80 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff4263e8640 10 monclient(hunting): authenticate will time out at 1947.709591s 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41f7fe640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff42009faa0 0x7ff42013bef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39084/0 (socket says 172.21.15.171:39084) 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41f7fe640 1 -- 172.21.15.171:0/4261401745 learned_addr learned my addr 172.21.15.171:0/4261401745 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4210776197 0 0) 0x7ff4200a0220 con 0x7ff42009faa0 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.213 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41effd640 10 monclient(hunting): my global_id is 164131 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.204+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff40c003520 con 0x7ff42009faa0 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2337950914 0 0) 0x7ff420074130 con 0x7ff4200a4c80 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient(hunting): my global_id is 154253 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4200a0220 con 0x7ff4200a4c80 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1184978240 0 0) 0x7ff40c003520 con 0x7ff42009faa0 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 >> v1:172.21.15.148:6790/0 conn(0x7ff4200a4c80 legacy=0x7ff4200a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 >> v1:172.21.15.148:6789/0 conn(0x7ff4200a5770 legacy=0x7ff42013f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.214 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: _finish_hunting 0 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 monclient: found mon.noname-c 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff420072850 con 0x7ff42009faa0 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: _finish_auth 0 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: _check_auth_tickets 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff408001610 con 0x7ff42009faa0 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 5 monclient: authenticate success, global_id 164131 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: dump: 2024-09-16T04:26:12.215 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff408001870 con 0x7ff42009faa0 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 1 -- 172.21.15.171:0/4261401745 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff408001c00 con 0x7ff42009faa0 2024-09-16T04:26:12.216 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 4 set_mon_vals no callback set 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff41effd640 10 monclient: dump: 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.217 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 10 monclient: shutdown 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 1 -- 172.21.15.171:0/4261401745 >> v1:172.21.15.171:6789/0 conn(0x7ff42009faa0 legacy=0x7ff42013bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 1 -- 172.21.15.171:0/4261401745 shutdown_connections 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 1 -- 172.21.15.171:0/4261401745 >> 172.21.15.171:0/4261401745 conn(0x7ff4200a98a0 msgr2=0x7ff4200a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 1 -- 172.21.15.171:0/4261401745 shutdown_connections 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.205+0000 7ff4263e8640 1 -- 172.21.15.171:0/4261401745 wait complete. 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.206+0000 7ff4263e8640 10 monclient: build_initial_monmap 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.206+0000 7ff4263e8640 10 monclient: monmap: 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:26:12.218 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.206+0000 7ff4263e8640 1 Processor -- start 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.206+0000 7ff4263e8640 1 -- start start 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.206+0000 7ff4263e8640 10 monclient: init 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff42009faa0 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:12.219 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff4200a4c80 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff4200a5770 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42013fd50 con 0x7ff4200a4c80 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42013ee60 con 0x7ff42009faa0 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff420140030 con 0x7ff4200a5770 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff4263e8640 10 monclient(hunting): authenticate will time out at 1947.712421s 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41ffff640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7ff42009faa0 0x7ff4201cbe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39094/0 (socket says 172.21.15.171:39094) 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41ffff640 1 -- 172.21.15.171:0/4133711250 learned_addr learned my addr 172.21.15.171:0/4133711250 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040783032 0 0) 0x7ff42013ee60 con 0x7ff42009faa0 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.220 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): my global_id is 164134 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff400003520 con 0x7ff42009faa0 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89538099 0 0) 0x7ff420140030 con 0x7ff4200a5770 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): my global_id is 154256 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff42013ee60 con 0x7ff4200a5770 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1570200742 0 0) 0x7ff42013fd50 con 0x7ff4200a4c80 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): my global_id is 144369 2024-09-16T04:26:12.221 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff420140030 con 0x7ff4200a4c80 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2503804009 0 0) 0x7ff400003520 con 0x7ff42009faa0 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff42013fd50 con 0x7ff42009faa0 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff410001970 con 0x7ff42009faa0 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): dump: 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.222 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2999333494 0 0) 0x7ff42013ee60 con 0x7ff4200a5770 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff400003520 con 0x7ff4200a5770 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff418001d80 con 0x7ff4200a5770 2024-09-16T04:26:12.223 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.207+0000 7ff41cff9640 10 monclient(hunting): dump: 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.224 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1140450210 0 0) 0x7ff420140030 con 0x7ff4200a4c80 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff42013ee60 con 0x7ff4200a4c80 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff408001370 con 0x7ff4200a4c80 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient(hunting): dump: 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.225 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2379933986 0 0) 0x7ff42013fd50 con 0x7ff42009faa0 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 >> v1:172.21.15.148:6790/0 conn(0x7ff4200a5770 legacy=0x7ff4201d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 >> v1:172.21.15.148:6789/0 conn(0x7ff4200a4c80 legacy=0x7ff4201d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: _finish_hunting 0 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 monclient: found mon.b 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42009ecc0 con 0x7ff42009faa0 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: _finish_auth 0 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:26:12.226 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 5 monclient: authenticate success, global_id 164134 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 10 monclient: _renew_subs 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff4200a1aa0 con 0x7ff42009faa0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 10 monclient: _renew_subs 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff420140a10 con 0x7ff42009faa0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff410001d20 con 0x7ff42009faa0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff410003a60 con 0x7ff42009faa0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41dffb640 4 set_mon_vals no callback set 2024-09-16T04:26:12.227 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 10 monclient: dump: 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.208+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff410003a60 con 0x7ff42009faa0 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.209+0000 7ff4263e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:12.228 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.209+0000 7ff4263e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.209+0000 7ff4263e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.209+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff404005080 con 0x7ff42009faa0 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.209+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7ff4100478e0 con 0x7ff42009faa0 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.209+0000 7ff41f7fe640 10 monclient: get_auth_request con 0x7ff400039190 auth_method 0 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.213+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff410015b40 con 0x7ff42009faa0 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.213+0000 7ff41cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:12.229 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.213+0000 7ff41cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff4263e8640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T04:26:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff4263e8640 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-16T04:26:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff4263e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "health"} v 0) -- 0x7ff404005b80 con 0x7ff42009faa0 2024-09-16T04:26:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "health"}]=0 v0) ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7ff410015440 con 0x7ff42009faa0 2024-09-16T04:26:12.421 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff41cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-16T04:26:12.422 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.419+0000 7ff41cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:12.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff4263e8640 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-16T04:26:12.551 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff4263e8640 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-16T04:26:12.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff4263e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "health"} v 0) -- 0x7ff4040057b0 con 0x7ff42009faa0 2024-09-16T04:26:12.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff41cff9640 1 -- 172.21.15.171:0/4133711250 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "health"}]=0 v0) ==== 54+0+99 (unknown 2555570491 0 746166484) 0x7ff41001dff0 con 0x7ff42009faa0 2024-09-16T04:26:12.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff41cff9640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-16T04:26:12.552 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.550+0000 7ff41cff9640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:26:12.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 >> v1:172.21.15.171:6812/7327273 conn(0x7ff400039190 legacy=0x7ff40003b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.553 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 10 monclient: shutdown 2024-09-16T04:26:12.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:12.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 >> v1:172.21.15.171:6789/0 conn(0x7ff42009faa0 legacy=0x7ff4201cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 shutdown_connections 2024-09-16T04:26:12.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 >> 172.21.15.171:0/4133711250 conn(0x7ff4200a98a0 msgr2=0x7ff4200a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:12.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 shutdown_connections 2024-09-16T04:26:12.554 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.552+0000 7ff4263e8640 1 -- 172.21.15.171:0/4133711250 wait complete. 2024-09-16T04:26:12.563 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph health detail 2024-09-16T04:26:12.563 INFO:tasks.workunit.client.0.smithi171.stderr:+ grep client.bad 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.634+0000 7fe37b8d7640 10 monclient: get_monmap_and_config 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.634+0000 7fe37b8d7640 10 monclient: build_initial_monmap 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.634+0000 7fe37b8d7640 10 monclient: monmap: 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:26:12.635546+0000 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:26:12.635546+0000 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:12.636 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.637 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:12.637 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:12.637 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:12.637 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.635+0000 7fe37b8d7640 1 Processor -- start 2024-09-16T04:26:12.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.635+0000 7fe37b8d7640 1 -- start start 2024-09-16T04:26:12.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.635+0000 7fe37b8d7640 10 monclient: init 2024-09-16T04:26:12.644 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.635+0000 7fe37b8d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.635+0000 7fe37b8d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.635+0000 7fe37b8d7640 10 monclient(hunting): picked mon.noname-b con 0x7fe3740736b0 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe374133100 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe374139680 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37409e510 con 0x7fe374139680 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37409e7f0 con 0x7fe374133100 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37409ead0 con 0x7fe3740736b0 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 10 monclient(hunting): authenticate will time out at 1948.141053s 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe378e72640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe374133100 0x7fe374137f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39108/0 (socket says 172.21.15.171:39108) 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe378e72640 1 -- 172.21.15.171:0/266003887 learned_addr learned my addr 172.21.15.171:0/266003887 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:12.645 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 938353679 0 0) 0x7fe37409e7f0 con 0x7fe374133100 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): my global_id is 164137 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe360004630 con 0x7fe374133100 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906259341 0 0) 0x7fe37409e510 con 0x7fe374139680 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.646 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): my global_id is 144372 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe37409e7f0 con 0x7fe374139680 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955061429 0 0) 0x7fe37409ead0 con 0x7fe3740736b0 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient(hunting): my global_id is 154259 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe37409e510 con 0x7fe3740736b0 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1528052584 0 0) 0x7fe360004630 con 0x7fe374133100 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 >> v1:172.21.15.148:6790/0 conn(0x7fe3740736b0 legacy=0x7fe374073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 >> v1:172.21.15.148:6789/0 conn(0x7fe374139680 legacy=0x7fe37413ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 monclient: found mon.noname-c 2024-09-16T04:26:12.647 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3740a0650 con 0x7fe374133100 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: _finish_auth 0 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe370001610 con 0x7fe374133100 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 5 monclient: authenticate success, global_id 164137 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: dump: 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.648 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe370001930 con 0x7fe374133100 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 1 -- 172.21.15.171:0/266003887 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe370001d00 con 0x7fe374133100 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe37b8d7640 4 set_mon_vals no callback set 2024-09-16T04:26:12.649 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.636+0000 7fe36bfff640 10 monclient: dump: 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 10 monclient: shutdown 2024-09-16T04:26:12.650 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 1 -- 172.21.15.171:0/266003887 >> v1:172.21.15.171:6789/0 conn(0x7fe374133100 legacy=0x7fe374137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 1 -- 172.21.15.171:0/266003887 shutdown_connections 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 1 -- 172.21.15.171:0/266003887 >> 172.21.15.171:0/266003887 conn(0x7fe3740a52c0 msgr2=0x7fe3740a56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 1 -- 172.21.15.171:0/266003887 shutdown_connections 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 1 -- 172.21.15.171:0/266003887 wait complete. 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 10 monclient: build_initial_monmap 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.637+0000 7fe37b8d7640 10 monclient: monmap: 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:12.651 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 1 Processor -- start 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 1 -- start start 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient: init 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient(hunting): picked mon.noname-a con 0x7fe3740736b0 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient(hunting): picked mon.noname-c con 0x7fe374133100 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient(hunting): picked mon.noname-b con 0x7fe374139680 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe374136fd0 con 0x7fe3740736b0 2024-09-16T04:26:12.652 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3741384f0 con 0x7fe374139680 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe374138210 con 0x7fe374133100 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe37b8d7640 10 monclient(hunting): authenticate will time out at 1948.143694s 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe379e74640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7fe374139680 0x7fe3741d0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39112/0 (socket says 172.21.15.171:39112) 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe379e74640 1 -- 172.21.15.171:0/4168949338 learned_addr learned my addr 172.21.15.171:0/4168949338 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582835523 0 0) 0x7fe3741384f0 con 0x7fe374139680 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe369ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe369ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe369ffb640 10 monclient(hunting): my global_id is 164140 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.638+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe358003520 con 0x7fe374139680 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511381628 0 0) 0x7fe374138210 con 0x7fe374133100 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.653 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): my global_id is 154262 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3741384f0 con 0x7fe374133100 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2262446482 0 0) 0x7fe374136fd0 con 0x7fe3740736b0 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): my global_id is 144375 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe374138210 con 0x7fe3740736b0 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2620991185 0 0) 0x7fe358003520 con 0x7fe374139680 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe374136fd0 con 0x7fe374139680 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe36c001c30 con 0x7fe374139680 2024-09-16T04:26:12.654 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): dump: 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:12.655 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4036291983 0 0) 0x7fe3741384f0 con 0x7fe374133100 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe358003520 con 0x7fe374133100 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe3700015d0 con 0x7fe374133100 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): dump: 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.656 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1070410835 0 0) 0x7fe374138210 con 0x7fe3740736b0 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe3741384f0 con 0x7fe3740736b0 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe364001740 con 0x7fe3740736b0 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient(hunting): dump: 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.657 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3674311604 0 0) 0x7fe374136fd0 con 0x7fe374139680 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 >> v1:172.21.15.148:6790/0 conn(0x7fe374133100 legacy=0x7fe3741cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 >> v1:172.21.15.148:6789/0 conn(0x7fe3740736b0 legacy=0x7fe3741c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 monclient: found mon.b 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe374137f30 con 0x7fe374139680 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: _finish_auth 0 2024-09-16T04:26:12.658 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 5 monclient: authenticate success, global_id 164140 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 10 monclient: _renew_subs 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe3741359d0 con 0x7fe374139680 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe36c003990 con 0x7fe374139680 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe36c003d60 con 0x7fe374139680 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe369ffb640 10 monclient: dump: 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:12.659 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe36b7fe640 4 set_mon_vals no callback set 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 10 monclient: _renew_subs 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.639+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe3741c8180 con 0x7fe374139680 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe36c003d60 con 0x7fe374139680 2024-09-16T04:26:12.660 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fe36c047950 con 0x7fe374139680 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe379673640 10 monclient: get_auth_request con 0x7fe3580391f0 auth_method 0 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe37b8d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe37b8d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe37b8d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.640+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37409c0c0 con 0x7fe374139680 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.645+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe36c015c30 con 0x7fe374139680 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.645+0000 7fe369ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:12.661 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.645+0000 7fe369ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:12.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.873+0000 7fe37b8d7640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T04:26:12.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.873+0000 7fe37b8d7640 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T04:26:12.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.873+0000 7fe37b8d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:12.875 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.873+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) -- 0x7fe37409c6c0 con 0x7fe374139680 2024-09-16T04:26:12.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.874+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7fe36c0198e0 con 0x7fe374139680 2024-09-16T04:26:12.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.874+0000 7fe369ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-16T04:26:12.876 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:12.874+0000 7fe369ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe37b8d7640 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe37b8d7640 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe37b8d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "health", "detail": "detail"} v 0) -- 0x7fe37409d500 con 0x7fe374139680 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe369ffb640 1 -- 172.21.15.171:0/4168949338 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) ==== 74+0+1041 (unknown 501213350 0 2456344155) 0x7fe36c01e1a0 con 0x7fe374139680 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe369ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-16T04:26:13.038 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.036+0000 7fe369ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:26:13.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.038+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 >> v1:172.21.15.171:6812/7327273 conn(0x7fe3580391f0 legacy=0x7fe35803b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.038+0000 7fe37b8d7640 10 monclient: shutdown 2024-09-16T04:26:13.040 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.038+0000 7fe37b8d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.038+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 >> v1:172.21.15.171:6789/0 conn(0x7fe374139680 legacy=0x7fe3741d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.039+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 shutdown_connections 2024-09-16T04:26:13.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.039+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 >> 172.21.15.171:0/4168949338 conn(0x7fe3740a52c0 msgr2=0x7fe37412ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.039+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 shutdown_connections 2024-09-16T04:26:13.041 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.039+0000 7fe37b8d7640 1 -- 172.21.15.171:0/4168949338 wait complete. 2024-09-16T04:26:13.049 INFO:tasks.workunit.client.0.smithi171.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T04:26:13.049 INFO:tasks.workunit.client.0.smithi171.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T04:26:13.049 INFO:tasks.workunit.client.0.smithi171.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T04:26:13.049 INFO:tasks.workunit.client.0.smithi171.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-16T04:26:13.049 INFO:tasks.workunit.client.0.smithi171.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-16T04:26:13.050 INFO:tasks.workunit.client.0.smithi171.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-16T04:26:13.050 INFO:tasks.workunit.client.0.smithi171.stderr:+ ceph auth rm client.bad 2024-09-16T04:26:13.078 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:26:13.078 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:26:13.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.122+0000 7f51972c0640 10 monclient: get_monmap_and_config 2024-09-16T04:26:13.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.122+0000 7f51972c0640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.124 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.122+0000 7f51972c0640 10 monclient: monmap: 2024-09-16T04:26:13.124 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:26:13.123376+0000 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:26:13.123376+0000 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 1 Processor -- start 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 1 -- start start 2024-09-16T04:26:13.125 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient: init 2024-09-16T04:26:13.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:26:13.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient(hunting): picked mon.noname-a con 0x7f519009b790 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient(hunting): picked mon.noname-b con 0x7f5190136760 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.135 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient(hunting): picked mon.noname-c con 0x7f519013a270 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519009d9b0 con 0x7f519009b790 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519009e070 con 0x7f519013a270 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519009dbf0 con 0x7f5190136760 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f51972c0640 10 monclient(hunting): authenticate will time out at 1948.628863s 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.123+0000 7f519585d640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f519013a270 0x7f519013c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39122/0 (socket says 172.21.15.171:39122) 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f519585d640 1 -- 172.21.15.171:0/4076506875 learned_addr learned my addr 172.21.15.171:0/4076506875 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:13.136 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4269991826 0 0) 0x7f519009e070 con 0x7f519013a270 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): my global_id is 164143 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f517c003520 con 0x7f519013a270 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490558927 0 0) 0x7f519009dbf0 con 0x7f5190136760 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): my global_id is 154265 2024-09-16T04:26:13.137 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f519009e070 con 0x7f5190136760 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3236234273 0 0) 0x7f519009d9b0 con 0x7f519009b790 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient(hunting): my global_id is 144378 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f519009dbf0 con 0x7f519009b790 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2685465472 0 0) 0x7f517c003520 con 0x7f519013a270 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 >> v1:172.21.15.148:6790/0 conn(0x7f5190136760 legacy=0x7f5190138b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 >> v1:172.21.15.148:6789/0 conn(0x7f519009b790 legacy=0x7f519009bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 monclient: found mon.noname-c 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.138 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519009f0a0 con 0x7f519013a270 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f51972c0640 5 monclient: authenticate success, global_id 164143 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5188001990 con 0x7f519013a270 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: dump: 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.139 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f51972c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5188001c40 con 0x7f519013a270 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 1 -- 172.21.15.171:0/4076506875 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5188002010 con 0x7f519013a270 2024-09-16T04:26:13.140 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f51972c0640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f51972c0640 4 set_mon_vals no callback set 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.124+0000 7f5187fff640 10 monclient: dump: 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.141 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 10 monclient: shutdown 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 1 -- 172.21.15.171:0/4076506875 >> v1:172.21.15.171:6789/0 conn(0x7f519013a270 legacy=0x7f519013c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 1 -- 172.21.15.171:0/4076506875 shutdown_connections 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 1 -- 172.21.15.171:0/4076506875 >> 172.21.15.171:0/4076506875 conn(0x7f51900a5570 msgr2=0x7f51900a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 1 -- 172.21.15.171:0/4076506875 shutdown_connections 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 1 -- 172.21.15.171:0/4076506875 wait complete. 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 10 monclient: monmap: 2024-09-16T04:26:13.142 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 0 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 0.000000 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:created 0.000000 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.143 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.125+0000 7f51972c0640 1 Processor -- start 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 1 -- start start 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient: init 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient(hunting): picked mon.noname-b con 0x7f519009b790 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient(hunting): picked mon.noname-c con 0x7f5190136760 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient(hunting): picked mon.noname-a con 0x7f519013a270 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5190132d80 con 0x7f519013a270 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519012cc90 con 0x7f519009b790 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f519012cf70 con 0x7f5190136760 2024-09-16T04:26:13.144 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f51972c0640 10 monclient(hunting): authenticate will time out at 1948.631549s 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f519505c640 1 --1- >> v1:172.21.15.171:6789/0 conn(0x7f519009b790 0x7f519012aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.171:39134/0 (socket says 172.21.15.171:39134) 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f519505c640 1 -- 172.21.15.171:0/817253277 learned_addr learned my addr 172.21.15.171:0/817253277 (peer_addr_for_me v1:172.21.15.171:0/0) 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1070433262 0 0) 0x7f519012cc90 con 0x7f519009b790 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): my global_id is 164146 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5174003520 con 0x7f519009b790 2024-09-16T04:26:13.145 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 547931535 0 0) 0x7f519012cf70 con 0x7f5190136760 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): my global_id is 154268 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f519012cc90 con 0x7f5190136760 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810471836 0 0) 0x7f5190132d80 con 0x7f519013a270 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.146 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 10 monclient(hunting): my global_id is 144381 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f519012cf70 con 0x7f519013a270 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.126+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2973762414 0 0) 0x7f5174003520 con 0x7f519009b790 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5190132d80 con 0x7f519009b790 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5180001990 con 0x7f519009b790 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.147 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): dump: 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.148 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2874882577 0 0) 0x7f519012cc90 con 0x7f5190136760 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5174003520 con 0x7f5190136760 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f518c0018b0 con 0x7f5190136760 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.149 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): dump: 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.150 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3558908885 0 0) 0x7f519012cf70 con 0x7f519013a270 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f519012cc90 con 0x7f519013a270 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5188001df0 con 0x7f519013a270 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient(hunting): dump: 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.151 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1633626612 0 0) 0x7f5190132d80 con 0x7f519009b790 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 >> v1:172.21.15.148:6790/0 conn(0x7f5190136760 legacy=0x7f51901d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 >> v1:172.21.15.148:6789/0 conn(0x7f519013a270 legacy=0x7f51901d3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 monclient: found mon.b 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51900730d0 con 0x7f519009b790 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.152 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 5 monclient: authenticate success, global_id 164146 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 10 monclient: _renew_subs 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f519012c6d0 con 0x7f519009b790 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51800038c0 con 0x7f519009b790 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5180003c90 con 0x7f519009b790 2024-09-16T04:26:13.153 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 10 monclient: dump: 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:epoch 1 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:election_strategy: 1 2024-09-16T04:26:13.154 INFO:tasks.workunit.client.0.smithi171.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr: 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 10 monclient: _renew_subs 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51877fe640 4 set_mon_vals no callback set 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f519012b7e0 con 0x7f519009b790 2024-09-16T04:26:13.155 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.127+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5180003f30 con 0x7f519009b790 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.128+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f5180047870 con 0x7f519009b790 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.128+0000 7f51972c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.128+0000 7f51972c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.128+0000 7f51972c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.128+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f519009c310 con 0x7f519009b790 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.128+0000 7f519485b640 10 monclient: get_auth_request con 0x7f51740391f0 auth_method 0 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.132+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5180015bb0 con 0x7f519009b790 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.132+0000 7f5185ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.156 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.132+0000 7f5185ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:13.183 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.182+0000 7f5e92cba640 10 monclient: get_monmap_and_config 2024-09-16T04:26:13.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.182+0000 7f5e92cba640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.184 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.182+0000 7f5e92cba640 10 monclient: monmap: 2024-09-16T04:26:13.184 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.184 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.184 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:13.183026+0000 2024-09-16T04:26:13.184 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:13.183026+0000 2024-09-16T04:26:13.185 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.185 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.185 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.185 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:13.185 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:13.185 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.201 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.182+0000 7f5e92cba640 1 Processor -- start 2024-09-16T04:26:13.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.182+0000 7f5e92cba640 1 -- start start 2024-09-16T04:26:13.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.182+0000 7f5e92cba640 10 monclient: init 2024-09-16T04:26:13.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:26:13.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.202 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c0a0e00 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c0a10e0 con 0x7f5e8c13d060 2024-09-16T04:26:13.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c0a4970 con 0x7f5e8c0a2410 2024-09-16T04:26:13.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.203 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e92cba640 10 monclient(hunting): authenticate will time out at 1993.808375s 2024-09-16T04:26:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5e8c0a1b10 0x7f5e8c13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37216/0 (socket says 172.21.15.148:37216) 2024-09-16T04:26:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e90a56640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f5e8c0a2410 0x7f5e8c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56606/0 (socket says 172.21.15.148:56606) 2024-09-16T04:26:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8bfff640 1 -- 172.21.15.148:0/4112426581 learned_addr learned my addr 172.21.15.148:0/4112426581 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632910152 0 0) 0x7f5e8c0a0e00 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): my global_id is 144384 2024-09-16T04:26:13.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e74003520 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204662718 0 0) 0x7f5e8c0a4970 con 0x7f5e8c0a2410 2024-09-16T04:26:13.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): my global_id is 154271 2024-09-16T04:26:13.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e8c0a0e00 con 0x7f5e8c0a2410 2024-09-16T04:26:13.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3879923369 0 0) 0x7f5e8c0a10e0 con 0x7f5e8c13d060 2024-09-16T04:26:13.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 10 monclient(hunting): my global_id is 164149 2024-09-16T04:26:13.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e8c0a4970 con 0x7f5e8c13d060 2024-09-16T04:26:13.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2492643861 0 0) 0x7f5e74003520 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.183+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 >> v1:172.21.15.148:6790/0 conn(0x7f5e8c0a2410 legacy=0x7f5e8c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 >> v1:172.21.15.171:6789/0 conn(0x7f5e8c13d060 legacy=0x7f5e8c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:26:13.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c06dfd0 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e80001540 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 5 monclient: authenticate success, global_id 144384 2024-09-16T04:26:13.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.210 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:13.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: dump: 2024-09-16T04:26:13.211 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.211 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.211 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.211 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e800017e0 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 1 -- 172.21.15.148:0/4112426581 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e80001bf0 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:13.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 4 set_mon_vals no callback set 2024-09-16T04:26:13.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e8b7fe640 10 monclient: dump: 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.214 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.215 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.215 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.215 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.215 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.215 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 10 monclient: shutdown 2024-09-16T04:26:13.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 1 -- 172.21.15.148:0/4112426581 >> v1:172.21.15.148:6789/0 conn(0x7f5e8c0a1b10 legacy=0x7f5e8c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 1 -- 172.21.15.148:0/4112426581 shutdown_connections 2024-09-16T04:26:13.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 1 -- 172.21.15.148:0/4112426581 >> 172.21.15.148:0/4112426581 conn(0x7f5e8c0a97b0 msgr2=0x7f5e8c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 1 -- 172.21.15.148:0/4112426581 shutdown_connections 2024-09-16T04:26:13.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.184+0000 7f5e92cba640 1 -- 172.21.15.148:0/4112426581 wait complete. 2024-09-16T04:26:13.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.185+0000 7f5e92cba640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.185+0000 7f5e92cba640 10 monclient: monmap: 2024-09-16T04:26:13.217 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.217 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:13.217 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:13.218 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:13.218 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.218 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.218 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.218 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:13.218 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:13.219 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.185+0000 7f5e92cba640 1 Processor -- start 2024-09-16T04:26:13.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.185+0000 7f5e92cba640 1 -- start start 2024-09-16T04:26:13.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.185+0000 7f5e92cba640 10 monclient: init 2024-09-16T04:26:13.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:13.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c138030 con 0x7f5e8c0a2410 2024-09-16T04:26:13.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c137c10 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c0a0130 con 0x7f5e8c13d060 2024-09-16T04:26:13.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e92cba640 10 monclient(hunting): authenticate will time out at 1993.811250s 2024-09-16T04:26:13.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e8bfff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f5e8c0a2410 0x7f5e8c09d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37228/0 (socket says 172.21.15.148:37228) 2024-09-16T04:26:13.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e8bfff640 1 -- 172.21.15.148:0/1444526446 learned_addr learned my addr 172.21.15.148:0/1444526446 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e91257640 1 --1- 172.21.15.148:0/1444526446 >> v1:172.21.15.148:6790/0 conn(0x7f5e8c13d060 0x7f5e8c1d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56608/0 (socket says 172.21.15.148:56608) 2024-09-16T04:26:13.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645001876 0 0) 0x7f5e8c0a0130 con 0x7f5e8c13d060 2024-09-16T04:26:13.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): my global_id is 154274 2024-09-16T04:26:13.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e78003520 con 0x7f5e8c13d060 2024-09-16T04:26:13.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 581550710 0 0) 0x7f5e8c06dfd0 con 0x7f5e8c0a2410 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): my global_id is 144387 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e8c0a0130 con 0x7f5e8c0a2410 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741826421 0 0) 0x7f5e8c137c10 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): my global_id is 164152 2024-09-16T04:26:13.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e8c06dfd0 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1997491378 0 0) 0x7f5e78003520 con 0x7f5e8c13d060 2024-09-16T04:26:13.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e8c137c10 con 0x7f5e8c13d060 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e84001c30 con 0x7f5e8c13d060 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): dump: 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.225 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2728309565 0 0) 0x7f5e8c0a0130 con 0x7f5e8c0a2410 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e78003520 con 0x7f5e8c0a2410 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e80001440 con 0x7f5e8c0a2410 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): dump: 2024-09-16T04:26:13.227 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.228 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3492756002 0 0) 0x7f5e8c06dfd0 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e8c0a0130 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e7c001a10 con 0x7f5e8c0a1b10 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 10 monclient(hunting): dump: 2024-09-16T04:26:13.229 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.230 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.230 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.230 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.230 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.230 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.230 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1970627998 0 0) 0x7f5e8c137c10 con 0x7f5e8c13d060 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.186+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 >> v1:172.21.15.171:6789/0 conn(0x7f5e8c0a1b10 legacy=0x7f5e8c09ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 >> v1:172.21.15.148:6789/0 conn(0x7f5e8c0a2410 legacy=0x7f5e8c09d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 1 monclient: found mon.c 2024-09-16T04:26:13.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c137930 con 0x7f5e8c13d060 2024-09-16T04:26:13.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 5 monclient: authenticate success, global_id 154274 2024-09-16T04:26:13.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 10 monclient: _renew_subs 2024-09-16T04:26:13.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8c09f000 con 0x7f5e8c13d060 2024-09-16T04:26:13.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 10 monclient: _renew_subs 2024-09-16T04:26:13.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5e8c09db10 con 0x7f5e8c13d060 2024-09-16T04:26:13.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e84003970 con 0x7f5e8c13d060 2024-09-16T04:26:13.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f5e84003cc0 con 0x7f5e8c13d060 2024-09-16T04:26:13.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e8affd640 4 set_mon_vals no callback set 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 10 monclient: dump: 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.235 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.236 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.236 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.236 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.236 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.236 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.236 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.187+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f5e84003ee0 con 0x7f5e8c13d060 2024-09-16T04:26:13.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.188+0000 7f5e92cba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.188+0000 7f5e92cba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.188+0000 7f5e92cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.188+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6c005080 con 0x7f5e8c13d060 2024-09-16T04:26:13.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.188+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f5e84047b70 con 0x7f5e8c13d060 2024-09-16T04:26:13.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.188+0000 7f5e90a56640 10 monclient: get_auth_request con 0x7f5e78039260 auth_method 0 2024-09-16T04:26:13.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.192+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f5e84015d70 con 0x7f5e8c13d060 2024-09-16T04:26:13.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.192+0000 7f5e897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.192+0000 7f5e897fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:13.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.362+0000 7f51972c0640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T04:26:13.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.362+0000 7f51972c0640 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T04:26:13.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.362+0000 7f51972c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.364 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.362+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) -- 0x7f5190073940 con 0x7f519009b790 2024-09-16T04:26:13.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.365+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v30) ==== 79+0+0 (unknown 6236318 0 0) 0x7f51800154b0 con 0x7f519009b790 2024-09-16T04:26:13.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.366+0000 7f5185ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T04:26:13.367 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.366+0000 7f5185ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:13.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.424+0000 7f5e92cba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:26:13.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.424+0000 7f5e92cba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:13.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.424+0000 7f5e92cba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.424+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e6c005370 con 0x7f5e8c13d060 2024-09-16T04:26:13.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.425+0000 7f5e897fa640 1 -- 172.21.15.148:0/1444526446 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2307338552) 0x7f5e84019a20 con 0x7f5e8c13d060 2024-09-16T04:26:13.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.425+0000 7f5e897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:13.427 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.425+0000 7f5e897fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:13.428 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:26:13.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 >> v1:172.21.15.171:6812/7327273 conn(0x7f5e78039260 legacy=0x7f5e7803b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 10 monclient: shutdown 2024-09-16T04:26:13.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 >> v1:172.21.15.148:6790/0 conn(0x7f5e8c13d060 legacy=0x7f5e8c1d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 shutdown_connections 2024-09-16T04:26:13.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 >> 172.21.15.148:0/1444526446 conn(0x7f5e8c0a97b0 msgr2=0x7f5e8c0a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 shutdown_connections 2024-09-16T04:26:13.430 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.427+0000 7f5e92cba640 1 -- 172.21.15.148:0/1444526446 wait complete. 2024-09-16T04:26:13.439 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:26:13.439 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:26:13.440 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:26:13.440 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:26:13.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.528+0000 7f51972c0640 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T04:26:13.529 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.528+0000 7f51972c0640 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T04:26:13.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.528+0000 7f51972c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:13.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.528+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) -- 0x7f51900008d0 con 0x7f519009b790 2024-09-16T04:26:13.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.529+0000 7f5185ffb640 1 -- 172.21.15.171:0/817253277 <== mon.1 v1:172.21.15.171:6789/0 11 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v30) ==== 79+0+0 (unknown 6236318 0 0) 0x7f5180019800 con 0x7f519009b790 2024-09-16T04:26:13.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.529+0000 7f5185ffb640 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-16T04:26:13.530 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.529+0000 7f5185ffb640 10 monclient: _finish_command 3 = system:0 2024-09-16T04:26:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 >> v1:172.21.15.171:6812/7327273 conn(0x7f51740391f0 legacy=0x7f517403b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 10 monclient: shutdown 2024-09-16T04:26:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.532 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 >> v1:172.21.15.171:6789/0 conn(0x7f519009b790 legacy=0x7f519012aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 shutdown_connections 2024-09-16T04:26:13.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 >> 172.21.15.171:0/817253277 conn(0x7f51900a5570 msgr2=0x7f51900a1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 shutdown_connections 2024-09-16T04:26:13.533 INFO:tasks.workunit.client.0.smithi171.stderr:2024-09-16T04:26:13.531+0000 7f51972c0640 1 -- 172.21.15.171:0/817253277 wait complete. 2024-09-16T04:26:13.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.537+0000 7f8e170ba640 10 monclient: get_monmap_and_config 2024-09-16T04:26:13.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.537+0000 7f8e170ba640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.538 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.537+0000 7f8e170ba640 10 monclient: monmap: 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:13.538215+0000 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:13.538215+0000 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:13.539 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.546 INFO:tasks.workunit.client.0.smithi171.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-16T04:26:13.546 INFO:tasks.workunit.client.0.smithi171.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-16T04:26:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-16T04:26:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:+ expected_ret=1 2024-09-16T04:26:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T04:26:13.547 INFO:tasks.workunit.client.0.smithi171.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 1 Processor -- start 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 1 -- start start 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient: init 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient(hunting): picked mon.noname-b con 0x7f8e100a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.550 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1013d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e100a0e00 con 0x7f8e1013d060 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e100a10e0 con 0x7f8e100a1b10 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e100a4970 con 0x7f8e100a2410 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e170ba640 10 monclient(hunting): authenticate will time out at 1994.163557s 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e15657640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8e1013d060 0x7f8e1013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37236/0 (socket says 172.21.15.148:37236) 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e15657640 1 -- 172.21.15.148:0/716794095 learned_addr learned my addr 172.21.15.148:0/716794095 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e14e56640 1 --1- 172.21.15.148:0/716794095 >> v1:172.21.15.148:6790/0 conn(0x7f8e100a2410 0x7f8e100a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56616/0 (socket says 172.21.15.148:56616) 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278893400 0 0) 0x7f8e100a4970 con 0x7f8e100a2410 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): my global_id is 154277 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dfc003520 con 0x7f8e100a2410 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160287092 0 0) 0x7f8e100a0e00 con 0x7f8e1013d060 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 10 monclient(hunting): my global_id is 144390 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.538+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e100a4970 con 0x7f8e1013d060 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1576115374 0 0) 0x7f8e100a10e0 con 0x7f8e100a1b10 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient(hunting): my global_id is 164155 2024-09-16T04:26:13.552 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e100a0e00 con 0x7f8e100a1b10 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1741716001 0 0) 0x7f8e100a4970 con 0x7f8e1013d060 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 >> v1:172.21.15.148:6790/0 conn(0x7f8e100a2410 legacy=0x7f8e100a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 >> v1:172.21.15.171:6789/0 conn(0x7f8e100a1b10 legacy=0x7f8e1013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1006dfd0 con 0x7f8e1013d060 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.553 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 5 monclient: authenticate success, global_id 144390 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8df8001570 con 0x7f8e1013d060 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: dump: 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.554 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8df80017d0 con 0x7f8e1013d060 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 1 -- 172.21.15.148:0/716794095 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8df8001c20 con 0x7f8e1013d060 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e0f7fe640 10 monclient: dump: 2024-09-16T04:26:13.555 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.556 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.557 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.557 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.557 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.557 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.557 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 4 set_mon_vals no callback set 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 10 monclient: shutdown 2024-09-16T04:26:13.558 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 1 -- 172.21.15.148:0/716794095 >> v1:172.21.15.148:6789/0 conn(0x7f8e1013d060 legacy=0x7f8e1013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 1 -- 172.21.15.148:0/716794095 shutdown_connections 2024-09-16T04:26:13.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 1 -- 172.21.15.148:0/716794095 >> 172.21.15.148:0/716794095 conn(0x7f8e100a97b0 msgr2=0x7f8e100a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.559 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 1 -- 172.21.15.148:0/716794095 shutdown_connections 2024-09-16T04:26:13.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.539+0000 7f8e170ba640 1 -- 172.21.15.148:0/716794095 wait complete. 2024-09-16T04:26:13.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.540+0000 7f8e170ba640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.560 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.540+0000 7f8e170ba640 10 monclient: monmap: 2024-09-16T04:26:13.560 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.560 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:13.560 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.540+0000 7f8e170ba640 1 Processor -- start 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.540+0000 7f8e170ba640 1 -- start start 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.540+0000 7f8e170ba640 10 monclient: init 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:26:13.561 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient(hunting): picked mon.noname-a con 0x7f8e100a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient(hunting): picked mon.noname-c con 0x7f8e100a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1013d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e10137fc0 con 0x7f8e100a1b10 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e10137ba0 con 0x7f8e1013d060 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1009d9a0 con 0x7f8e100a2410 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e170ba640 10 monclient(hunting): authenticate will time out at 1994.166282s 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e14e56640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8e100a1b10 0x7f8e1009cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37246/0 (socket says 172.21.15.148:37246) 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e14e56640 1 -- 172.21.15.148:0/739522371 learned_addr learned my addr 172.21.15.148:0/739522371 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3911273374 0 0) 0x7f8e080011f0 con 0x7f8e100a1b10 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.562 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): my global_id is 144393 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df0003520 con 0x7f8e100a1b10 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725743606 0 0) 0x7f8e1009d9a0 con 0x7f8e100a2410 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): my global_id is 154280 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e080011f0 con 0x7f8e100a2410 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260221155 0 0) 0x7f8e10137ba0 con 0x7f8e1013d060 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.563 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): my global_id is 164158 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1009d9a0 con 0x7f8e1013d060 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3085894478 0 0) 0x7f8df0003520 con 0x7f8e100a1b10 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e10137ba0 con 0x7f8e100a1b10 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e080014c0 con 0x7f8e100a1b10 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): dump: 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.564 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 875603277 0 0) 0x7f8e080011f0 con 0x7f8e100a2410 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df0003520 con 0x7f8e100a2410 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e00001c30 con 0x7f8e100a2410 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): dump: 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.565 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898961125 0 0) 0x7f8e1009d9a0 con 0x7f8e1013d060 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e080011f0 con 0x7f8e1013d060 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8df8001b20 con 0x7f8e1013d060 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.566 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 10 monclient(hunting): dump: 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.541+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1529595945 0 0) 0x7f8e10137ba0 con 0x7f8e100a1b10 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.148:6790/0 conn(0x7f8e100a2410 legacy=0x7f8e1009d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.171:6789/0 conn(0x7f8e1013d060 legacy=0x7f8e101d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.567 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 monclient: found mon.a 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e100a0130 con 0x7f8e100a1b10 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 5 monclient: authenticate success, global_id 144393 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 10 monclient: _renew_subs 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1013cb20 con 0x7f8e100a1b10 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 10 monclient: _renew_subs 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.568 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e1009f0c0 con 0x7f8e100a1b10 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e08004a90 con 0x7f8e100a1b10 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e08004de0 con 0x7f8e100a1b10 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 10 monclient: dump: 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.569 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0effd640 4 set_mon_vals no callback set 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8e08004de0 con 0x7f8e100a1b10 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.542+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8e100008d0 con 0x7f8e1013db00 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f8e08047b50 con 0x7f8e100a1b10 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070480592 0 0) 0x7f8e100008d0 con 0x7f8e1013db00 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.570 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient(hunting): my global_id is 144396 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e10137ba0 con 0x7f8e1013db00 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e15657640 10 monclient: get_auth_request con 0x7f8df0039200 auth_method 0 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4021924669 0 0) 0x7f8e10137ba0 con 0x7f8e1013db00 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e100008d0 con 0x7f8e1013db00 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e00003b30 con 0x7f8e1013db00 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient: dump: 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.571 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1090754241 0 0) 0x7f8e100008d0 con 0x7f8e1013db00 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e1009f560 con 0x7f8e1013db00 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8e00005050 con 0x7f8e1013db00 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.148:6789/0 conn(0x7f8e1013db00 legacy=0x7f8e1013dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.572 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.543+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.148:6789/0 conn(0x7f8e1013db00 legacy=0x7f8e1013dee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.549+0000 7f8e170ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.549+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8e1013e9d0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569323777 0 0) 0x7f8e1013e9d0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient(hunting): my global_id is 144399 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e100008d0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.574 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1114708941 0 0) 0x7f8e100008d0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e1013e9d0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8e00005420 con 0x7f8e1013e5f0 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient: dump: 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.575 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 402147378 0 0) 0x7f8e1013e9d0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e1013c5c0 con 0x7f8e1013e5f0 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4313 (unknown 0 0 2056873036) 0x7f8e00004a60 con 0x7f8e1013e5f0 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.148:6789/0 conn(0x7f8e1013e5f0 legacy=0x7f8e101d9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.576 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e0d7fa640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.148:6789/0 conn(0x7f8e1013e5f0 legacy=0x7f8e101d9280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 76, 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 218250, 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.577 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.578 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.579 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:13.580 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.581 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:13.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.584 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:13.585 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.587 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.588 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:13.589 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.171:6812/7327273 conn(0x7f8df0039200 legacy=0x7f8df003b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e170ba640 10 monclient: shutdown 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e170ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 >> v1:172.21.15.148:6789/0 conn(0x7f8e100a1b10 legacy=0x7f8e1009cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e15657640 1 -- 172.21.15.148:0/739522371 reap_dead start 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.550+0000 7f8e15657640 1 -- 172.21.15.148:0/739522371 reap_dead start 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.551+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 shutdown_connections 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.551+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 >> 172.21.15.148:0/739522371 conn(0x7f8e100a97b0 msgr2=0x7f8e100a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.551+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 shutdown_connections 2024-09-16T04:26:13.591 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.551+0000 7f8e170ba640 1 -- 172.21.15.148:0/739522371 wait complete. 2024-09-16T04:26:13.592 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:26:13.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:26:13.592 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:26:13.690 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.688+0000 7f26603e4640 10 monclient: get_monmap_and_config 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.688+0000 7f26603e4640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.688+0000 7f26603e4640 10 monclient: monmap: 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:13.689785+0000 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:13.689785+0000 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.691 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:13.692 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:13.692 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.689+0000 7f26603e4640 1 Processor -- start 2024-09-16T04:26:13.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.689+0000 7f26603e4640 1 -- start start 2024-09-16T04:26:13.692 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.689+0000 7f26603e4640 10 monclient: init 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient(hunting): picked mon.noname-c con 0x7f265809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient(hunting): picked mon.noname-a con 0x7f265809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient(hunting): picked mon.noname-b con 0x7f265809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26580a1010 con 0x7f265809c660 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265809f030 con 0x7f265809bb70 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265809dc60 con 0x7f265809cc30 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.701 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient(hunting): authenticate will time out at 1994.315218s 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d97f640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f265809c660 0x7f2658138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37272/0 (socket says 172.21.15.148:37272) 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d97f640 1 -- 172.21.15.148:0/1409265421 learned_addr learned my addr 172.21.15.148:0/1409265421 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265e981640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f265809cc30 0x7f265813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56634/0 (socket says 172.21.15.148:56634) 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1891928282 0 0) 0x7f265809dc60 con 0x7f265809cc30 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): my global_id is 154289 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2644003520 con 0x7f265809cc30 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230883268 0 0) 0x7f26580a1010 con 0x7f265809c660 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): my global_id is 144408 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f265809dc60 con 0x7f265809c660 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474961001 0 0) 0x7f265809f030 con 0x7f265809bb70 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient(hunting): my global_id is 164167 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26580a1010 con 0x7f265809bb70 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2391429175 0 0) 0x7f2644003520 con 0x7f265809cc30 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 >> v1:172.21.15.171:6789/0 conn(0x7f265809bb70 legacy=0x7f265809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 >> v1:172.21.15.148:6789/0 conn(0x7f265809c660 legacy=0x7f2658138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 monclient: found mon.noname-b 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265809df40 con 0x7f265809cc30 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2650001540 con 0x7f265809cc30 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 5 monclient: authenticate success, global_id 154289 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:13.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: dump: 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26500017f0 con 0x7f265809cc30 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 1 -- 172.21.15.148:0/1409265421 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2650001bc0 con 0x7f265809cc30 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f26603e4640 4 set_mon_vals no callback set 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.690+0000 7f265d17e640 10 monclient: dump: 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.706 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 10 monclient: shutdown 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 1 -- 172.21.15.148:0/1409265421 >> v1:172.21.15.148:6790/0 conn(0x7f265809cc30 legacy=0x7f265813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 1 -- 172.21.15.148:0/1409265421 shutdown_connections 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 1 -- 172.21.15.148:0/1409265421 >> 172.21.15.148:0/1409265421 conn(0x7f26580a5640 msgr2=0x7f26580a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 1 -- 172.21.15.148:0/1409265421 shutdown_connections 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.691+0000 7f26603e4640 1 -- 172.21.15.148:0/1409265421 wait complete. 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient: monmap: 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:13.708 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 1 Processor -- start 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 1 -- start start 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient: init 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient(hunting): picked mon.noname-b con 0x7f265809bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient(hunting): picked mon.noname-c con 0x7f265809c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient(hunting): picked mon.noname-a con 0x7f265809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2658126680 con 0x7f265809cc30 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2658126c40 con 0x7f265809bb70 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2658125230 con 0x7f265809c660 2024-09-16T04:26:13.709 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f26603e4640 10 monclient(hunting): authenticate will time out at 1994.317911s 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f265d97f640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f265809c660 0x7f26581d0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56644/0 (socket says 172.21.15.148:56644) 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f265e981640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f265809cc30 0x7f26581d42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37284/0 (socket says 172.21.15.148:37284) 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.692+0000 7f265d97f640 1 -- 172.21.15.148:0/4207762737 learned_addr learned my addr 172.21.15.148:0/4207762737 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421358040 0 0) 0x7f2658125230 con 0x7f265809c660 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): my global_id is 154292 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f263c003520 con 0x7f265809c660 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1624962518 0 0) 0x7f2658126680 con 0x7f265809cc30 2024-09-16T04:26:13.710 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): my global_id is 144411 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2658125230 con 0x7f265809cc30 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984174090 0 0) 0x7f2658126c40 con 0x7f265809bb70 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): my global_id is 164170 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2658126680 con 0x7f265809bb70 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 399135300 0 0) 0x7f2658125230 con 0x7f265809cc30 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2658126c40 con 0x7f265809cc30 2024-09-16T04:26:13.711 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2650001500 con 0x7f265809cc30 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): dump: 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.712 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1078748874 0 0) 0x7f2658126680 con 0x7f265809bb70 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2658125230 con 0x7f265809bb70 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1043972412 0 0) 0x7f263c003520 con 0x7f265809c660 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2658126680 con 0x7f265809c660 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2654001bb0 con 0x7f265809bb70 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): dump: 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.713 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f26480017f0 con 0x7f265809c660 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient(hunting): dump: 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 521643407 0 0) 0x7f2658126c40 con 0x7f265809cc30 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.148:6790/0 conn(0x7f265809c660 legacy=0x7f26581d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.171:6789/0 conn(0x7f265809bb70 legacy=0x7f26581cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 monclient: found mon.a 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2658123cc0 con 0x7f265809cc30 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 5 monclient: authenticate success, global_id 144411 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 10 monclient: _renew_subs 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2658123fb0 con 0x7f265809cc30 2024-09-16T04:26:13.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 10 monclient: _renew_subs 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f26581cc2d0 con 0x7f265809cc30 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26500017f0 con 0x7f265809cc30 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2650004c30 con 0x7f265809cc30 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f265c97d640 4 set_mon_vals no callback set 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.693+0000 7f264effd640 10 monclient: dump: 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.717 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f26603e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f26400065c0 con 0x7f2640002860 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f265000fec0 con 0x7f265809cc30 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f26500476f0 con 0x7f265809cc30 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187944263 0 0) 0x7f26400065c0 con 0x7f2640002860 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 10 monclient(hunting): my global_id is 154295 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2658126c40 con 0x7f2640002860 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.694+0000 7f265d97f640 10 monclient: get_auth_request con 0x7f263c039180 auth_method 0 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2818898654 0 0) 0x7f2658126c40 con 0x7f2640002860 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f26400065c0 con 0x7f2640002860 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2654003d40 con 0x7f2640002860 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 10 monclient: dump: 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.719 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2980197214 0 0) 0x7f26400065c0 con 0x7f2640002860 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f264000a150 con 0x7f2640002860 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2654006090 con 0x7f2640002860 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.148:6790/0 conn(0x7f2640002860 legacy=0x7f2640004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.695+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.148:6790/0 conn(0x7f2640002860 legacy=0x7f2640004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.701+0000 7f26603e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:13.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.701+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2640009e40 con 0x7f2640009100 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983394516 0 0) 0x7f2640009e40 con 0x7f2640009100 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient(hunting): my global_id is 154298 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26400065c0 con 0x7f2640009100 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 181318614 0 0) 0x7f26400065c0 con 0x7f2640009100 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2640009e40 con 0x7f2640009100 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2654004380 con 0x7f2640009100 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient: dump: 2024-09-16T04:26:13.723 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.724 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3419519597 0 0) 0x7f2640009e40 con 0x7f2640009100 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f264000c3d0 con 0x7f2640009100 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 3036670851) 0x7f2654005720 con 0x7f2640009100 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.148:6790/0 conn(0x7f2640009100 legacy=0x7f2640009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.725 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.702+0000 7f264effd640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.148:6790/0 conn(0x7f2640009100 legacy=0x7f2640009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 76, 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 110610, 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:13.726 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.727 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.728 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:13.729 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.730 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:13.731 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:13.732 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:13.733 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:13.734 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.735 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.736 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 4 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:13.737 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.171:6812/7327273 conn(0x7f263c039180 legacy=0x7f263c03b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 10 monclient: shutdown 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 >> v1:172.21.15.148:6789/0 conn(0x7f265809cc30 legacy=0x7f26581d42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f265e981640 1 -- 172.21.15.148:0/4207762737 reap_dead start 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f265e981640 1 -- 172.21.15.148:0/4207762737 reap_dead start 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 shutdown_connections 2024-09-16T04:26:13.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 >> 172.21.15.148:0/4207762737 conn(0x7f26580a5640 msgr2=0x7f26581338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 shutdown_connections 2024-09-16T04:26:13.740 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.703+0000 7f26603e4640 1 -- 172.21.15.148:0/4207762737 wait complete. 2024-09-16T04:26:13.740 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:26:13.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:26:13.740 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:26:13.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.835+0000 7fd965ae4640 10 monclient: get_monmap_and_config 2024-09-16T04:26:13.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.835+0000 7fd965ae4640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.835+0000 7fd965ae4640 10 monclient: monmap: 2024-09-16T04:26:13.837 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.837 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.837 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:13.836715+0000 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:13.836715+0000 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:13.838 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.836+0000 7fd965ae4640 1 Processor -- start 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.836+0000 7fd965ae4640 1 -- start start 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.836+0000 7fd965ae4640 10 monclient: init 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient(hunting): picked mon.noname-b con 0x7fd96009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient(hunting): picked mon.noname-c con 0x7fd96009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient(hunting): picked mon.noname-a con 0x7fd96009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9600a1010 con 0x7fd96009cc30 2024-09-16T04:26:13.847 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96009f030 con 0x7fd96009c660 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96009dc60 con 0x7fd96009bb70 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient(hunting): authenticate will time out at 1994.462126s 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95ffff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fd96009cc30 0x7fd96013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:37296/0 (socket says 172.21.15.148:37296) 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95f7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd96009bb70 0x7fd96009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56686/0 (socket says 172.21.15.148:56686) 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95ffff640 1 -- 172.21.15.148:0/2992588272 learned_addr learned my addr 172.21.15.148:0/2992588272 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402979521 0 0) 0x7fd96009dc60 con 0x7fd96009bb70 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): my global_id is 154301 2024-09-16T04:26:13.848 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd948003520 con 0x7fd96009bb70 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343294670 0 0) 0x7fd9600a1010 con 0x7fd96009cc30 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): my global_id is 144414 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96009dc60 con 0x7fd96009cc30 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017888557 0 0) 0x7fd96009f030 con 0x7fd96009c660 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient(hunting): my global_id is 164173 2024-09-16T04:26:13.849 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9600a1010 con 0x7fd96009c660 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1646536510 0 0) 0x7fd948003520 con 0x7fd96009bb70 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 >> v1:172.21.15.171:6789/0 conn(0x7fd96009c660 legacy=0x7fd960138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 >> v1:172.21.15.148:6789/0 conn(0x7fd96009cc30 legacy=0x7fd96013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 monclient: found mon.noname-b 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96009df40 con 0x7fd96009bb70 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd950001990 con 0x7fd96009bb70 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 5 monclient: authenticate success, global_id 154301 2024-09-16T04:26:13.850 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: dump: 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.851 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd950001bf0 con 0x7fd96009bb70 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 1 -- 172.21.15.148:0/2992588272 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd950001f80 con 0x7fd96009bb70 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd965ae4640 4 set_mon_vals no callback set 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.837+0000 7fd95e7fc640 10 monclient: dump: 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.852 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 10 monclient: shutdown 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 1 -- 172.21.15.148:0/2992588272 >> v1:172.21.15.148:6790/0 conn(0x7fd96009bb70 legacy=0x7fd96009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 1 -- 172.21.15.148:0/2992588272 shutdown_connections 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 1 -- 172.21.15.148:0/2992588272 >> 172.21.15.148:0/2992588272 conn(0x7fd9600a5640 msgr2=0x7fd9600a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.853 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 1 -- 172.21.15.148:0/2992588272 shutdown_connections 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 1 -- 172.21.15.148:0/2992588272 wait complete. 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 10 monclient: build_initial_monmap 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.838+0000 7fd965ae4640 10 monclient: monmap: 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:13.854 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 1 Processor -- start 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 1 -- start start 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient: init 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient(hunting): picked mon.noname-c con 0x7fd96009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient(hunting): picked mon.noname-a con 0x7fd96009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient(hunting): picked mon.noname-b con 0x7fd96009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9601ca560 con 0x7fd96009c660 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96013c5a0 con 0x7fd96009cc30 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96013cae0 con 0x7fd96009bb70 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:13.855 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd965ae4640 10 monclient(hunting): authenticate will time out at 1994.464898s 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd95f7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fd96009bb70 0x7fd9601c9740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56702/0 (socket says 172.21.15.148:56702) 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.839+0000 7fd95f7fe640 1 -- 172.21.15.148:0/587908131 learned_addr learned my addr 172.21.15.148:0/587908131 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631798015 0 0) 0x7fd9601ca560 con 0x7fd96009c660 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): my global_id is 144417 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd94c003550 con 0x7fd96009c660 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998678735 0 0) 0x7fd96013cae0 con 0x7fd96009bb70 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.856 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): my global_id is 154304 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9601ca560 con 0x7fd96009bb70 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942468016 0 0) 0x7fd96013c5a0 con 0x7fd96009cc30 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): my global_id is 164176 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96013cae0 con 0x7fd96009cc30 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3877248246 0 0) 0x7fd94c003550 con 0x7fd96009c660 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd96013c5a0 con 0x7fd96009c660 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd9540018b0 con 0x7fd96009c660 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.857 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): dump: 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3045927032 0 0) 0x7fd9601ca560 con 0x7fd96009bb70 2024-09-16T04:26:13.858 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd94c003550 con 0x7fd96009bb70 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1050043880 0 0) 0x7fd96013cae0 con 0x7fd96009cc30 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9601ca560 con 0x7fd96009cc30 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd950001890 con 0x7fd96009bb70 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): dump: 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.859 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd958001870 con 0x7fd96009cc30 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient(hunting): dump: 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.860 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3349316948 0 0) 0x7fd96013c5a0 con 0x7fd96009c660 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.148:6790/0 conn(0x7fd96009bb70 legacy=0x7fd9601c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.171:6789/0 conn(0x7fd96009cc30 legacy=0x7fd9601d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: _finish_hunting 0 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 monclient: found mon.a 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9600a3f30 con 0x7fd96009c660 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: _finish_auth 0 2024-09-16T04:26:13.861 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: _check_auth_tickets 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 5 monclient: authenticate success, global_id 144417 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 10 monclient: _renew_subs 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd96011cb60 con 0x7fd96009c660 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 10 monclient: _renew_subs 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd96013b4c0 con 0x7fd96009c660 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd954003610 con 0x7fd96009c660 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd954003960 con 0x7fd96009c660 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.862 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd95dffb640 4 set_mon_vals no callback set 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.840+0000 7fd947fff640 10 monclient: dump: 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.841+0000 7fd965ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.863 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.841+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd9400065c0 con 0x7fd940002860 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.841+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fd95400ebf0 con 0x7fd96009c660 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.841+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fd954046620 con 0x7fd96009c660 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.841+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221032005 0 0) 0x7fd9400065c0 con 0x7fd940002860 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient(hunting): my global_id is 164179 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96013c5a0 con 0x7fd940002860 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd95ffff640 10 monclient: get_auth_request con 0x7fd94c039150 auth_method 0 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2481677107 0 0) 0x7fd96013c5a0 con 0x7fd940002860 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9400065c0 con 0x7fd940002860 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd950004eb0 con 0x7fd940002860 2024-09-16T04:26:13.864 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient: dump: 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.865 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2610053720 0 0) 0x7fd9400065c0 con 0x7fd940002860 2024-09-16T04:26:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd94000a150 con 0x7fd940002860 2024-09-16T04:26:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd950007090 con 0x7fd940002860 2024-09-16T04:26:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.171:6789/0 conn(0x7fd940002860 legacy=0x7fd940004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.866 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.842+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.171:6789/0 conn(0x7fd940002860 legacy=0x7fd940004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:13.867 INFO:tasks.workunit.client.0.smithi171.stderr:+ ret=1 2024-09-16T04:26:13.867 INFO:tasks.workunit.client.0.smithi171.stderr:+ [[ 1 -ne 1 ]] 2024-09-16T04:26:13.867 INFO:tasks.workunit.client.0.smithi171.stderr:+ return 0 2024-09-16T04:26:13.867 INFO:tasks.workunit.client.0.smithi171.stdout:OK 2024-09-16T04:26:13.869 INFO:tasks.workunit.client.0.smithi171.stderr:+ echo OK 2024-09-16T04:26:13.869 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T04:26:13.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.848+0000 7fd965ae4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.848+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd940009e40 con 0x7fd940009100 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88757382 0 0) 0x7fd940009e40 con 0x7fd940009100 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient(hunting): my global_id is 164182 2024-09-16T04:26:13.870 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9400065c0 con 0x7fd940009100 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2322003220 0 0) 0x7fd9400065c0 con 0x7fd940009100 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd940009e40 con 0x7fd940009100 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fd950005480 con 0x7fd940009100 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient: dump: 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:13.871 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:13.872 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:13.872 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:13.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3610427238 0 0) 0x7fd940009e40 con 0x7fd940009100 2024-09-16T04:26:13.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd94000c3d0 con 0x7fd940009100 2024-09-16T04:26:13.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3975 (unknown 0 0 3917165730) 0x7fd950006720 con 0x7fd940009100 2024-09-16T04:26:13.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.171:6789/0 conn(0x7fd940009100 legacy=0x7fd940009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd947fff640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.171:6789/0 conn(0x7fd940009100 legacy=0x7fd940009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:26:13.873 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 76, 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 2856, 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 2, 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.874 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.875 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.876 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:13.877 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:13.878 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.879 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.880 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.881 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:13.882 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:13.883 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:13.885 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:13.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.171:6812/7327273 conn(0x7fd94c039150 legacy=0x7fd94c03b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd965ae4640 10 monclient: shutdown 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd965ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 >> v1:172.21.15.148:6789/0 conn(0x7fd96009c660 legacy=0x7fd9601c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.849+0000 7fd95ffff640 1 -- 172.21.15.148:0/587908131 reap_dead start 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.850+0000 7fd95ffff640 1 -- 172.21.15.148:0/587908131 reap_dead start 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.850+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 shutdown_connections 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.850+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 >> 172.21.15.148:0/587908131 conn(0x7fd9600a5640 msgr2=0x7fd960134940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.850+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 shutdown_connections 2024-09-16T04:26:13.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:13.850+0000 7fd965ae4640 1 -- 172.21.15.148:0/587908131 wait complete. 2024-09-16T04:26:13.887 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:26:13.888 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:26:13.888 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:26:13.888 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:26:13.888 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:26:13.888 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:26:13.888 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:26:13.900 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:26:13.900 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:26:13.900 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:26:13.934 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-16T04:26:13.934 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T04:26:14.036 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.035+0000 7fca05f31640 10 monclient: get_monmap_and_config 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.035+0000 7fca05f31640 10 monclient: build_initial_monmap 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.035+0000 7fca05f31640 10 monclient: monmap: 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:14.036084+0000 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:14.036084+0000 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:14.037 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 1 Processor -- start 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 1 -- start start 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient: init 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient(hunting): picked mon.noname-c con 0x7fca000a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient(hunting): picked mon.noname-a con 0x7fca000a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient(hunting): picked mon.noname-b con 0x7fca0013d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:14.045 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000a0e00 con 0x7fca000a1b10 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000a10e0 con 0x7fca000a2410 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000a4970 con 0x7fca0013d060 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fca05f31640 10 monclient(hunting): authenticate will time out at 1994.661658s 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fc9f7fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fca000a1b10 legacy=0x7fca0013b940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fc9fffff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fca0013d060 0x7fca0013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56714/0 (socket says 172.21.15.148:56714) 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fc9fffff640 1 -- 172.21.15.148:0/197479910 learned_addr learned my addr 172.21.15.148:0/197479910 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.036+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213688045 0 0) 0x7fca000a4970 con 0x7fca0013d060 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:14.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): my global_id is 154307 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9ec003520 con 0x7fca0013d060 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3737822728 0 0) 0x7fca000a10e0 con 0x7fca000a2410 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): _negotiate 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient(hunting): my global_id is 164185 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca000a4970 con 0x7fca000a2410 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3404412772 0 0) 0x7fc9ec003520 con 0x7fca0013d060 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 >> v1:172.21.15.171:6789/0 conn(0x7fca000a2410 legacy=0x7fca000a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 >> v1:172.21.15.148:6789/0 conn(0x7fca000a1b10 legacy=0x7fca0013b940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: _finish_hunting 0 2024-09-16T04:26:14.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 monclient: found mon.noname-b 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0006dfd0 con 0x7fca0013d060 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: _finish_auth 0 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: _check_auth_tickets 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9f8001990 con 0x7fca0013d060 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: dump: 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:14.048 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 5 monclient: authenticate success, global_id 154307 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9f8001cb0 con 0x7fca0013d060 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:14.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 1 -- 172.21.15.148:0/197479910 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9f8002080 con 0x7fca0013d060 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 4 set_mon_vals no callback set 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fc9feffd640 10 monclient: dump: 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 10 monclient: shutdown 2024-09-16T04:26:14.050 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 1 -- 172.21.15.148:0/197479910 >> v1:172.21.15.148:6790/0 conn(0x7fca0013d060 legacy=0x7fca0013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 1 -- 172.21.15.148:0/197479910 shutdown_connections 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 1 -- 172.21.15.148:0/197479910 >> 172.21.15.148:0/197479910 conn(0x7fca000a97b0 msgr2=0x7fca000a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 1 -- 172.21.15.148:0/197479910 shutdown_connections 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.037+0000 7fca05f31640 1 -- 172.21.15.148:0/197479910 wait complete. 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.038+0000 7fca05f31640 10 monclient: build_initial_monmap 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.038+0000 7fca05f31640 10 monclient: monmap: 2024-09-16T04:26:14.051 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.038+0000 7fca05f31640 1 Processor -- start 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.038+0000 7fca05f31640 1 -- start start 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.038+0000 7fca05f31640 10 monclient: init 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:14.052 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient(hunting): picked mon.noname-a con 0x7fca000a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient(hunting): picked mon.noname-c con 0x7fca000a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient(hunting): picked mon.noname-b con 0x7fca0013d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca00137cb0 con 0x7fca000a1b10 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca001cc910 con 0x7fca0013d060 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca000a0130 con 0x7fca000a2410 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient(hunting): authenticate will time out at 1994.664226s 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9ff7fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fca000a1b10 legacy=0x7fca001cbaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9f7fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fca000a2410 0x7fca001cc200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56718/0 (socket says 172.21.15.148:56718) 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9f7fff640 1 -- 172.21.15.148:0/1050042696 learned_addr learned my addr 172.21.15.148:0/1050042696 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680700277 0 0) 0x7fca000a0130 con 0x7fca000a2410 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:14.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): my global_id is 154310 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e0003520 con 0x7fca000a2410 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265343189 0 0) 0x7fca001cc910 con 0x7fca0013d060 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _negotiate 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): my global_id is 164188 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca000a0130 con 0x7fca0013d060 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2910927876 0 0) 0x7fc9e0003520 con 0x7fca000a2410 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fca001cc910 con 0x7fca000a2410 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9e8004150 con 0x7fca000a2410 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:14.054 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): dump: 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2122406626 0 0) 0x7fca000a0130 con 0x7fca0013d060 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9e0003520 con 0x7fca0013d060 2024-09-16T04:26:14.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9f80017b0 con 0x7fca0013d060 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient(hunting): dump: 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 583480848 0 0) 0x7fca001cc910 con 0x7fca000a2410 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 >> v1:172.21.15.171:6789/0 conn(0x7fca0013d060 legacy=0x7fca001d3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 >> v1:172.21.15.148:6789/0 conn(0x7fca000a1b10 legacy=0x7fca001cbaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient: _finish_hunting 0 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 monclient: found mon.c 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0009e7a0 con 0x7fca000a2410 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient: _finish_auth 0 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fc9fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 5 monclient: authenticate success, global_id 154310 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient: _renew_subs 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:14.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fca001379d0 con 0x7fca000a2410 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient: _renew_subs 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.039+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fca0009db10 con 0x7fca000a2410 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc9e8004400 con 0x7fca000a2410 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc9e80047d0 con 0x7fca000a2410 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fe7fc640 4 set_mon_vals no callback set 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 10 monclient: dump: 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:14.058 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc9e8004970 con 0x7fca000a2410 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fca05f31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fca05f31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fca05f31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:14.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.040+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e4005080 con 0x7fca000a2410 2024-09-16T04:26:14.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.041+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fc9e80479e0 con 0x7fca000a2410 2024-09-16T04:26:14.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.041+0000 7fc9ff7fe640 10 monclient: get_auth_request con 0x7fc9e0038f10 auth_method 0 2024-09-16T04:26:14.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.045+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fc9e8015be0 con 0x7fca000a2410 2024-09-16T04:26:14.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.045+0000 7fc9fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:14.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.045+0000 7fc9fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:14.268 DEBUG:teuthology.parallel:result is None 2024-09-16T04:26:14.268 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:26:14.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fca05f31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:26:14.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fca05f31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:14.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fca05f31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:14.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc9e4005370 con 0x7fca000a2410 2024-09-16T04:26:14.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fc9fcff9640 1 -- 172.21.15.148:0/1050042696 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 9163912) 0x7fc9e8034590 con 0x7fca000a2410 2024-09-16T04:26:14.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fc9fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:14.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.276+0000 7fc9fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:14.279 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:26:14.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 >> v1:172.21.15.171:6812/7327273 conn(0x7fc9e0038f10 legacy=0x7fc9e003b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:14.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 10 monclient: shutdown 2024-09-16T04:26:14.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:14.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 >> v1:172.21.15.148:6790/0 conn(0x7fca000a2410 legacy=0x7fca001cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:14.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 shutdown_connections 2024-09-16T04:26:14.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 >> 172.21.15.148:0/1050042696 conn(0x7fca000a97b0 msgr2=0x7fca00135990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:14.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 shutdown_connections 2024-09-16T04:26:14.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:14.278+0000 7fca05f31640 1 -- 172.21.15.148:0/1050042696 wait complete. 2024-09-16T04:26:14.293 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:26:14.293 DEBUG:teuthology.orchestra.run.smithi171:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T04:26:14.366 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:26:14.366 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T04:26:14.376 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T04:26:17.291 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:26:17.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.402+0000 7f8f27fb7640 10 monclient: get_monmap_and_config 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.402+0000 7f8f27fb7640 10 monclient: build_initial_monmap 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.402+0000 7f8f27fb7640 10 monclient: monmap: 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:17.403003+0000 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:17.403003+0000 2024-09-16T04:26:17.404 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:17.405 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:17.405 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:17.405 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:17.405 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:17.405 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:17.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.402+0000 7f8f27fb7640 1 Processor -- start 2024-09-16T04:26:17.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.402+0000 7f8f27fb7640 1 -- start start 2024-09-16T04:26:17.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.402+0000 7f8f27fb7640 10 monclient: init 2024-09-16T04:26:17.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:17.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient(hunting): picked mon.noname-a con 0x7f8f200a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient(hunting): picked mon.noname-b con 0x7f8f200a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient(hunting): picked mon.noname-c con 0x7f8f2013d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f200a0e00 con 0x7f8f200a2410 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f200a10e0 con 0x7f8f2013d060 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f200a4970 con 0x7f8f200a1b10 2024-09-16T04:26:17.418 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:17.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f27fb7640 10 monclient(hunting): authenticate will time out at 1998.028522s 2024-09-16T04:26:17.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f25d53640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8f200a2410 legacy=0x7f8f200a27f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:17.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f25552640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8f200a1b10 0x7f8f2013b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56724/0 (socket says 172.21.15.148:56724) 2024-09-16T04:26:17.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f25552640 1 -- 172.21.15.148:0/3421970240 learned_addr learned my addr 172.21.15.148:0/3421970240 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:17.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263182742 0 0) 0x7f8f200a4970 con 0x7f8f200a1b10 2024-09-16T04:26:17.419 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f24d51640 10 monclient(hunting): _negotiate 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f24d51640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f24d51640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f24d51640 10 monclient(hunting): my global_id is 154313 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.403+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f08003520 con 0x7f8f200a1b10 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618503954 0 0) 0x7f8f200a10e0 con 0x7f8f2013d060 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient(hunting): _negotiate 2024-09-16T04:26:17.420 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient(hunting): my global_id is 164191 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f200a4970 con 0x7f8f2013d060 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 674292575 0 0) 0x7f8f08003520 con 0x7f8f200a1b10 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 >> v1:172.21.15.171:6789/0 conn(0x7f8f2013d060 legacy=0x7f8f2013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 >> v1:172.21.15.148:6789/0 conn(0x7f8f200a2410 legacy=0x7f8f200a27f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:17.421 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: _finish_hunting 0 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 monclient: found mon.noname-b 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2006dfd0 con 0x7f8f200a1b10 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: _finish_auth 0 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: _check_auth_tickets 2024-09-16T04:26:17.422 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f14001610 con 0x7f8f200a1b10 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 5 monclient: authenticate success, global_id 154313 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: dump: 2024-09-16T04:26:17.423 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:17.424 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f14001870 con 0x7f8f200a1b10 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 1 -- 172.21.15.148:0/3421970240 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f14001c40 con 0x7f8f200a1b10 2024-09-16T04:26:17.425 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 4 set_mon_vals no callback set 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f24d51640 10 monclient: dump: 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:17.426 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:17.427 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:17.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 10 monclient: shutdown 2024-09-16T04:26:17.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:17.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 1 -- 172.21.15.148:0/3421970240 >> v1:172.21.15.148:6790/0 conn(0x7f8f200a1b10 legacy=0x7f8f2013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:17.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 1 -- 172.21.15.148:0/3421970240 shutdown_connections 2024-09-16T04:26:17.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 1 -- 172.21.15.148:0/3421970240 >> 172.21.15.148:0/3421970240 conn(0x7f8f200a97b0 msgr2=0x7f8f200a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:17.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 1 -- 172.21.15.148:0/3421970240 shutdown_connections 2024-09-16T04:26:17.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.404+0000 7f8f27fb7640 1 -- 172.21.15.148:0/3421970240 wait complete. 2024-09-16T04:26:17.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.405+0000 7f8f27fb7640 10 monclient: build_initial_monmap 2024-09-16T04:26:17.429 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.405+0000 7f8f27fb7640 10 monclient: monmap: 2024-09-16T04:26:17.429 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:17.429 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:17.430 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.405+0000 7f8f27fb7640 1 Processor -- start 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.405+0000 7f8f27fb7640 1 -- start start 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.405+0000 7f8f27fb7640 10 monclient: init 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient(hunting): picked mon.noname-a con 0x7f8f200a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:17.431 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient(hunting): picked mon.noname-b con 0x7f8f200a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient(hunting): picked mon.noname-c con 0x7f8f2013d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f25d53640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f8f200a1b10 legacy=0x7f8f2009c780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2009ed00 con 0x7f8f200a1b10 2024-09-16T04:26:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2009d5a0 con 0x7f8f200a2410 2024-09-16T04:26:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2009ef80 con 0x7f8f2013d060 2024-09-16T04:26:17.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f27fb7640 10 monclient(hunting): authenticate will time out at 1998.031376s 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f26554640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8f2013d060 0x7f8f201d3bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56738/0 (socket says 172.21.15.148:56738) 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f26554640 1 -- 172.21.15.148:0/1438280003 learned_addr learned my addr 172.21.15.148:0/1438280003 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177661385 0 0) 0x7f8f2009ef80 con 0x7f8f2013d060 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:17.433 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): my global_id is 154316 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f0c003550 con 0x7f8f2013d060 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042588890 0 0) 0x7f8f2009d5a0 con 0x7f8f200a2410 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:17.434 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): my global_id is 164194 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f2009ef80 con 0x7f8f200a2410 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1210856013 0 0) 0x7f8f0c003550 con 0x7f8f2013d060 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f2009d5a0 con 0x7f8f2013d060 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f18001980 con 0x7f8f2013d060 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:17.435 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): dump: 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:17.436 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4217173546 0 0) 0x7f8f2009d5a0 con 0x7f8f2013d060 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 >> v1:172.21.15.171:6789/0 conn(0x7f8f200a2410 legacy=0x7f8f2009ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:17.437 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.406+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 >> v1:172.21.15.148:6789/0 conn(0x7f8f200a1b10 legacy=0x7f8f2009c780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 1 monclient: found mon.c 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f201379e0 con 0x7f8f2013d060 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:26:17.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 5 monclient: authenticate success, global_id 154316 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 monclient: _renew_subs 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2013cab0 con 0x7f8f2013d060 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 monclient: _renew_subs 2024-09-16T04:26:17.439 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f20137420 con 0x7f8f2013d060 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f180038c0 con 0x7f8f2013d060 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f18003c90 con 0x7f8f2013d060 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:17.440 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1ffff640 4 set_mon_vals no callback set 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 10 monclient: dump: 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:17.441 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8f18003e30 con 0x7f8f2013d060 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:17.442 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.407+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f00005080 con 0x7f8f2013d060 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.408+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f8f18047b50 con 0x7f8f2013d060 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.408+0000 7f8f25d53640 10 monclient: get_auth_request con 0x7f8f0c038bc0 auth_method 0 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.412+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f8f18015dd0 con 0x7f8f2013d060 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.412+0000 7f8f1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:17.443 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.412+0000 7f8f1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:17.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.638+0000 7f8f27fb7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.638+0000 7f8f27fb7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.638+0000 7f8f27fb7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.638+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8f00005370 con 0x7f8f2013d060 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.639+0000 7f8f1e7fc640 1 -- 172.21.15.148:0/1438280003 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 2792114714) 0x7f8f180156d0 con 0x7f8f2013d060 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.639+0000 7f8f1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.639+0000 7f8f1e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:17.641 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 >> v1:172.21.15.171:6812/7327273 conn(0x7f8f0c038bc0 legacy=0x7f8f0c03b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 10 monclient: shutdown 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 >> v1:172.21.15.148:6790/0 conn(0x7f8f2013d060 legacy=0x7f8f201d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 shutdown_connections 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 >> 172.21.15.148:0/1438280003 conn(0x7f8f200a97b0 msgr2=0x7f8f200a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:17.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 shutdown_connections 2024-09-16T04:26:17.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:17.641+0000 7f8f27fb7640 1 -- 172.21.15.148:0/1438280003 wait complete. 2024-09-16T04:26:20.655 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:26:20.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.762+0000 7f4f9f067640 10 monclient: get_monmap_and_config 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.762+0000 7f4f9f067640 10 monclient: build_initial_monmap 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.762+0000 7f4f9f067640 10 monclient: monmap: 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:20.763247+0000 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:20.763247+0000 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:20.764 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:20.765 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 1 Processor -- start 2024-09-16T04:26:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 1 -- start start 2024-09-16T04:26:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient: init 2024-09-16T04:26:20.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f980a1010 con 0x7f4f9809bb70 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9809f030 con 0x7f4f9809c660 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9809dc60 con 0x7f4f9809cc30 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9f067640 10 monclient(hunting): authenticate will time out at 2001.388678s 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9ce03640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f9809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9d604640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f4f9809cc30 0x7f4f9813c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:56742/0 (socket says 172.21.15.148:56742) 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 learned_addr learned my addr 172.21.15.148:0/1342672948 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:20.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.763+0000 7f4f97fff640 1 --1- 172.21.15.148:0/1342672948 >> v1:172.21.15.171:6789/0 conn(0x7f4f9809c660 0x7f4f98138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.171:6789/0 says I am v1:172.21.15.148:51062/0 (socket says 172.21.15.148:51062) 2024-09-16T04:26:20.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:20.964+0000 7f4f9ce03640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f9809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:21.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:21.365+0000 7f4f9ce03640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f9809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:22.167 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:22.166+0000 7f4f9ce03640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f9809bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): tick 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6790/0 conn(0x7f4f9809cc30 legacy=0x7f4f9813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.171:6789/0 conn(0x7f4f9809c660 legacy=0x7f4f98138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f9809bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c0036c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:23.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:23.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:23.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9809df40 con 0x7f4f9809bb70 2024-09-16T04:26:23.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f980a1010 con 0x7f4f7c0036c0 2024-09-16T04:26:23.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:23.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c010370 con 0x7f4f9809cc30 2024-09-16T04:26:23.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.764+0000 7f4f967fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:23.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:23.965+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:24.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:24.366+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:25.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:25.167+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): tick 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6790/0 conn(0x7f4f9809cc30 legacy=0x7f4f7c0093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.171:6789/0 conn(0x7f4f7c0036c0 legacy=0x7f4f7c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:26.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c00f3f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c011620 con 0x7f4f9809bb70 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9809df40 con 0x7f4f7c00f3f0 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c0100a0 con 0x7f4f9809cc30 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f967fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:26.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.764+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:26.966 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:26.965+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:27.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:27.366+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.168 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.167+0000 7f4f9d604640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.400 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376086804 0 0) 0x7f4f7c0100a0 con 0x7f4f9809cc30 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 10 monclient(hunting): my global_id is 154319 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f88003520 con 0x7f4f9809cc30 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 842090212 0 0) 0x7f4f88003520 con 0x7f4f9809cc30 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.171:6789/0 conn(0x7f4f7c00f3f0 legacy=0x7f4f7c00f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f7c00cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 10 monclient: _finish_hunting 0 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 1 monclient: found mon.noname-b 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.397+0000 7f4f977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.401 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c004b10 con 0x7f4f9809cc30 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: _finish_auth 0 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f84001510 con 0x7f4f9809cc30 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 5 monclient: authenticate success, global_id 154319 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: dump: 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.402 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f840017f0 con 0x7f4f9809cc30 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 1 -- 172.21.15.148:0/1342672948 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f84001bc0 con 0x7f4f9809cc30 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.403 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f977fe640 10 monclient: dump: 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:28.404 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 4 set_mon_vals no callback set 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 10 monclient: shutdown 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.398+0000 7f4f9f067640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 1 -- 172.21.15.148:0/1342672948 >> v1:172.21.15.148:6790/0 conn(0x7f4f9809cc30 legacy=0x7f4f7c0093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 1 -- 172.21.15.148:0/1342672948 shutdown_connections 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 1 -- 172.21.15.148:0/1342672948 >> 172.21.15.148:0/1342672948 conn(0x7f4f980a5640 msgr2=0x7f4f980a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 1 -- 172.21.15.148:0/1342672948 shutdown_connections 2024-09-16T04:26:28.405 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 1 -- 172.21.15.148:0/1342672948 wait complete. 2024-09-16T04:26:28.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 10 monclient: build_initial_monmap 2024-09-16T04:26:28.409 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.399+0000 7f4f9f067640 10 monclient: monmap: 2024-09-16T04:26:28.409 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:28.409 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:28.409 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:28.409 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.400+0000 7f4f9f067640 1 Processor -- start 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.400+0000 7f4f9f067640 1 -- start start 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.400+0000 7f4f9f067640 10 monclient: init 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c00f3f0 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:28.410 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9809bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9809cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f98125580 con 0x7f4f9809bb70 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f981cc8f0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f980722c0 con 0x7f4f9809cc30 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9f067640 10 monclient(hunting): authenticate will time out at 2009.026220s 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f97fff640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f981cc1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9d604640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f4f9809cc30 0x7f4f981d3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49162/0 (socket says 172.21.15.148:49162) 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.401+0000 7f4f9d604640 1 -- 172.21.15.148:0/4123258526 learned_addr learned my addr 172.21.15.148:0/4123258526 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.404+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55108611 0 0) 0x7f4f981cc8f0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.404+0000 7f4f957fa640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.404+0000 7f4f957fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.404+0000 7f4f957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.411 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.404+0000 7f4f957fa640 10 monclient(hunting): my global_id is 164197 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.404+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f7c0115c0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1580716301 0 0) 0x7f4f7c0115c0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f981cc8f0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f8c0041e0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient(hunting): dump: 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.412 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2805561642 0 0) 0x7f4f981cc8f0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 >> v1:172.21.15.148:6790/0 conn(0x7f4f9809cc30 legacy=0x7f4f981d3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 >> v1:172.21.15.148:6789/0 conn(0x7f4f9809bb70 legacy=0x7f4f981cc1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: _finish_hunting 0 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 monclient: found mon.b 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:28.413 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f980a42d0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: _finish_auth 0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: _check_auth_tickets 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 5 monclient: authenticate success, global_id 164197 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 10 monclient: _renew_subs 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f980740a0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 10 monclient: _renew_subs 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f98122980 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f8c004630 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f4f8c004a00 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.414 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f96ffd640 4 set_mon_vals no callback set 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.405+0000 7f4f957fa640 10 monclient: dump: 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.415 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f4f8c004ba0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f4f8c0478e0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f9f067640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f9f067640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f9f067640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f78005080 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.406+0000 7f4f97fff640 10 monclient: get_auth_request con 0x7f4f7c039f10 auth_method 0 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.411+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f4f8c015ae0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.416 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.411+0000 7f4f957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.417 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.411+0000 7f4f957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.646+0000 7f4f9f067640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:26:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.646+0000 7f4f9f067640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.646+0000 7f4f9f067640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.171:6789/0 2024-09-16T04:26:28.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.646+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 --> v1:172.21.15.171:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f78005370 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.647+0000 7f4f957fa640 1 -- 172.21.15.148:0/4123258526 <== mon.1 v1:172.21.15.171:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3796884351) 0x7f4f8c0153e0 con 0x7f4f7c00f3f0 2024-09-16T04:26:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.647+0000 7f4f957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:26:28.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.647+0000 7f4f957fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:28.649 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 >> v1:172.21.15.171:6812/7327273 conn(0x7f4f7c039f10 legacy=0x7f4f7c03c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 10 monclient: shutdown 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 >> v1:172.21.15.171:6789/0 conn(0x7f4f7c00f3f0 legacy=0x7f4f981cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 shutdown_connections 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 >> 172.21.15.148:0/4123258526 conn(0x7f4f980a5640 msgr2=0x7f4f981b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 shutdown_connections 2024-09-16T04:26:28.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.649+0000 7f4f9f067640 1 -- 172.21.15.148:0/4123258526 wait complete. 2024-09-16T04:26:28.662 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:26:28.662 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:26:28.662 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:26:28.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.759+0000 7f635936d640 10 monclient: get_monmap_and_config 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.759+0000 7f635936d640 10 monclient: build_initial_monmap 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.759+0000 7f635936d640 10 monclient: monmap: 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:28.760457+0000 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:28.760457+0000 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:28.775 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 1 Processor -- start 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 1 -- start start 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient: init 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient(hunting): picked mon.noname-c con 0x7f635409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient(hunting): picked mon.noname-a con 0x7f635409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient(hunting): picked mon.noname-b con 0x7f635409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63540a1010 con 0x7f635409c660 2024-09-16T04:26:28.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635409f030 con 0x7f635409bb70 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635409dc60 con 0x7f635409cc30 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f635936d640 10 monclient(hunting): authenticate will time out at 2009.385632s 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f63527fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f635409c660 legacy=0x7f6354138960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f63537fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f635409cc30 0x7f635413c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49174/0 (socket says 172.21.15.148:49174) 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f63537fe640 1 -- 172.21.15.148:0/23226007 learned_addr learned my addr 172.21.15.148:0/23226007 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480438223 0 0) 0x7f635409dc60 con 0x7f635409cc30 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f6351ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f6351ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f6351ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f6351ffb640 10 monclient(hunting): my global_id is 154322 2024-09-16T04:26:28.777 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.760+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f633c003520 con 0x7f635409cc30 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3521234321 0 0) 0x7f633c003520 con 0x7f635409cc30 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 >> v1:172.21.15.171:6789/0 conn(0x7f635409bb70 legacy=0x7f635409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 >> v1:172.21.15.148:6789/0 conn(0x7f635409c660 legacy=0x7f6354138960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 monclient: found mon.noname-b 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635409df40 con 0x7f635409cc30 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: _finish_auth 0 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:28.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f634c001990 con 0x7f635409cc30 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 5 monclient: authenticate success, global_id 154322 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: dump: 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.779 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f634c001c80 con 0x7f635409cc30 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 1 -- 172.21.15.148:0/23226007 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f634c002090 con 0x7f635409cc30 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 4 set_mon_vals no callback set 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f6351ffb640 10 monclient: dump: 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.780 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 10 monclient: shutdown 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 1 -- 172.21.15.148:0/23226007 >> v1:172.21.15.148:6790/0 conn(0x7f635409cc30 legacy=0x7f635413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 1 -- 172.21.15.148:0/23226007 shutdown_connections 2024-09-16T04:26:28.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 1 -- 172.21.15.148:0/23226007 >> 172.21.15.148:0/23226007 conn(0x7f63540a5640 msgr2=0x7f63540a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 1 -- 172.21.15.148:0/23226007 shutdown_connections 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.761+0000 7f635936d640 1 -- 172.21.15.148:0/23226007 wait complete. 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.762+0000 7f635936d640 10 monclient: build_initial_monmap 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.762+0000 7f635936d640 10 monclient: monmap: 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:28.782 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.762+0000 7f635936d640 1 Processor -- start 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.762+0000 7f635936d640 1 -- start start 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.762+0000 7f635936d640 10 monclient: init 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient(hunting): picked mon.noname-c con 0x7f635409bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient(hunting): picked mon.noname-b con 0x7f635409c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient(hunting): picked mon.noname-a con 0x7f635409cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635411f7a0 con 0x7f635409cc30 2024-09-16T04:26:28.783 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63541c8550 con 0x7f635409c660 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6354132440 con 0x7f635409bb70 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient(hunting): authenticate will time out at 2009.388308s 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f63537fe640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f635409cc30 legacy=0x7f63541cf990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f6352ffd640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f635409bb70 0x7f63541c7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49176/0 (socket says 172.21.15.148:49176) 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f6352ffd640 1 -- 172.21.15.148:0/3751511730 learned_addr learned my addr 172.21.15.148:0/3751511730 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1713943334 0 0) 0x7f6354132440 con 0x7f635409bb70 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): my global_id is 154325 2024-09-16T04:26:28.784 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6340003520 con 0x7f635409bb70 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555173432 0 0) 0x7f63541c8550 con 0x7f635409c660 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): my global_id is 164203 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6354132440 con 0x7f635409c660 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2943532248 0 0) 0x7f6340003520 con 0x7f635409bb70 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63541c8550 con 0x7f635409bb70 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6344001970 con 0x7f635409bb70 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:28.785 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): dump: 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 411077822 0 0) 0x7f63541c8550 con 0x7f635409bb70 2024-09-16T04:26:28.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.171:6789/0 conn(0x7f635409c660 legacy=0x7f63541c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.148:6789/0 conn(0x7f635409cc30 legacy=0x7f63541cf990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 monclient: found mon.c 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63540a4060 con 0x7f635409bb70 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f633b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 5 monclient: authenticate success, global_id 154325 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient: _renew_subs 2024-09-16T04:26:28.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63540740a0 con 0x7f635409bb70 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient: _renew_subs 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.763+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6354121350 con 0x7f635409bb70 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f63440038c0 con 0x7f635409bb70 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6344003af0 con 0x7f635409bb70 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f63517fa640 4 set_mon_vals no callback set 2024-09-16T04:26:28.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 10 monclient: dump: 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f635936d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f63340065c0 con 0x7f6334002860 2024-09-16T04:26:28.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.764+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f6344003c90 con 0x7f635409bb70 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f6344047ac0 con 0x7f635409bb70 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 941848459 0 0) 0x7f63340065c0 con 0x7f6334002860 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient(hunting): my global_id is 154328 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63541c8550 con 0x7f6334002860 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f63537fe640 10 monclient: get_auth_request con 0x7f6340038fb0 auth_method 0 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1431915289 0 0) 0x7f63541c8550 con 0x7f6334002860 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63340065c0 con 0x7f6334002860 2024-09-16T04:26:28.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6348003fe0 con 0x7f6334002860 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient: dump: 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.791 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2446939945 0 0) 0x7f63340065c0 con 0x7f6334002860 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f633400a150 con 0x7f6334002860 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6348005440 con 0x7f6334002860 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.148:6790/0 conn(0x7f6334002860 legacy=0x7f6334004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.765+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.148:6790/0 conn(0x7f6334002860 legacy=0x7f6334004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.771+0000 7f635936d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.771+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6334009e20 con 0x7f63340090e0 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3250503656 0 0) 0x7f6334009e20 con 0x7f63340090e0 2024-09-16T04:26:28.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient(hunting): my global_id is 154331 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63340065c0 con 0x7f63340090e0 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 826871507 0 0) 0x7f63340065c0 con 0x7f63340090e0 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6334009e20 con 0x7f63340090e0 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f63480046e0 con 0x7f63340090e0 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient: dump: 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.793 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2930961318 0 0) 0x7f6334009e20 con 0x7f63340090e0 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f633400c390 con 0x7f63340090e0 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4133 (unknown 0 0 1559547470) 0x7f6348004a70 con 0x7f63340090e0 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:28.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.148:6790/0 conn(0x7f63340090e0 legacy=0x7f6334009570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f633b7fe640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.148:6790/0 conn(0x7f63340090e0 legacy=0x7f6334009570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 78, 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 125679, 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:28.795 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:28.796 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:28.797 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:28.798 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:28.799 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.800 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.801 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.802 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.803 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:28.804 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.805 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.806 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.807 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:28.807 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.171:6812/7327273 conn(0x7f6340038fb0 legacy=0x7f634003b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f635936d640 10 monclient: shutdown 2024-09-16T04:26:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f635936d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 >> v1:172.21.15.148:6790/0 conn(0x7f635409bb70 legacy=0x7f63541c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.772+0000 7f63537fe640 1 -- 172.21.15.148:0/3751511730 reap_dead start 2024-09-16T04:26:28.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.773+0000 7f63537fe640 1 -- 172.21.15.148:0/3751511730 reap_dead start 2024-09-16T04:26:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.773+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 shutdown_connections 2024-09-16T04:26:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.773+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 >> 172.21.15.148:0/3751511730 conn(0x7f63540a5640 msgr2=0x7f63540a31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.773+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 shutdown_connections 2024-09-16T04:26:28.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.773+0000 7f635936d640 1 -- 172.21.15.148:0/3751511730 wait complete. 2024-09-16T04:26:28.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:26:28.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:26:28.809 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:26:28.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.906+0000 7f2be54d0640 10 monclient: get_monmap_and_config 2024-09-16T04:26:28.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.906+0000 7f2be54d0640 10 monclient: build_initial_monmap 2024-09-16T04:26:28.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.907+0000 7f2be54d0640 10 monclient: monmap: 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:26:28.907909+0000 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:26:28.907909+0000 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:26:28.909 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.907+0000 7f2be54d0640 1 Processor -- start 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.907+0000 7f2be54d0640 1 -- start start 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.907+0000 7f2be54d0640 10 monclient: init 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2be009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:28.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2be009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2be009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be00a1010 con 0x7f2be009bb70 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bdeffd640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2be009bb70 legacy=0x7f2be009bf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be009f030 con 0x7f2be009c660 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be009dc60 con 0x7f2be009cc30 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient(hunting): authenticate will time out at 2009.533284s 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bdf7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2be009cc30 0x7f2be013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49200/0 (socket says 172.21.15.148:49200) 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bdf7fe640 1 -- 172.21.15.148:0/1374108994 learned_addr learned my addr 172.21.15.148:0/1374108994 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367103275 0 0) 0x7f2be009dc60 con 0x7f2be009cc30 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): my global_id is 154334 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bcc003520 con 0x7f2be009cc30 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889429614 0 0) 0x7f2be009f030 con 0x7f2be009c660 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient(hunting): my global_id is 164206 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be009dc60 con 0x7f2be009c660 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3081453709 0 0) 0x7f2bcc003520 con 0x7f2be009cc30 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 >> v1:172.21.15.171:6789/0 conn(0x7f2be009c660 legacy=0x7f2be0138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 >> v1:172.21.15.148:6789/0 conn(0x7f2be009bb70 legacy=0x7f2be009bf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 monclient: found mon.noname-b 2024-09-16T04:26:28.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be009df40 con 0x7f2be009cc30 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: _finish_auth 0 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 5 monclient: authenticate success, global_id 154334 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd8001990 con 0x7f2be009cc30 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: dump: 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.920 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2bd8001c40 con 0x7f2be009cc30 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 1 -- 172.21.15.148:0/1374108994 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd8002010 con 0x7f2be009cc30 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:26:28.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2be54d0640 4 set_mon_vals no callback set 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.908+0000 7f2bde7fc640 10 monclient: dump: 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.922 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 10 monclient: shutdown 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 1 -- 172.21.15.148:0/1374108994 >> v1:172.21.15.148:6790/0 conn(0x7f2be009cc30 legacy=0x7f2be013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 1 -- 172.21.15.148:0/1374108994 shutdown_connections 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 1 -- 172.21.15.148:0/1374108994 >> 172.21.15.148:0/1374108994 conn(0x7f2be00a5640 msgr2=0x7f2be00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 1 -- 172.21.15.148:0/1374108994 shutdown_connections 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 1 -- 172.21.15.148:0/1374108994 wait complete. 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 10 monclient: build_initial_monmap 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.909+0000 7f2be54d0640 10 monclient: monmap: 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:26:28.923 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 1 Processor -- start 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 1 -- start start 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 10 monclient: init 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2be009bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2be009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:26:28.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2be009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.910+0000 7f2be54d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be01335d0 con 0x7f2be009c660 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be0134c20 con 0x7f2be009bb70 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd67fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7f2be009c660 legacy=0x7f2be0132570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.148:6789/0 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be0132c80 con 0x7f2be009cc30 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 10 monclient(hunting): authenticate will time out at 2009.536037s 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bdf7fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f2be009cc30 0x7f2be01d1a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:49204/0 (socket says 172.21.15.148:49204) 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bdf7fe640 1 -- 172.21.15.148:0/2881955454 learned_addr learned my addr 172.21.15.148:0/2881955454 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:26:28.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013234174 0 0) 0x7f2be0132c80 con 0x7f2be009cc30 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): my global_id is 154337 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bc0003520 con 0x7f2be009cc30 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070269382 0 0) 0x7f2be0134c20 con 0x7f2be009bb70 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): my global_id is 164209 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be0132c80 con 0x7f2be009bb70 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1037969780 0 0) 0x7f2bc0003520 con 0x7f2be009cc30 2024-09-16T04:26:28.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be0134c20 con 0x7f2be009cc30 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd8001f90 con 0x7f2be009cc30 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): dump: 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.927 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4258290919 0 0) 0x7f2be0132c80 con 0x7f2be009bb70 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bc0003520 con 0x7f2be009bb70 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd00040c0 con 0x7f2be009bb70 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient(hunting): dump: 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.928 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3458996514 0 0) 0x7f2be0134c20 con 0x7f2be009cc30 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.171:6789/0 conn(0x7f2be009bb70 legacy=0x7f2be0131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.148:6789/0 conn(0x7f2be009c660 legacy=0x7f2be0132570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 monclient: found mon.c 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be006e320 con 0x7f2be009cc30 2024-09-16T04:26:28.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: _finish_auth 0 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 5 monclient: authenticate success, global_id 154337 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 10 monclient: _renew_subs 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2be009b6d0 con 0x7f2be009cc30 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 10 monclient: _renew_subs 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2be0133870 con 0x7f2be009cc30 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2bd8004fb0 con 0x7f2be009cc30 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd8005300 con 0x7f2be009cc30 2024-09-16T04:26:28.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bd7fff640 10 monclient: dump: 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.911+0000 7f2bddffb640 4 set_mon_vals no callback set 2024-09-16T04:26:28.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.912+0000 7f2be54d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.912+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2be00008d0 con 0x7f2be009aa90 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.912+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f2bd8005560 con 0x7f2be009cc30 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.912+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f2bd8047e50 con 0x7f2be009cc30 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd67fc640 10 monclient: get_auth_request con 0x7f2bc0038ef0 auth_method 0 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304668246 0 0) 0x7f2be00008d0 con 0x7f2be009aa90 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient(hunting): my global_id is 164212 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be0134c20 con 0x7f2be009aa90 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4289500998 0 0) 0x7f2be0134c20 con 0x7f2be009aa90 2024-09-16T04:26:28.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be00008d0 con 0x7f2be009aa90 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd0004720 con 0x7f2be009aa90 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient: dump: 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.933 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1742491243 0 0) 0x7f2be00008d0 con 0x7f2be009aa90 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2be0134a60 con 0x7f2be009aa90 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2bd0006b30 con 0x7f2be009aa90 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.171:6789/0 conn(0x7f2be009aa90 legacy=0x7f2be006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.913+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.171:6789/0 conn(0x7f2be009aa90 legacy=0x7f2be006f620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:28.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.919+0000 7f2be54d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.919+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2be00a0870 con 0x7f2be002d650 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232151710 0 0) 0x7f2be00a0870 con 0x7f2be002d650 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient(hunting): _negotiate 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient(hunting): my global_id is 164215 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be00008d0 con 0x7f2be002d650 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3759665814 0 0) 0x7f2be00008d0 con 0x7f2be002d650 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be00a0870 con 0x7f2be002d650 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2bd0004e60 con 0x7f2be002d650 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:26:28.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient: dump: 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2118096455 0 0) 0x7f2be00a0870 con 0x7f2be002d650 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2be009ae70 con 0x7f2be002d650 2024-09-16T04:26:28.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4301 (unknown 0 0 4004141826) 0x7f2bd00050c0 con 0x7f2be002d650 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.171:6789/0 conn(0x7f2be002d650 legacy=0x7f2be006fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.920+0000 7f2bd7fff640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.171:6789/0 conn(0x7f2be002d650 legacy=0x7f2be006fd30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 78, 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 17928, 2024-09-16T04:26:28.938 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:28.939 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:28.940 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:26:28.941 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:26:28.942 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:26:28.943 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:26:28.944 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:26:28.945 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:26:28.946 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:26:28.947 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:26:28.948 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:26:28.949 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:26:28.950 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:26:28.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.171:6812/7327273 conn(0x7f2bc0038ef0 legacy=0x7f2bc003b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 10 monclient: shutdown 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 >> v1:172.21.15.148:6790/0 conn(0x7f2be009cc30 legacy=0x7f2be01d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2bdf7fe640 1 -- 172.21.15.148:0/2881955454 reap_dead start 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2bdf7fe640 1 -- 172.21.15.148:0/2881955454 reap_dead start 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 shutdown_connections 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 >> 172.21.15.148:0/2881955454 conn(0x7f2be00a5640 msgr2=0x7f2be00a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 shutdown_connections 2024-09-16T04:26:28.952 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:26:28.921+0000 7f2be54d0640 1 -- 172.21.15.148:0/2881955454 wait complete. 2024-09-16T04:26:28.952 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:26:28.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T04:27:58.954 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:27:58.955 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:27:58.955 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:27:58.955 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:27:58.959 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:27:58.959 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:27:58.959 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.067+0000 7f88f7c9b640 10 monclient: get_monmap_and_config 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.067+0000 7f88f7c9b640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.067+0000 7f88f7c9b640 10 monclient: monmap: 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:27:59.069710+0000 2024-09-16T04:27:59.070 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:27:59.069710+0000 2024-09-16T04:27:59.071 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.071 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.071 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.071 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:27:59.071 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:27:59.071 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.068+0000 7f88f7c9b640 1 Processor -- start 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.068+0000 7f88f7c9b640 1 -- start start 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.068+0000 7f88f7c9b640 10 monclient: init 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient(hunting): picked mon.noname-a con 0x7f88f00a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient(hunting): picked mon.noname-c con 0x7f88f00a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient(hunting): picked mon.noname-b con 0x7f88f013d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f00a0e00 con 0x7f88f00a2410 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f00a10e0 con 0x7f88f00a1b10 2024-09-16T04:27:59.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f00a4970 con 0x7f88f013d060 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f7c9b640 10 monclient(hunting): authenticate will time out at 2099.695243s 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f5a37640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f88f00a2410 0x7f88f00a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40636/0 (socket says 172.21.15.148:40636) 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f6238640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88f013d060 0x7f88f013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51806/0 (socket says 172.21.15.148:51806) 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f5a37640 1 -- 172.21.15.148:0/2690752364 learned_addr learned my addr 172.21.15.148:0/2690752364 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 47807773 0 0) 0x7f88f00a10e0 con 0x7f88f00a1b10 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient(hunting): my global_id is 164218 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88dc003520 con 0x7f88f00a1b10 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3868293370 0 0) 0x7f88dc003520 con 0x7f88f00a1b10 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 >> v1:172.21.15.148:6790/0 conn(0x7f88f013d060 legacy=0x7f88f013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 >> v1:172.21.15.148:6789/0 conn(0x7f88f00a2410 legacy=0x7f88f00a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 monclient: found mon.noname-c 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.171:6789/0 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 --> v1:172.21.15.171:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f006dfd0 con 0x7f88f00a1b10 2024-09-16T04:27:59.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.069+0000 7f88f4a35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 5 monclient: authenticate success, global_id 164218 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88e0001610 con 0x7f88f00a1b10 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: dump: 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.078 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 <== mon.1 v1:172.21.15.171:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88e0001900 con 0x7f88f00a1b10 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 1 -- 172.21.15.148:0/2690752364 <== mon.1 v1:172.21.15.171:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88e0001d10 con 0x7f88f00a1b10 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 10 monclient: get_monmap_and_config success 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 4 set_mon_vals no callback set 2024-09-16T04:27:59.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f4a35640 10 monclient: dump: 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.080 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 10 monclient: shutdown 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2690752364 >> v1:172.21.15.171:6789/0 conn(0x7f88f00a1b10 legacy=0x7f88f013b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.070+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2690752364 shutdown_connections 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.071+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2690752364 >> 172.21.15.148:0/2690752364 conn(0x7f88f00a97b0 msgr2=0x7f88f00a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.071+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2690752364 shutdown_connections 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.071+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2690752364 wait complete. 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.071+0000 7f88f7c9b640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.071+0000 7f88f7c9b640 10 monclient: monmap: 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.081 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.072+0000 7f88f7c9b640 1 Processor -- start 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.072+0000 7f88f7c9b640 1 -- start start 2024-09-16T04:27:59.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.072+0000 7f88f7c9b640 10 monclient: init 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient(hunting): picked mon.noname-c con 0x7f88f00a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient(hunting): picked mon.noname-a con 0x7f88f00a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient(hunting): picked mon.noname-b con 0x7f88f013d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f0123f60 con 0x7f88f00a2410 2024-09-16T04:27:59.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f0122130 con 0x7f88f013d060 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f013cd00 con 0x7f88f00a1b10 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f7c9b640 10 monclient(hunting): authenticate will time out at 2099.699261s 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f5a37640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f88f00a1b10 0x7f88f011fe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51808/0 (socket says 172.21.15.148:51808) 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88f5a37640 1 -- 172.21.15.148:0/2939460196 learned_addr learned my addr 172.21.15.148:0/2939460196 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1411310491 0 0) 0x7f88f013cd00 con 0x7f88f00a1b10 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): my global_id is 154343 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88d4003520 con 0x7f88f00a1b10 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705260084 0 0) 0x7f88f0122130 con 0x7f88f013d060 2024-09-16T04:27:59.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): my global_id is 164221 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f013cd00 con 0x7f88f013d060 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2195985401 0 0) 0x7f88d4003520 con 0x7f88f00a1b10 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88f0122130 con 0x7f88f00a1b10 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88ec0017a0 con 0x7f88f00a1b10 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): dump: 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.089 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.090 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.073+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1795612725 0 0) 0x7f88f013cd00 con 0x7f88f013d060 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88d4003520 con 0x7f88f013d060 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88e8001aa0 con 0x7f88f013d060 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient(hunting): dump: 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.091 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1725396337 0 0) 0x7f88f0122130 con 0x7f88f00a1b10 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 >> v1:172.21.15.171:6789/0 conn(0x7f88f013d060 legacy=0x7f88f01d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 >> v1:172.21.15.148:6789/0 conn(0x7f88f00a2410 legacy=0x7f88f0120530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 monclient: found mon.c 2024-09-16T04:27:59.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f00a0130 con 0x7f88f00a1b10 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 5 monclient: authenticate success, global_id 154343 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 10 monclient: _renew_subs 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f88f0123860 con 0x7f88f00a1b10 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 10 monclient: _renew_subs 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.074+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f88f01225d0 con 0x7f88f00a1b10 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.075+0000 7f88f7c9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.075+0000 7f88f7c9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.075+0000 7f88f7c9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.075+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f013f470 con 0x7f88f00a1b10 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f88ec0047e0 con 0x7f88f00a1b10 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f88ec004b30 con 0x7f88f00a1b10 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e67fc640 10 monclient: dump: 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.094 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.077+0000 7f88e7fff640 4 set_mon_vals no callback set 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.078+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f88ec004d10 con 0x7f88f00a1b10 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.078+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f88ec047940 con 0x7f88f00a1b10 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.078+0000 7f88f5236640 10 monclient: get_auth_request con 0x7f88d4038ec0 auth_method 0 2024-09-16T04:27:59.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.082+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f88ec015b40 con 0x7f88f00a1b10 2024-09-16T04:27:59.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.082+0000 7f88e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.082+0000 7f88e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:27:59.326 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88f7c9b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88f7c9b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88f7c9b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f88f0062630 con 0x7f88f00a1b10 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88e67fc640 1 -- 172.21.15.148:0/2939460196 <== mon.2 v1:172.21.15.148:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 3896721907) 0x7f88ec015440 con 0x7f88f00a1b10 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.324+0000 7f88e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:27:59.327 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:27:59.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 >> v1:172.21.15.171:6812/7327273 conn(0x7f88d4038ec0 legacy=0x7f88d403b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 10 monclient: shutdown 2024-09-16T04:27:59.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 >> v1:172.21.15.148:6790/0 conn(0x7f88f00a1b10 legacy=0x7f88f011fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 shutdown_connections 2024-09-16T04:27:59.329 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 >> 172.21.15.148:0/2939460196 conn(0x7f88f00a97b0 msgr2=0x7f88f013feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 shutdown_connections 2024-09-16T04:27:59.330 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.326+0000 7f88f7c9b640 1 -- 172.21.15.148:0/2939460196 wait complete. 2024-09-16T04:27:59.340 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:27:59.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:27:59.340 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.435+0000 7f83c4aa6640 10 monclient: get_monmap_and_config 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.435+0000 7f83c4aa6640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.435+0000 7f83c4aa6640 10 monclient: monmap: 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:27:59.437641+0000 2024-09-16T04:27:59.438 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:27:59.437641+0000 2024-09-16T04:27:59.439 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.439 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.439 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.439 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:27:59.439 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:27:59.439 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 1 Processor -- start 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 1 -- start start 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 10 monclient: init 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 10 monclient(hunting): picked mon.noname-b con 0x7f83c009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 10 monclient(hunting): picked mon.noname-c con 0x7f83c009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.449 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 10 monclient(hunting): picked mon.noname-a con 0x7f83c009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.436+0000 7f83c4aa6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c00a1010 con 0x7f83c009cc30 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c009f030 con 0x7f83c009c660 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c009dc60 con 0x7f83c009bb70 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 10 monclient(hunting): authenticate will time out at 2100.063027s 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83be59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f83c009bb70 0x7f83c009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51818/0 (socket says 172.21.15.148:51818) 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83be59c640 1 -- 172.21.15.148:0/307299097 learned_addr learned my addr 172.21.15.148:0/307299097 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bed9d640 1 --1- 172.21.15.148:0/307299097 >> v1:172.21.15.148:6789/0 conn(0x7f83c009cc30 0x7f83c013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40664/0 (socket says 172.21.15.148:40664) 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 910755296 0 0) 0x7f83c009dc60 con 0x7f83c009bb70 2024-09-16T04:27:59.450 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): my global_id is 154346 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83ac003520 con 0x7f83c009bb70 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174887313 0 0) 0x7f83c00a1010 con 0x7f83c009cc30 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): my global_id is 174099 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83c009dc60 con 0x7f83c009cc30 2024-09-16T04:27:59.451 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 772900734 0 0) 0x7f83c009f030 con 0x7f83c009c660 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient(hunting): my global_id is 164224 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83c00a1010 con 0x7f83c009c660 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529410639 0 0) 0x7f83c009dc60 con 0x7f83c009cc30 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 >> v1:172.21.15.148:6790/0 conn(0x7f83c009bb70 legacy=0x7f83c009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 >> v1:172.21.15.171:6789/0 conn(0x7f83c009c660 legacy=0x7f83c0138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.452 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 monclient: found mon.noname-a 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c009df40 con 0x7f83c009cc30 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83b80018f0 con 0x7f83c009cc30 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: dump: 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.453 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 5 monclient: authenticate success, global_id 174099 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83b8001b60 con 0x7f83c009cc30 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 1 -- 172.21.15.148:0/307299097 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83b8001f70 con 0x7f83c009cc30 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 10 monclient: get_monmap_and_config success 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83c4aa6640 4 set_mon_vals no callback set 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.437+0000 7f83bdd9b640 10 monclient: dump: 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.455 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 10 monclient: shutdown 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 1 -- 172.21.15.148:0/307299097 >> v1:172.21.15.148:6789/0 conn(0x7f83c009cc30 legacy=0x7f83c013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 1 -- 172.21.15.148:0/307299097 shutdown_connections 2024-09-16T04:27:59.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 1 -- 172.21.15.148:0/307299097 >> 172.21.15.148:0/307299097 conn(0x7f83c00a5640 msgr2=0x7f83c00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 1 -- 172.21.15.148:0/307299097 shutdown_connections 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.438+0000 7f83c4aa6640 1 -- 172.21.15.148:0/307299097 wait complete. 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient: monmap: 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.457 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 1 Processor -- start 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 1 -- start start 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient: init 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient(hunting): picked mon.noname-a con 0x7f83c009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient(hunting): picked mon.noname-b con 0x7f83c009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.458 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient(hunting): picked mon.noname-c con 0x7f83c009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c011c5d0 con 0x7f83c009bb70 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c011c8b0 con 0x7f83c009c660 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c011b9b0 con 0x7f83c009cc30 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83c4aa6640 10 monclient(hunting): authenticate will time out at 2100.065866s 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83be59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f83c009bb70 0x7f83c013a410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40672/0 (socket says 172.21.15.148:40672) 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.439+0000 7f83be59c640 1 -- 172.21.15.148:0/3803925392 learned_addr learned my addr 172.21.15.148:0/3803925392 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397016498 0 0) 0x7f83c011c5d0 con 0x7f83c009bb70 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.459 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): my global_id is 174102 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83a0003520 con 0x7f83c009bb70 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475997170 0 0) 0x7f83c011b9b0 con 0x7f83c009cc30 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): my global_id is 154349 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83c011c5d0 con 0x7f83c009cc30 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8373727 0 0) 0x7f83c011c8b0 con 0x7f83c009c660 2024-09-16T04:27:59.460 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): my global_id is 164227 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83c011b9b0 con 0x7f83c009c660 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1137137418 0 0) 0x7f83a0003520 con 0x7f83c009bb70 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83c011c8b0 con 0x7f83c009bb70 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83b0001380 con 0x7f83c009bb70 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): dump: 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.461 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 669801990 0 0) 0x7f83c011c5d0 con 0x7f83c009cc30 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83a0003520 con 0x7f83c009cc30 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83b8001eb0 con 0x7f83c009cc30 2024-09-16T04:27:59.462 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): dump: 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.463 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3226319530 0 0) 0x7f83c011b9b0 con 0x7f83c009c660 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83c011c5d0 con 0x7f83c009c660 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83a8001870 con 0x7f83c009c660 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient(hunting): dump: 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.464 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3908564018 0 0) 0x7f83c011c8b0 con 0x7f83c009bb70 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.148:6790/0 conn(0x7f83c009cc30 legacy=0x7f83c01d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.171:6789/0 conn(0x7f83c009c660 legacy=0x7f83c013ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.465 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 monclient: found mon.a 2024-09-16T04:27:59.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c013be40 con 0x7f83c009bb70 2024-09-16T04:27:59.466 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 5 monclient: authenticate success, global_id 174102 2024-09-16T04:27:59.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 10 monclient: _renew_subs 2024-09-16T04:27:59.468 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83c011c010 con 0x7f83c009bb70 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 10 monclient: _renew_subs 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83c013b230 con 0x7f83c009bb70 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83b0001670 con 0x7f83c009bb70 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83b0004c30 con 0x7f83c009bb70 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83bd59a640 4 set_mon_vals no callback set 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.440+0000 7f83b6ffd640 10 monclient: dump: 2024-09-16T04:27:59.469 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83c4aa6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f83b000fec0 con 0x7f83c009bb70 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83a40065c0 con 0x7f83a4002860 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f83b00478a0 con 0x7f83c009bb70 2024-09-16T04:27:59.470 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2951316950 0 0) 0x7f83a40065c0 con 0x7f83a4002860 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient(hunting): my global_id is 174105 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83c011c8b0 con 0x7f83a4002860 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83bed9d640 10 monclient: get_auth_request con 0x7f83a0039170 auth_method 0 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2441102830 0 0) 0x7f83c011c8b0 con 0x7f83a4002860 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83a40065c0 con 0x7f83a4002860 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83a8003790 con 0x7f83a4002860 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.441+0000 7f83b6ffd640 10 monclient: dump: 2024-09-16T04:27:59.471 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1703959635 0 0) 0x7f83a40065c0 con 0x7f83a4002860 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83a400a150 con 0x7f83a4002860 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f83a8005a30 con 0x7f83a4002860 2024-09-16T04:27:59.472 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:27:59.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.148:6789/0 conn(0x7f83a4002860 legacy=0x7f83a4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.473 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.442+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.148:6789/0 conn(0x7f83a4002860 legacy=0x7f83a4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83c4aa6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:27:59.474 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83a4009e40 con 0x7f83a4009100 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2199731950 0 0) 0x7f83a4009e40 con 0x7f83a4009100 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient(hunting): my global_id is 174108 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83a40065c0 con 0x7f83a4009100 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1545774363 0 0) 0x7f83a40065c0 con 0x7f83a4009100 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83a4009e40 con 0x7f83a4009100 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f83a8005100 con 0x7f83a4009100 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.448+0000 7f83b6ffd640 10 monclient: dump: 2024-09-16T04:27:59.475 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 534409083 0 0) 0x7f83a4009e40 con 0x7f83a4009100 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 --> v1:172.21.15.148:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83a400c3d0 con 0x7f83a4009100 2024-09-16T04:27:59.476 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 <== mon.0 v1:172.21.15.148:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3976 (unknown 0 0 3103435236) 0x7f83a80046a0 con 0x7f83a4009100 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.148:6789/0 conn(0x7f83a4009100 legacy=0x7f83a4009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83b6ffd640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.148:6789/0 conn(0x7f83a4009100 legacy=0x7f83a4009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "leader", 2024-09-16T04:27:59.477 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 82, 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 404, 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.478 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:27:59.479 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.480 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:27:59.481 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.482 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.483 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.484 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.485 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.486 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.487 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.488 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:27:59.489 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.489 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.489 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.489 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:27:59.489 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:27:59.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.171:6812/7327273 conn(0x7f83a0039170 legacy=0x7f83a003b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 10 monclient: shutdown 2024-09-16T04:27:59.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 >> v1:172.21.15.148:6789/0 conn(0x7f83c009bb70 legacy=0x7f83c013a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83bed9d640 1 -- 172.21.15.148:0/3803925392 reap_dead start 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83bed9d640 1 -- 172.21.15.148:0/3803925392 reap_dead start 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 shutdown_connections 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 >> 172.21.15.148:0/3803925392 conn(0x7f83c00a5640 msgr2=0x7f83c013cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 shutdown_connections 2024-09-16T04:27:59.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.449+0000 7f83c4aa6640 1 -- 172.21.15.148:0/3803925392 wait complete. 2024-09-16T04:27:59.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:27:59.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:27:59.492 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:27:59.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.588+0000 7f7e29bcf640 10 monclient: get_monmap_and_config 2024-09-16T04:27:59.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.588+0000 7f7e29bcf640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.590 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.588+0000 7f7e29bcf640 10 monclient: monmap: 2024-09-16T04:27:59.590 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:27:59.590054+0000 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:27:59.590054+0000 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:27:59.591 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.588+0000 7f7e29bcf640 1 Processor -- start 2024-09-16T04:27:59.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.588+0000 7f7e29bcf640 1 -- start start 2024-09-16T04:27:59.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.588+0000 7f7e29bcf640 10 monclient: init 2024-09-16T04:27:59.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.600 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e240a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e240a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2413d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e240a0e00 con 0x7f7e240a1b10 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e240a10e0 con 0x7f7e2413d060 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e240a4970 con 0x7f7e240a2410 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e29bcf640 10 monclient(hunting): authenticate will time out at 2100.215465s 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e237fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f7e240a2410 0x7f7e240a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51832/0 (socket says 172.21.15.148:51832) 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e237fe640 1 -- 172.21.15.148:0/3009482878 learned_addr learned my addr 172.21.15.148:0/3009482878 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e1affd640 1 --1- 172.21.15.148:0/3009482878 >> v1:172.21.15.148:6789/0 conn(0x7f7e240a1b10 0x7f7e2413b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40692/0 (socket says 172.21.15.148:40692) 2024-09-16T04:27:59.601 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3551099150 0 0) 0x7f7e240a4970 con 0x7f7e240a2410 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): my global_id is 154352 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e10003520 con 0x7f7e240a2410 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840199449 0 0) 0x7f7e240a0e00 con 0x7f7e240a1b10 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 10 monclient(hunting): my global_id is 174111 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.589+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e240a4970 con 0x7f7e240a1b10 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 207959787 0 0) 0x7f7e10003520 con 0x7f7e240a2410 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 >> v1:172.21.15.171:6789/0 conn(0x7f7e2413d060 legacy=0x7f7e2413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.602 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 >> v1:172.21.15.148:6789/0 conn(0x7f7e240a1b10 legacy=0x7f7e2413b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 monclient: found mon.noname-b 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2406dfd0 con 0x7f7e240a2410 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e14001610 con 0x7f7e240a2410 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 5 monclient: authenticate success, global_id 154352 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:27:59.603 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: dump: 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.604 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e14001870 con 0x7f7e240a2410 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 1 -- 172.21.15.148:0/3009482878 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e14001c80 con 0x7f7e240a2410 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 10 monclient: get_monmap_and_config success 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 4 set_mon_vals no callback set 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e22ffd640 10 monclient: dump: 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.605 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 10 monclient: shutdown 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 1 -- 172.21.15.148:0/3009482878 >> v1:172.21.15.148:6790/0 conn(0x7f7e240a2410 legacy=0x7f7e240a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 1 -- 172.21.15.148:0/3009482878 shutdown_connections 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 1 -- 172.21.15.148:0/3009482878 >> 172.21.15.148:0/3009482878 conn(0x7f7e240a97b0 msgr2=0x7f7e240a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 1 -- 172.21.15.148:0/3009482878 shutdown_connections 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.590+0000 7f7e29bcf640 1 -- 172.21.15.148:0/3009482878 wait complete. 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.591+0000 7f7e29bcf640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.591+0000 7f7e29bcf640 10 monclient: monmap: 2024-09-16T04:27:59.606 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.591+0000 7f7e29bcf640 1 Processor -- start 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.591+0000 7f7e29bcf640 1 -- start start 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.591+0000 7f7e29bcf640 10 monclient: init 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e240a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e240a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2413d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2409f010 con 0x7f7e240a1b10 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e24137920 con 0x7f7e240a2410 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e24137c00 con 0x7f7e2413d060 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e29bcf640 10 monclient(hunting): authenticate will time out at 2100.218226s 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e237fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f7e240a1b10 0x7f7e241328c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40706/0 (socket says 172.21.15.148:40706) 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e237fe640 1 -- 172.21.15.148:0/296339733 learned_addr learned my addr 172.21.15.148:0/296339733 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890340744 0 0) 0x7f7e24137c00 con 0x7f7e2413d060 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): my global_id is 154355 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e04003520 con 0x7f7e2413d060 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365045522 0 0) 0x7f7e2406dfd0 con 0x7f7e240a1b10 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): my global_id is 174114 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e24137c00 con 0x7f7e240a1b10 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551470909 0 0) 0x7f7e24137920 con 0x7f7e240a2410 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): my global_id is 164233 2024-09-16T04:27:59.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e2406dfd0 con 0x7f7e240a2410 2024-09-16T04:27:59.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1914220611 0 0) 0x7f7e04003520 con 0x7f7e2413d060 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e24137920 con 0x7f7e2413d060 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e1c001b70 con 0x7f7e2413d060 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): dump: 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.611 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2838057688 0 0) 0x7f7e24137c00 con 0x7f7e240a1b10 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e04003520 con 0x7f7e240a1b10 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e14001bc0 con 0x7f7e240a1b10 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): dump: 2024-09-16T04:27:59.612 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1887558401 0 0) 0x7f7e2406dfd0 con 0x7f7e240a2410 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e24137c00 con 0x7f7e240a2410 2024-09-16T04:27:59.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e0c001310 con 0x7f7e240a2410 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient(hunting): dump: 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.614 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2930686926 0 0) 0x7f7e24137920 con 0x7f7e2413d060 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.171:6789/0 conn(0x7f7e240a2410 legacy=0x7f7e241d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.148:6789/0 conn(0x7f7e240a1b10 legacy=0x7f7e241328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 monclient: found mon.c 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2413cb90 con 0x7f7e2413d060 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.592+0000 7f7e20ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.615 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 5 monclient: authenticate success, global_id 154355 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 10 monclient: _renew_subs 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e2409eb50 con 0x7f7e2413d060 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 10 monclient: _renew_subs 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e2409f430 con 0x7f7e2413d060 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e1c003970 con 0x7f7e2413d060 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e1c003cc0 con 0x7f7e2413d060 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e20ff9640 10 monclient: dump: 2024-09-16T04:27:59.616 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e227fc640 4 set_mon_vals no callback set 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.593+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7e080065c0 con 0x7f7e08002860 2024-09-16T04:27:59.617 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f7e1c003cc0 con 0x7f7e2413d060 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f7e1c046810 con 0x7f7e2413d060 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1188776162 0 0) 0x7f7e080065c0 con 0x7f7e08002860 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient(hunting): my global_id is 154358 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e24137920 con 0x7f7e08002860 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e1affd640 10 monclient: get_auth_request con 0x7f7e04039100 auth_method 0 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2094789262 0 0) 0x7f7e24137920 con 0x7f7e08002860 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e080065c0 con 0x7f7e08002860 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e14004de0 con 0x7f7e08002860 2024-09-16T04:27:59.618 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 10 monclient: dump: 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.619 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 78088075 0 0) 0x7f7e080065c0 con 0x7f7e08002860 2024-09-16T04:27:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.594+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e0800a150 con 0x7f7e08002860 2024-09-16T04:27:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.595+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7e14007150 con 0x7f7e08002860 2024-09-16T04:27:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.595+0000 7f7e20ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.595+0000 7f7e20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:27:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.595+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.148:6790/0 conn(0x7f7e08002860 legacy=0x7f7e08004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.595+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.148:6790/0 conn(0x7f7e08002860 legacy=0x7f7e08004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e29bcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7e08009e40 con 0x7f7e08009100 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3612970943 0 0) 0x7f7e08009e40 con 0x7f7e08009100 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient(hunting): my global_id is 154361 2024-09-16T04:27:59.622 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e080065c0 con 0x7f7e08009100 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2292674578 0 0) 0x7f7e080065c0 con 0x7f7e08009100 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e08009e40 con 0x7f7e08009100 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7e140054a0 con 0x7f7e08009100 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient: dump: 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.623 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3271333416 0 0) 0x7f7e08009e40 con 0x7f7e08009100 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 --> v1:172.21.15.148:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e0800c3d0 con 0x7f7e08009100 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 <== mon.2 v1:172.21.15.148:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4144 (unknown 0 0 889039710) 0x7f7e14007300 con 0x7f7e08009100 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.148:6790/0 conn(0x7f7e08009100 legacy=0x7f7e08009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.601+0000 7f7e20ff9640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.148:6790/0 conn(0x7f7e08009100 legacy=0x7f7e08009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 82, 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 216509, 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:27:59.625 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.626 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.627 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:27:59.628 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.629 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.630 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.631 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.632 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:27:59.633 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.634 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.635 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:27:59.636 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 5 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:27:59.637 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:27:59.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.171:6812/7327273 conn(0x7f7e04039100 legacy=0x7f7e0403b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 10 monclient: shutdown 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 >> v1:172.21.15.148:6790/0 conn(0x7f7e2413d060 legacy=0x7f7e241d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e23fff640 1 -- 172.21.15.148:0/296339733 reap_dead start 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e23fff640 1 -- 172.21.15.148:0/296339733 reap_dead start 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 shutdown_connections 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 >> 172.21.15.148:0/296339733 conn(0x7f7e240a97b0 msgr2=0x7f7e240a37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 shutdown_connections 2024-09-16T04:27:59.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.602+0000 7f7e29bcf640 1 -- 172.21.15.148:0/296339733 wait complete. 2024-09-16T04:27:59.640 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:27:59.640 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:27:59.640 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:27:59.738 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.736+0000 7f641dd2a640 10 monclient: get_monmap_and_config 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.736+0000 7f641dd2a640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.736+0000 7f641dd2a640 10 monclient: monmap: 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:27:59.738048+0000 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:27:59.738048+0000 2024-09-16T04:27:59.739 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:27:59.740 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:27:59.740 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.736+0000 7f641dd2a640 1 Processor -- start 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.736+0000 7f641dd2a640 1 -- start start 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.736+0000 7f641dd2a640 10 monclient: init 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient(hunting): picked mon.noname-b con 0x7f64180a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient(hunting): picked mon.noname-a con 0x7f64180a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.750 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient(hunting): picked mon.noname-c con 0x7f641813d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64180a0e00 con 0x7f64180a1b10 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64180a10e0 con 0x7f641813d060 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64180a4970 con 0x7f64180a2410 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f641dd2a640 10 monclient(hunting): authenticate will time out at 2100.363573s 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64177fe640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f64180a2410 0x7f64180a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51864/0 (socket says 172.21.15.148:51864) 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64177fe640 1 -- 172.21.15.148:0/867775588 learned_addr learned my addr 172.21.15.148:0/867775588 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f6416ffd640 1 --1- 172.21.15.148:0/867775588 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 0x7f641813b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40710/0 (socket says 172.21.15.148:40710) 2024-09-16T04:27:59.751 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725078445 0 0) 0x7f64180a0e00 con 0x7f64180a1b10 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): my global_id is 174117 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6400003520 con 0x7f64180a1b10 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2951235503 0 0) 0x7f64180a4970 con 0x7f64180a2410 2024-09-16T04:27:59.752 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 10 monclient(hunting): my global_id is 154364 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.737+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64180a0e00 con 0x7f64180a2410 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1098595111 0 0) 0x7f6400003520 con 0x7f64180a1b10 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 >> v1:172.21.15.148:6790/0 conn(0x7f64180a2410 legacy=0x7f64180a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 >> v1:172.21.15.171:6789/0 conn(0x7f641813d060 legacy=0x7f641813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 monclient: found mon.noname-a 2024-09-16T04:27:59.753 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641806dfd0 con 0x7f64180a1b10 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f640c001610 con 0x7f64180a1b10 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 5 monclient: authenticate success, global_id 174117 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: dump: 2024-09-16T04:27:59.754 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.755 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f640c001870 con 0x7f64180a1b10 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 10 monclient: get_monmap_and_config success 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 1 -- 172.21.15.148:0/867775588 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f640c001c80 con 0x7f64180a1b10 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 4 set_mon_vals no callback set 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f64167fc640 10 monclient: dump: 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.756 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 10 monclient: shutdown 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 1 -- 172.21.15.148:0/867775588 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 legacy=0x7f641813b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 1 -- 172.21.15.148:0/867775588 shutdown_connections 2024-09-16T04:27:59.757 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 1 -- 172.21.15.148:0/867775588 >> 172.21.15.148:0/867775588 conn(0x7f64180a97b0 msgr2=0x7f64180a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 1 -- 172.21.15.148:0/867775588 shutdown_connections 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.738+0000 7f641dd2a640 1 -- 172.21.15.148:0/867775588 wait complete. 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.739+0000 7f641dd2a640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.739+0000 7f641dd2a640 10 monclient: monmap: 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:27:59.758 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.739+0000 7f641dd2a640 1 Processor -- start 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.739+0000 7f641dd2a640 1 -- start start 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.739+0000 7f641dd2a640 10 monclient: init 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient(hunting): picked mon.noname-a con 0x7f64180a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient(hunting): picked mon.noname-b con 0x7f64180a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient(hunting): picked mon.noname-c con 0x7f641813d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64181cbb80 con 0x7f64180a1b10 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f641809ea80 con 0x7f64180a2410 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6418137b00 con 0x7f641813d060 2024-09-16T04:27:59.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f641dd2a640 10 monclient(hunting): authenticate will time out at 2100.366345s 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f64177fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 0x7f641813c060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40712/0 (socket says 172.21.15.148:40712) 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f64177fe640 1 -- 172.21.15.148:0/2641603488 learned_addr learned my addr 172.21.15.148:0/2641603488 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f64177fe640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 legacy=0x7f641813c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f64177fe640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 legacy=0x7f641813c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f64177fe640 1 --1- 172.21.15.148:0/2641603488 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 0x7f641813c060 :-1 s=OPENED pgs=735 cs=1 l=1).handle_message read tag failed 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f64177fe640 1 --1- 172.21.15.148:0/2641603488 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 0x7f641813c060 :-1 s=OPENED pgs=735 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4174854674 0 0) 0x7f6418137b00 con 0x7f641813d060 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): my global_id is 154367 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6404003550 con 0x7f641813d060 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1063487248 0 0) 0x7f641809ea80 con 0x7f64180a2410 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): my global_id is 164239 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6418137b00 con 0x7f64180a2410 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2602065767 0 0) 0x7f6404003550 con 0x7f641813d060 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f641809ea80 con 0x7f641813d060 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6410001850 con 0x7f641813d060 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:27:59.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): dump: 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:27:59.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.740+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 704350751 0 0) 0x7f641809ea80 con 0x7f641813d060 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.171:6789/0 conn(0x7f64180a2410 legacy=0x7f64181d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.148:6789/0 conn(0x7f64180a1b10 legacy=0x7f641813c060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 monclient: found mon.c 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64181403b0 con 0x7f641813d060 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 5 monclient: authenticate success, global_id 154367 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 10 monclient: _renew_subs 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f641813cd80 con 0x7f641813d060 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 10 monclient: _renew_subs 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6418137820 con 0x7f641813d060 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6410001c60 con 0x7f641813d060 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6410003a60 con 0x7f641813d060 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f6415ffb640 4 set_mon_vals no callback set 2024-09-16T04:27:59.764 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 10 monclient: dump: 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.765 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f63f80065c0 con 0x7f63f8002860 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.741+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f641000ee90 con 0x7f641813d060 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f6410046650 con 0x7f641813d060 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f6416ffd640 10 monclient: get_auth_request con 0x7f6404038bf0 auth_method 0 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404240809 0 0) 0x7f63f80065c0 con 0x7f63f8002860 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient(hunting): my global_id is 164242 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f641809ea80 con 0x7f63f8002860 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4070313271 0 0) 0x7f641809ea80 con 0x7f63f8002860 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63f80065c0 con 0x7f63f8002860 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6408001db0 con 0x7f63f8002860 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.766 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 10 monclient: dump: 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2220593338 0 0) 0x7f63f80065c0 con 0x7f63f8002860 2024-09-16T04:27:59.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.742+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63f800a150 con 0x7f63f8002860 2024-09-16T04:27:59.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.743+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6408001c10 con 0x7f63f8002860 2024-09-16T04:27:59.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.743+0000 7f63fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.743+0000 7f63fffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:27:59.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.743+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.171:6789/0 conn(0x7f63f8002860 legacy=0x7f63f8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.743+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.171:6789/0 conn(0x7f63f8002860 legacy=0x7f63f8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f641dd2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:27:59.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f63f8009e40 con 0x7f63f8009100 2024-09-16T04:27:59.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214087145 0 0) 0x7f63f8009e40 con 0x7f63f8009100 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 10 monclient(hunting): my global_id is 164245 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63f80065c0 con 0x7f63f8009100 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.749+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3673991575 0 0) 0x7f63f80065c0 con 0x7f63f8009100 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63f8009e40 con 0x7f63f8009100 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6408004ef0 con 0x7f63f8009100 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 10 monclient: dump: 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.770 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3740859584 0 0) 0x7f63f8009e40 con 0x7f63f8009100 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 --> v1:172.21.15.171:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f63f800c3d0 con 0x7f63f8009100 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 <== mon.1 v1:172.21.15.171:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 1155784895) 0x7f6408006360 con 0x7f63f8009100 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:27:59.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.171:6789/0 conn(0x7f63f8009100 legacy=0x7f63f8009590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f63fffff640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.171:6789/0 conn(0x7f63f8009100 legacy=0x7f63f8009590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout: "state": "peon", 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout: "election_epoch": 82, 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout: "uptime": 108758, 2024-09-16T04:27:59.772 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum": [ 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: 0, 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: 1, 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: 2 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_age": 0, 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "required_con": "2450521708007866372", 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "required_mon": [ 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.773 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "quorum_mon": [ 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.774 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "outside_quorum": [], 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "extra_probe_peers": [], 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "sync_provider": [], 2024-09-16T04:27:59.775 INFO:teuthology.orchestra.run.smithi148.stdout: "monmap": { 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "epoch": 1, 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "fsid": "bc50e21b-589f-47d0-85d5-a865de7d0d68", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "modified": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "created": "2024-09-16T04:05:49.094408Z", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release": 19, 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "min_mon_release_name": "squid", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "election_strategy": 1, 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "disallowed_leaders: ": "", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false, 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "tiebreaker_mon": "", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "removed_ranks: ": "", 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "features": { 2024-09-16T04:27:59.776 INFO:teuthology.orchestra.run.smithi148.stdout: "persistent": [ 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "kraken", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "luminous", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "mimic", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap-prune", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "nautilus", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "octopus", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "pacific", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "elector-pinging", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "quincy", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "reef", 2024-09-16T04:27:59.777 INFO:teuthology.orchestra.run.smithi148.stdout: "squid" 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: "optional": [] 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: "mons": [ 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 0, 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "a", 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.778 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789", 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6789/0", 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6789/0", 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.779 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 1, 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "b", 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789", 2024-09-16T04:27:59.780 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.171:6789/0", 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.171:6789/0", 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.781 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "rank": 2, 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "name": "c", 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addrs": { 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "addrvec": [ 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "type": "v1", 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790", 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: "nonce": 0 2024-09-16T04:27:59.782 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: "addr": "172.21.15.148:6790/0", 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: "public_addr": "172.21.15.148:6790/0", 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: "priority": 0, 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: "weight": 0, 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: "crush_location": "{}" 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.783 INFO:teuthology.orchestra.run.smithi148.stdout: "feature_map": { 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: "mon": [ 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: "osd": [ 2024-09-16T04:27:59.784 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "client": [ 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 2 2024-09-16T04:27:59.785 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: ], 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: "mgr": [ 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: { 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: "release": "squid", 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: "num": 1 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout: "stretch_mode": false 2024-09-16T04:27:59.786 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.171:6812/7327273 conn(0x7f6404038bf0 legacy=0x7f640403b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f641dd2a640 10 monclient: shutdown 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f641dd2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 >> v1:172.21.15.148:6790/0 conn(0x7f641813d060 legacy=0x7f64181d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f6417fff640 1 -- 172.21.15.148:0/2641603488 reap_dead start 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f6417fff640 1 -- 172.21.15.148:0/2641603488 reap_dead start 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 shutdown_connections 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.750+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 >> 172.21.15.148:0/2641603488 conn(0x7f64180a97b0 msgr2=0x7f64180a7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.751+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 shutdown_connections 2024-09-16T04:27:59.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.751+0000 7f641dd2a640 1 -- 172.21.15.148:0/2641603488 wait complete. 2024-09-16T04:27:59.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:27:59.789 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:27:59.789 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:27:59.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.884+0000 7f9dc07d8640 10 monclient: get_monmap_and_config 2024-09-16T04:27:59.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.884+0000 7f9dc07d8640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.884+0000 7f9dc07d8640 10 monclient: monmap: 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:27:59.886801+0000 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:27:59.886801+0000 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.888 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:27:59.889 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:27:59.889 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.885+0000 7f9dc07d8640 1 Processor -- start 2024-09-16T04:27:59.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.885+0000 7f9dc07d8640 1 -- start start 2024-09-16T04:27:59.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.885+0000 7f9dc07d8640 10 monclient: init 2024-09-16T04:27:59.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient(hunting): picked mon.noname-a con 0x7f9db8134b40 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient(hunting): picked mon.noname-b con 0x7f9db8134610 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient(hunting): picked mon.noname-c con 0x7f9db813b050 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db80a1910 con 0x7f9db8134b40 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db8099690 con 0x7f9db813b050 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db8099970 con 0x7f9db8134610 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient(hunting): authenticate will time out at 2100.512194s 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbe574640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9db8134b40 0x7f9db8134f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40726/0 (socket says 172.21.15.148:40726) 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbe574640 1 -- 172.21.15.148:0/1444621369 learned_addr learned my addr 172.21.15.148:0/1444621369 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925240077 0 0) 0x7f9db80a1910 con 0x7f9db8134b40 2024-09-16T04:27:59.899 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): my global_id is 174120 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9da4003520 con 0x7f9db8134b40 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326321151 0 0) 0x7f9db8099970 con 0x7f9db8134610 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): my global_id is 154370 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9db80a1910 con 0x7f9db8134610 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856434871 0 0) 0x7f9db8099690 con 0x7f9db813b050 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient(hunting): my global_id is 164248 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9db8099970 con 0x7f9db813b050 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 773330236 0 0) 0x7f9db80a1910 con 0x7f9db8134610 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 >> v1:172.21.15.171:6789/0 conn(0x7f9db813b050 legacy=0x7f9db813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 >> v1:172.21.15.148:6789/0 conn(0x7f9db8134b40 legacy=0x7f9db8134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 monclient: found mon.noname-b 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db809f160 con 0x7f9db8134610 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9da8001570 con 0x7f9db8134610 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 5 monclient: authenticate success, global_id 154370 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: dump: 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.902 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9da80018d0 con 0x7f9db8134610 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 1 -- 172.21.15.148:0/1444621369 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9da8001ce0 con 0x7f9db8134610 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dc07d8640 4 set_mon_vals no callback set 2024-09-16T04:27:59.903 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.886+0000 7f9dbd572640 10 monclient: dump: 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.904 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 10 monclient: shutdown 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 1 -- 172.21.15.148:0/1444621369 >> v1:172.21.15.148:6790/0 conn(0x7f9db8134610 legacy=0x7f9db8139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 1 -- 172.21.15.148:0/1444621369 shutdown_connections 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 1 -- 172.21.15.148:0/1444621369 >> 172.21.15.148:0/1444621369 conn(0x7f9db80a5860 msgr2=0x7f9db80a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 1 -- 172.21.15.148:0/1444621369 shutdown_connections 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.887+0000 7f9dc07d8640 1 -- 172.21.15.148:0/1444621369 wait complete. 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient: build_initial_monmap 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient: monmap: 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:27:59.905 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 1 Processor -- start 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 1 -- start start 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient: init 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:27:59.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient(hunting): picked mon.noname-b con 0x7f9db8134610 addr v1:172.21.15.171:6789/0 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient(hunting): picked mon.noname-c con 0x7f9db8134b40 addr v1:172.21.15.148:6790/0 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient(hunting): picked mon.noname-a con 0x7f9db813b050 addr v1:172.21.15.148:6789/0 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db80723e0 con 0x7f9db813b050 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db806fe60 con 0x7f9db8134610 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9db809c630 con 0x7f9db8134b40 2024-09-16T04:27:59.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dc07d8640 10 monclient(hunting): authenticate will time out at 2100.514945s 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dbdd73640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f9db8134b40 0x7f9db806f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51902/0 (socket says 172.21.15.148:51902) 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.888+0000 7f9dbdd73640 1 -- 172.21.15.148:0/3616014001 learned_addr learned my addr 172.21.15.148:0/3616014001 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4245498267 0 0) 0x7f9db809c630 con 0x7f9db8134b40 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): my global_id is 154373 2024-09-16T04:27:59.908 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d9c003550 con 0x7f9db8134b40 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134759031 0 0) 0x7f9db80723e0 con 0x7f9db813b050 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): my global_id is 174123 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9db809c630 con 0x7f9db813b050 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1542682513 0 0) 0x7f9db806fe60 con 0x7f9db8134610 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): my global_id is 164251 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9db80723e0 con 0x7f9db8134610 2024-09-16T04:27:59.909 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3198396973 0 0) 0x7f9db809c630 con 0x7f9db813b050 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9db806fe60 con 0x7f9db813b050 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9db0001c30 con 0x7f9db813b050 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): dump: 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.910 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3911262183 0 0) 0x7f9d9c003550 con 0x7f9db8134b40 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9db809c630 con 0x7f9db8134b40 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9da8001530 con 0x7f9db8134b40 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): dump: 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.911 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 842869398 0 0) 0x7f9db80723e0 con 0x7f9db8134610 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d9c003550 con 0x7f9db8134610 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9db4001190 con 0x7f9db8134610 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient(hunting): dump: 2024-09-16T04:27:59.912 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1368128377 0 0) 0x7f9db806fe60 con 0x7f9db813b050 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 >> v1:172.21.15.148:6790/0 conn(0x7f9db8134b40 legacy=0x7f9db806f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 >> v1:172.21.15.171:6789/0 conn(0x7f9db8134610 legacy=0x7f9db806f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:27:59.913 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 monclient: found mon.a 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db8070c50 con 0x7f9db813b050 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: _finish_auth 0 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 5 monclient: authenticate success, global_id 174123 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 10 monclient: _renew_subs 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9db8072100 con 0x7f9db813b050 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 10 monclient: _renew_subs 2024-09-16T04:27:59.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9db8139f60 con 0x7f9db813b050 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9db0003990 con 0x7f9db813b050 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9db0003ce0 con 0x7f9db813b050 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9dbcd71640 4 set_mon_vals no callback set 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.889+0000 7f9daeffd640 10 monclient: dump: 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:27:59.915 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9db000f190 con 0x7f9db813b050 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9dc07d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9dc07d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9dc07d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da0005080 con 0x7f9db813b050 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f9db00468e0 con 0x7f9db813b050 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.890+0000 7f9dbe574640 10 monclient: get_auth_request con 0x7f9d9c0391a0 auth_method 0 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.895+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9db00145d0 con 0x7f9db813b050 2024-09-16T04:27:59.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.895+0000 7f9daeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:27:59.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:27:59.895+0000 7f9daeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:00.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9dc07d8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:28:00.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9dc07d8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:28:00.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9dc07d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:00.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9da0005b80 con 0x7f9db813b050 2024-09-16T04:28:00.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9daeffd640 1 -- 172.21.15.148:0/3616014001 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f9db0018870 con 0x7f9db813b050 2024-09-16T04:28:00.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9daeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:28:00.128 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.125+0000 7f9daeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:28:00.129 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.126+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.129 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.126+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3029974473,config=692619927,health=528802886,kv=2270513034,logm=1005265975}) 2024-09-16T04:28:00.129 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.126+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=4285682009,config=692619927,health=995757702,kv=2270513034,logm=2586383354}) 2024-09-16T04:28:00.129 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.126+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.129 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.126+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3029974473,config=692619927,health=528802886,kv=2270513034,logm=1005265975}) 2024-09-16T04:28:00.129 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.126+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=1,health=10,kv=9,logm=53} crc {auth=4255498478,config=3678701417,health=3842293808,kv=82740207,logm=122447975}) 2024-09-16T04:28:00.130 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.127+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.130 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.127+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521936062}) 2024-09-16T04:28:00.130 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.127+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742716254}) 2024-09-16T04:28:00.130 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.127+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.130 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.127+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521936062}) 2024-09-16T04:28:00.130 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.127+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115092978}) 2024-09-16T04:28:00.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.128+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.128+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40444205}) 2024-09-16T04:28:00.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.128+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=80738416}) 2024-09-16T04:28:00.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.128+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.128+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40444205}) 2024-09-16T04:28:00.131 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.128+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1724721486}) 2024-09-16T04:28:00.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 >> v1:172.21.15.171:6812/7327273 conn(0x7f9d9c0391a0 legacy=0x7f9d9c03b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:00.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 10 monclient: shutdown 2024-09-16T04:28:00.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:00.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 >> v1:172.21.15.148:6789/0 conn(0x7f9db813b050 legacy=0x7f9db81d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:00.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 shutdown_connections 2024-09-16T04:28:00.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 >> 172.21.15.148:0/3616014001 conn(0x7f9db80a5860 msgr2=0x7f9db813d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:00.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 shutdown_connections 2024-09-16T04:28:00.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:00.128+0000 7f9dc07d8640 1 -- 172.21.15.148:0/3616014001 wait complete. 2024-09-16T04:28:00.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.129+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.129+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742532336}) 2024-09-16T04:28:00.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.129+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3069731899}) 2024-09-16T04:28:00.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.129+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.135 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.129+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742532336}) 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.129+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953505713}) 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.130+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.130+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398231619}) 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.130+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2399921586}) 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.130+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.130+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398231619}) 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.130+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037604017}) 2024-09-16T04:28:00.136 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447454693}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=601167962}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447454693}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426009795}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310656140}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1289344269}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310656140}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.131+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1066935175}) 2024-09-16T04:28:00.137 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.132+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.132+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142695162}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.132+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2882126479}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.132+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.132+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142695162}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.132+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258700649}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942565519}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1043393221}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942565519}) 2024-09-16T04:28:00.138 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4144263043}) 2024-09-16T04:28:00.142 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.142 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458497470}) 2024-09-16T04:28:00.142 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3423634209}) 2024-09-16T04:28:00.142 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.142 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458497470}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.133+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=160931433}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.134+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.134+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476121342}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.134+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659464503}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.134+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.134+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476121342}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.134+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=747776120}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862213010}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567024112}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862213010}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440429224}) 2024-09-16T04:28:00.143 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033687546}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379354453}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033687546}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.135+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480311828}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.136+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.136+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051680435}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.136+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1202433894}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.136+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.136+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051680435}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.136+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836514084}) 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.144 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519182881}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2384899235}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519182881}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960856241}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541874052}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1693506391}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541874052}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.137+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1272986592}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.138+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.138+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866619565}) 2024-09-16T04:28:00.145 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.138+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=137588576}) 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.138+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.138+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866619565}) 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.138+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104757108}) 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122836467}) 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14133219}) 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.146 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=122836467}) 2024-09-16T04:28:00.147 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=237592185}) 2024-09-16T04:28:00.148 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690770490}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.139+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57050476}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690770490}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657422322}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170122899}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=365121734}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170122899}) 2024-09-16T04:28:00.163 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.140+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380529529}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154836922}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835417384}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154836922}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220848497}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178450758}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238843926}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.141+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178450758}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=891636476}) 2024-09-16T04:28:00.164 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751422091}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3709972212}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751422091}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.142+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517626352}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402939866}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1264779041}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402939866}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953093754}) 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.165 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791668952}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.143+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443220687}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791668952}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2310021268}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014387986}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129969656}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014387986}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.144+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938278324}) 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.166 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976528281}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1231968799}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976528281}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609533642}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.145+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797507801}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797154270}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797507801}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4253959271}) 2024-09-16T04:28:00.167 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010424382}) 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320064492}) 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010424382}) 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.146+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900651528}) 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.147+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.147+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889066458}) 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.147+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1314117798}) 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.147+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.168 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.147+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889066458}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.147+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421054603}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.148+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.148+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444606394}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.148+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288327892}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.148+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.148+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444606394}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.148+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532042304}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.149+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.149+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398393618}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.149+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167827824}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.149+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.149+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398393618}) 2024-09-16T04:28:00.169 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.149+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909399722}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1284272700}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24977514}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1284272700}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2391594618}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042900070}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1926722570}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042900070}) 2024-09-16T04:28:00.170 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.150+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452175532}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.151+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.151+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849513852}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.151+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1911226181}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.151+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.151+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849513852}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.151+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570387326}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.152+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.152+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568304069}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.152+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2751409246}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.152+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.152+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568304069}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.152+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1015939902}) 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.171 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023055660}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550927761}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023055660}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772768934}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161735599}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2357230193}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.153+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161735599}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944345965}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710201676}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1936783252}) 2024-09-16T04:28:00.172 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710201676}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.154+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749895604}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.155+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.155+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155504794}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.155+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4227176586}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.155+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.155+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155504794}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.155+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737233009}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.156+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.156+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749073505}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.156+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4035773594}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.156+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.156+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749073505}) 2024-09-16T04:28:00.173 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.156+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247030928}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694047244}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160695418}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694047244}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561413692}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884615060}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1843019167}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884615060}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.157+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597778499}) 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.159+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.174 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.159+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069705353}) 2024-09-16T04:28:00.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.159+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=296412836}) 2024-09-16T04:28:00.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.159+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.159+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069705353}) 2024-09-16T04:28:00.175 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.159+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2561701841}) 2024-09-16T04:28:00.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.161+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.161+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346820055}) 2024-09-16T04:28:00.177 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.161+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2893488778}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.161+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.161+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346820055}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.162+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465783728}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.162+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.162+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=322599807}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.162+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754046932}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.163+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.163+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=322599807}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.163+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606122503}) 2024-09-16T04:28:00.178 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.169+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.169+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743320546}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.169+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481596248}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.169+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.169+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2743320546}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.169+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119031114}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.172+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.172+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019356850}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.172+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387757273}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.172+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.172+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019356850}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.172+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=30007701}) 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.173+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.179 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.173+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893102305}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.173+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093672642}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.173+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.173+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893102305}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.173+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243026069}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.174+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.174+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546919951}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.174+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2287136828}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.174+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.174+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546919951}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.174+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2197582423}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.175+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.175+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849503725}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.175+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4203059890}) 2024-09-16T04:28:00.180 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.175+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.175+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849503725}) 2024-09-16T04:28:00.181 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.175+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281217078}) 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613221063}) 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2366443732}) 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613221063}) 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332226029}) 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.183 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844476724}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=664367384}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.176+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844476724}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2356919302}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400518270}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115821168}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400518270}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.177+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2835775884}) 2024-09-16T04:28:00.184 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.178+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.178+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008564080}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.178+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=69560650}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.178+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.178+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008564080}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.178+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016703861}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.179+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.179+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908431357}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.179+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471177061}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.179+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.179+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908431357}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.179+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907305948}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.180+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.180+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175853163}) 2024-09-16T04:28:00.185 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.180+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2861756930}) 2024-09-16T04:28:00.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.180+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.180+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175853163}) 2024-09-16T04:28:00.186 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.180+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973687040}) 2024-09-16T04:28:00.192 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.181+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.192 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.181+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550801234}) 2024-09-16T04:28:00.192 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.181+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64126721}) 2024-09-16T04:28:00.192 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.181+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.181+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550801234}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.181+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2231670435}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.182+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.182+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147157378}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.182+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3953939302}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.182+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.182+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147157378}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.182+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078499272}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.183+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.183+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895925424}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.183+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97172261}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.183+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.183+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895925424}) 2024-09-16T04:28:00.193 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.183+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491622808}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.184+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.184+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354906015}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.184+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436334919}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.184+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.184+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354906015}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.184+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3042250286}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.185+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.185+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876670792}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.185+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938025620}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.185+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.185+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876670792}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.185+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=369497382}) 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.186+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.194 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.186+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957126808}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.186+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1989567638}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.186+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.186+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957126808}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.186+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=117249698}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.187+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.187+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169965250}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.187+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402433588}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.187+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.187+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169965250}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.187+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492854592}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432458208}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319689723}) 2024-09-16T04:28:00.195 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432458208}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2749653698}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.188+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294822481}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984938756}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294822481}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=540417159}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2470943946}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4036024347}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2470943946}) 2024-09-16T04:28:00.196 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.189+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722386600}) 2024-09-16T04:28:00.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.190+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.190+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834524463}) 2024-09-16T04:28:00.200 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.190+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697631264}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.190+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.190+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3834524463}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.190+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4058592934}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.191+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.191+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696351163}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.191+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943149705}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.191+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.191+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696351163}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.191+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472170441}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.192+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.192+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1540059753}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.192+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754765380}) 2024-09-16T04:28:00.201 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.192+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.192+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1540059753}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.192+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314260843}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.193+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.193+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=934581952}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.193+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809756544}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.193+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.193+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=934581952}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.193+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656107182}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.195+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.195+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=9} crc {logm=519495419,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3566317500,mon_config_key=888502937,monmap=2633035914,osd_metadata=3209840783,osd_pg_creating=1354844604,osd_snap=3823785925,osdmap=3988579883}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.195+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=9} crc {logm=3541632226,mdsmap=3850802020,mgr=397011006,mgr_command_descs=3961477881,mgrstat=744490753,mon_config_key=2589496441,monmap=2633035914,osd_metadata=1015068370,osd_pg_creating=1354844604,osd_snap=3823785925,osdmap=1533020961}) 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.195+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.202 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.195+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=9} crc {logm=519495419,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=3566317500,mon_config_key=888502937,monmap=2633035914,osd_metadata=3209840783,osd_pg_creating=1354844604,osd_snap=3823785925,osdmap=3988579883}) 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.195+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=6} crc {logm=3507307554,mdsmap=1097804592,mgr=1586861337,mgr_command_descs=3961477881,mgr_metadata=582288965,mgrstat=744490753,mon_config_key=193808867,monmap=2633035914,osd_metadata=4007021953,osd_pg_creating=1354844604,osd_snap=3823785925,osdmap=3203639570}) 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.196+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.196+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2988073267}) 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.196+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=1298219752}) 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.196+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.196+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2988073267}) 2024-09-16T04:28:00.203 INFO:tasks.ceph.mon.a.smithi148.stderr:2024-09-16T04:28:00.196+0000 7fa67f0a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=41} crc {osdmap=4196733653}) 2024-09-16T04:28:01.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:28:01.149 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient: get_monmap_and_config 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient: build_initial_monmap 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient: monmap: 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:01.259546+0000 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:01.259546+0000 2024-09-16T04:28:01.260 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:01.261 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:01.261 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 1 Processor -- start 2024-09-16T04:28:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 1 -- start start 2024-09-16T04:28:01.268 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient: init 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient(hunting): picked mon.noname-a con 0x7f58fc09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient(hunting): picked mon.noname-b con 0x7f58fc09c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient(hunting): picked mon.noname-c con 0x7f58fc09cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc0a1010 con 0x7f58fc09bb70 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc09f030 con 0x7f58fc09cc30 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc09dc60 con 0x7f58fc09c660 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.258+0000 7f5900c5e640 10 monclient(hunting): authenticate will time out at 2101.884964s 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58fa59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f58fc09bb70 0x7f58fc09bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40730/0 (socket says 172.21.15.148:40730) 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58fa59c640 1 -- 172.21.15.148:0/4178163494 learned_addr learned my addr 172.21.15.148:0/4178163494 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:01.269 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111208347 0 0) 0x7f58fc0a1010 con 0x7f58fc09bb70 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): my global_id is 174126 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58e4003520 con 0x7f58fc09bb70 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166684077 0 0) 0x7f58fc09dc60 con 0x7f58fc09c660 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient(hunting): my global_id is 154376 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58fc0a1010 con 0x7f58fc09c660 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 967985478 0 0) 0x7f58fc0a1010 con 0x7f58fc09c660 2024-09-16T04:28:01.270 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 >> v1:172.21.15.171:6789/0 conn(0x7f58fc09cc30 legacy=0x7f58fc13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 >> v1:172.21.15.148:6789/0 conn(0x7f58fc09bb70 legacy=0x7f58fc09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: _finish_hunting 0 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 monclient: found mon.noname-b 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc09df40 con 0x7f58fc09c660 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: _finish_auth 0 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: _check_auth_tickets 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f58f0001610 con 0x7f58fc09c660 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f5900c5e640 5 monclient: authenticate success, global_id 154376 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f5900c5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.271 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: dump: 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f5900c5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:01.272 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58f0001800 con 0x7f58fc09c660 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 1 -- 172.21.15.148:0/4178163494 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f58f0001b90 con 0x7f58fc09c660 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f58f959a640 10 monclient: dump: 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.273 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f5900c5e640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.259+0000 7f5900c5e640 4 set_mon_vals no callback set 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 10 monclient: shutdown 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 1 -- 172.21.15.148:0/4178163494 >> v1:172.21.15.148:6790/0 conn(0x7f58fc09c660 legacy=0x7f58fc138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 1 -- 172.21.15.148:0/4178163494 shutdown_connections 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 1 -- 172.21.15.148:0/4178163494 >> 172.21.15.148:0/4178163494 conn(0x7f58fc0a5640 msgr2=0x7f58fc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 1 -- 172.21.15.148:0/4178163494 shutdown_connections 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.260+0000 7f5900c5e640 1 -- 172.21.15.148:0/4178163494 wait complete. 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient: build_initial_monmap 2024-09-16T04:28:01.274 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient: monmap: 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 1 Processor -- start 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 1 -- start start 2024-09-16T04:28:01.275 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient: init 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient(hunting): picked mon.noname-a con 0x7f58fc09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.261+0000 7f5900c5e640 10 monclient(hunting): picked mon.noname-b con 0x7f58fc09c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient(hunting): picked mon.noname-c con 0x7f58fc09cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc139b70 con 0x7f58fc09bb70 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc13bae0 con 0x7f58fc09c660 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc13b520 con 0x7f58fc09cc30 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient(hunting): authenticate will time out at 2101.888068s 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58fa59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f58fc09bb70 0x7f58fc138d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40746/0 (socket says 172.21.15.148:40746) 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58fa59c640 1 -- 172.21.15.148:0/3818566353 learned_addr learned my addr 172.21.15.148:0/3818566353 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:01.276 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1238808697 0 0) 0x7f58fc139b70 con 0x7f58fc09bb70 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): my global_id is 174129 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58dc003520 con 0x7f58fc09bb70 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330847352 0 0) 0x7f58fc13b520 con 0x7f58fc09cc30 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): my global_id is 154379 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58fc139b70 con 0x7f58fc09cc30 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511480363 0 0) 0x7f58fc13bae0 con 0x7f58fc09c660 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): my global_id is 164257 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58fc13b520 con 0x7f58fc09c660 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 612986238 0 0) 0x7f58dc003520 con 0x7f58fc09bb70 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58fc13bae0 con 0x7f58fc09bb70 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f58ec001960 con 0x7f58fc09bb70 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): dump: 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.278 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 204058958 0 0) 0x7f58fc139b70 con 0x7f58fc09cc30 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58dc003520 con 0x7f58fc09cc30 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f58f4001af0 con 0x7f58fc09cc30 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:01.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient(hunting): dump: 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 217144890 0 0) 0x7f58fc13bae0 con 0x7f58fc09bb70 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 >> v1:172.21.15.148:6790/0 conn(0x7f58fc09cc30 legacy=0x7f58fc1d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.280 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 >> v1:172.21.15.171:6789/0 conn(0x7f58fc09c660 legacy=0x7f58fc139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 monclient: found mon.a 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc13b800 con 0x7f58fc09bb70 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient: _finish_auth 0 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 5 monclient: authenticate success, global_id 174129 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient: _renew_subs 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc13a570 con 0x7f58fc09bb70 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient: _renew_subs 2024-09-16T04:28:01.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f58fc09acd0 con 0x7f58fc09bb70 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.262+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58ec0047e0 con 0x7f58fc09bb70 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f58ec004b30 con 0x7f58fc09bb70 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58f8d99640 4 set_mon_vals no callback set 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58eaffd640 10 monclient: dump: 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.282 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f58ec00fdc0 con 0x7f58fc09bb70 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f5900c5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f5900c5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f5900c5e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.263+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e0005080 con 0x7f58fc09bb70 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.264+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f58ec0478b0 con 0x7f58fc09bb70 2024-09-16T04:28:01.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.264+0000 7f58f9d9b640 10 monclient: get_auth_request con 0x7f58dc038e50 auth_method 0 2024-09-16T04:28:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.269+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f58ec019700 con 0x7f58fc09bb70 2024-09-16T04:28:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.269+0000 7f58eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.284 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.269+0000 7f58eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:01.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f5900c5e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:28:01.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f5900c5e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:28:01.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f5900c5e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f58e0005370 con 0x7f58fc09bb70 2024-09-16T04:28:01.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f58eaffd640 1 -- 172.21.15.148:0/3818566353 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1412 (unknown 254746657 0 4290182242) 0x7f58ec01dfc0 con 0x7f58fc09bb70 2024-09-16T04:28:01.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f58eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:28:01.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.497+0000 7f58eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:28:01.501 INFO:teuthology.orchestra.run.smithi148.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","modified":"2024-09-16T04:05:49.094408Z","created":"2024-09-16T04:05:49.094408Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6790","nonce":0}]},"addr":"172.21.15.148:6790/0","public_addr":"172.21.15.148:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:28:01.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 >> v1:172.21.15.171:6812/7327273 conn(0x7f58dc038e50 legacy=0x7f58dc03b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 10 monclient: shutdown 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 >> v1:172.21.15.148:6789/0 conn(0x7f58fc09bb70 legacy=0x7f58fc138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 shutdown_connections 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 >> 172.21.15.148:0/3818566353 conn(0x7f58fc0a5640 msgr2=0x7f58fc0a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 shutdown_connections 2024-09-16T04:28:01.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.500+0000 7f5900c5e640 1 -- 172.21.15.148:0/3818566353 wait complete. 2024-09-16T04:28:01.513 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:28:01.513 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:28:01.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:28:01.524 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:28:01.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.618+0000 7f79e3dc6640 10 monclient: get_monmap_and_config 2024-09-16T04:28:01.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.618+0000 7f79e3dc6640 10 monclient: build_initial_monmap 2024-09-16T04:28:01.620 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.618+0000 7f79e3dc6640 10 monclient: monmap: 2024-09-16T04:28:01.620 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:01.619836+0000 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:01.619836+0000 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:01.621 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.618+0000 7f79e3dc6640 1 Processor -- start 2024-09-16T04:28:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.618+0000 7f79e3dc6640 1 -- start start 2024-09-16T04:28:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.618+0000 7f79e3dc6640 10 monclient: init 2024-09-16T04:28:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:28:01.630 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f79dc09bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f79dc09c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f79dc09cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc0a1010 con 0x7f79dc09c660 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc09f030 con 0x7f79dc09bb70 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc09dc60 con 0x7f79dc09cc30 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient(hunting): authenticate will time out at 2102.245235s 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e2363640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f79dc09cc30 0x7f79dc13c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51934/0 (socket says 172.21.15.148:51934) 2024-09-16T04:28:01.631 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e1361640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f79dc09c660 0x7f79dc138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40756/0 (socket says 172.21.15.148:40756) 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e2363640 1 -- 172.21.15.148:0/2136506154 learned_addr learned my addr 172.21.15.148:0/2136506154 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842232005 0 0) 0x7f79dc09dc60 con 0x7f79dc09cc30 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): my global_id is 154382 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79c8003520 con 0x7f79dc09cc30 2024-09-16T04:28:01.632 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177284245 0 0) 0x7f79dc0a1010 con 0x7f79dc09c660 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient(hunting): my global_id is 174132 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79dc09dc60 con 0x7f79dc09c660 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1552048107 0 0) 0x7f79c8003520 con 0x7f79dc09cc30 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 >> v1:172.21.15.171:6789/0 conn(0x7f79dc09bb70 legacy=0x7f79dc09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 >> v1:172.21.15.148:6789/0 conn(0x7f79dc09c660 legacy=0x7f79dc138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: _finish_hunting 0 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 monclient: found mon.noname-b 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:01.633 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc09df40 con 0x7f79dc09cc30 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: _finish_auth 0 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: _check_auth_tickets 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f79d4001610 con 0x7f79dc09cc30 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 5 monclient: authenticate success, global_id 154382 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.634 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: dump: 2024-09-16T04:28:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.635 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79d4001880 con 0x7f79dc09cc30 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 1 -- 172.21.15.148:0/2136506154 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f79d4001c90 con 0x7f79dc09cc30 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e3dc6640 4 set_mon_vals no callback set 2024-09-16T04:28:01.636 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.619+0000 7f79e0b60640 10 monclient: dump: 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 10 monclient: shutdown 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 1 -- 172.21.15.148:0/2136506154 >> v1:172.21.15.148:6790/0 conn(0x7f79dc09cc30 legacy=0x7f79dc13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 1 -- 172.21.15.148:0/2136506154 shutdown_connections 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 1 -- 172.21.15.148:0/2136506154 >> 172.21.15.148:0/2136506154 conn(0x7f79dc0a5640 msgr2=0x7f79dc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 1 -- 172.21.15.148:0/2136506154 shutdown_connections 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.620+0000 7f79e3dc6640 1 -- 172.21.15.148:0/2136506154 wait complete. 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient: build_initial_monmap 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient: monmap: 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:01.638 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 1 Processor -- start 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 1 -- start start 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient: init 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f79dc09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f79dc09c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:01.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.621+0000 7f79e3dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f79dc09cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc0740a0 con 0x7f79dc09bb70 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc13c070 con 0x7f79dc09cc30 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc134300 con 0x7f79dc09c660 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 10 monclient(hunting): authenticate will time out at 2102.248049s 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e1361640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f79dc09c660 0x7f79dc13b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51938/0 (socket says 172.21.15.148:51938) 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e1361640 1 -- 172.21.15.148:0/1513822634 learned_addr learned my addr 172.21.15.148:0/1513822634 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1341330426 0 0) 0x7f79dc09f030 con 0x7f79dc09bb70 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): my global_id is 174135 2024-09-16T04:28:01.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79c0003520 con 0x7f79dc09bb70 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2951405007 0 0) 0x7f79dc134300 con 0x7f79dc09c660 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): my global_id is 154385 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79dc09f030 con 0x7f79dc09c660 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940656735 0 0) 0x7f79dc13c070 con 0x7f79dc09cc30 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): my global_id is 164263 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79dc134300 con 0x7f79dc09cc30 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1988586061 0 0) 0x7f79c0003520 con 0x7f79dc09bb70 2024-09-16T04:28:01.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79dc13c070 con 0x7f79dc09bb70 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3622718905 0 0) 0x7f79dc09f030 con 0x7f79dc09c660 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79c0003520 con 0x7f79dc09c660 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f79d00017f0 con 0x7f79dc09bb70 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.642 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f79d8001c10 con 0x7f79dc09c660 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.643 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3564305576 0 0) 0x7f79dc134300 con 0x7f79dc09cc30 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79dc09f030 con 0x7f79dc09cc30 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f79d4001a30 con 0x7f79dc09cc30 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.644 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1456463718 0 0) 0x7f79dc13c070 con 0x7f79dc09bb70 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 >> v1:172.21.15.148:6790/0 conn(0x7f79dc09c660 legacy=0x7f79dc13b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 >> v1:172.21.15.171:6789/0 conn(0x7f79dc09cc30 legacy=0x7f79dc1d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 monclient: found mon.a 2024-09-16T04:28:01.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc11c9b0 con 0x7f79dc09bb70 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: _finish_auth 0 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 5 monclient: authenticate success, global_id 174135 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 10 monclient: _renew_subs 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc09ad00 con 0x7f79dc09bb70 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 10 monclient: _renew_subs 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79dc1d2eb0 con 0x7f79dc09bb70 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79d00035f0 con 0x7f79dc09bb70 2024-09-16T04:28:01.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f79d0003940 con 0x7f79dc09bb70 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79cffff640 4 set_mon_vals no callback set 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.622+0000 7f79ce7fc640 10 monclient: dump: 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.647 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.623+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f79d0003940 con 0x7f79dc09bb70 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.623+0000 7f79e3dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.623+0000 7f79e3dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.623+0000 7f79e3dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.623+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c4005080 con 0x7f79dc09bb70 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.623+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f79d0046550 con 0x7f79dc09bb70 2024-09-16T04:28:01.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.624+0000 7f79e1361640 10 monclient: get_auth_request con 0x7f79c00391a0 auth_method 0 2024-09-16T04:28:01.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.628+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f79dc0740a0 con 0x7f79dc09bb70 2024-09-16T04:28:01.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.628+0000 7f79ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.628+0000 7f79ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:01.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.789+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f79c4000b90 con 0x7f79c00391a0 2024-09-16T04:28:01.794 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.792+0000 7f79ce7fc640 1 -- 172.21.15.148:0/1513822634 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77985 (unknown 2965378022 0 2825265590) 0x7f79c4000b90 con 0x7f79c00391a0 2024-09-16T04:28:01.794 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:28:01.795 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:28:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 >> v1:172.21.15.171:6812/7327273 conn(0x7f79c00391a0 legacy=0x7f79c003b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 10 monclient: shutdown 2024-09-16T04:28:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 >> v1:172.21.15.148:6789/0 conn(0x7f79dc09bb70 legacy=0x7f79dc13b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 shutdown_connections 2024-09-16T04:28:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 >> 172.21.15.148:0/1513822634 conn(0x7f79dc0a5640 msgr2=0x7f79dc072880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 shutdown_connections 2024-09-16T04:28:01.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.794+0000 7f79e3dc6640 1 -- 172.21.15.148:0/1513822634 wait complete. 2024-09-16T04:28:01.806 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":897,"stamp":"2024-09-16T04:28:01.549244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197136,"kb_used_data":17608,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566033904,"statfs":{"total":579820584960,"available":579618717696,"internally_reserved":0,"allocated":18030592,"data_stored":15149110,"data_compressed":1198654,"data_compressed_allocated":2351104,"data_compressed_original":6012928,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003016"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":306,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:35.308006+0000","last_change":"2024-09-16T04:11:11.973056+0000","last_active":"2024-09-16T04:24:35.308006+0000","last_peered":"2024-09-16T04:24:35.308006+0000","last_clean":"2024-09-16T04:24:35.308006+0000","last_became_active":"2024-09-16T04:11:11.972654+0000","last_became_peered":"2024-09-16T04:11:11.972654+0000","last_unstale":"2024-09-16T04:24:35.308006+0000","last_undegraded":"2024-09-16T04:24:35.308006+0000","last_fullsized":"2024-09-16T04:24:35.308006+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:33.067826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298089+0000","last_change":"2024-09-16T04:11:11.971214+0000","last_active":"2024-09-16T04:24:11.298089+0000","last_peered":"2024-09-16T04:24:11.298089+0000","last_clean":"2024-09-16T04:24:11.298089+0000","last_became_active":"2024-09-16T04:11:11.970821+0000","last_became_peered":"2024-09-16T04:11:11.970821+0000","last_unstale":"2024-09-16T04:24:11.298089+0000","last_undegraded":"2024-09-16T04:24:11.298089+0000","last_fullsized":"2024-09-16T04:24:11.298089+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:04.704581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":279,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:40.308623+0000","last_change":"2024-09-16T04:11:11.972099+0000","last_active":"2024-09-16T04:24:40.308623+0000","last_peered":"2024-09-16T04:24:40.308623+0000","last_clean":"2024-09-16T04:24:40.308623+0000","last_became_active":"2024-09-16T04:11:11.971647+0000","last_became_peered":"2024-09-16T04:11:11.971647+0000","last_unstale":"2024-09-16T04:24:40.308623+0000","last_undegraded":"2024-09-16T04:24:40.308623+0000","last_fullsized":"2024-09-16T04:24:40.308623+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:24.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298065+0000","last_change":"2024-09-16T04:11:11.974362+0000","last_active":"2024-09-16T04:24:11.298065+0000","last_peered":"2024-09-16T04:24:11.298065+0000","last_clean":"2024-09-16T04:24:11.298065+0000","last_became_active":"2024-09-16T04:11:11.974022+0000","last_became_peered":"2024-09-16T04:11:11.974022+0000","last_unstale":"2024-09-16T04:24:11.298065+0000","last_undegraded":"2024-09-16T04:24:11.298065+0000","last_fullsized":"2024-09-16T04:24:11.298065+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:35.453472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:11.304629+0000","last_change":"2024-09-16T04:11:12.969110+0000","last_active":"2024-09-16T04:25:11.304629+0000","last_peered":"2024-09-16T04:25:11.304629+0000","last_clean":"2024-09-16T04:25:11.304629+0000","last_became_active":"2024-09-16T04:11:12.968688+0000","last_became_peered":"2024-09-16T04:11:12.968688+0000","last_unstale":"2024-09-16T04:25:11.304629+0000","last_undegraded":"2024-09-16T04:25:11.304629+0000","last_fullsized":"2024-09-16T04:25:11.304629+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:52.202804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":294,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:40.308632+0000","last_change":"2024-09-16T04:11:11.972073+0000","last_active":"2024-09-16T04:24:40.308632+0000","last_peered":"2024-09-16T04:24:40.308632+0000","last_clean":"2024-09-16T04:24:40.308632+0000","last_became_active":"2024-09-16T04:11:11.971583+0000","last_became_peered":"2024-09-16T04:11:11.971583+0000","last_unstale":"2024-09-16T04:24:40.308632+0000","last_undegraded":"2024-09-16T04:24:40.308632+0000","last_fullsized":"2024-09-16T04:24:40.308632+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:15.678578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395998+0000","last_change":"2024-09-16T04:11:11.975563+0000","last_active":"2024-09-16T04:24:11.395998+0000","last_peered":"2024-09-16T04:24:11.395998+0000","last_clean":"2024-09-16T04:24:11.395998+0000","last_became_active":"2024-09-16T04:11:11.975180+0000","last_became_peered":"2024-09-16T04:11:11.975180+0000","last_unstale":"2024-09-16T04:24:11.395998+0000","last_undegraded":"2024-09-16T04:24:11.395998+0000","last_fullsized":"2024-09-16T04:24:11.395998+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:27.461051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403830+0000","last_change":"2024-09-16T04:11:11.970716+0000","last_active":"2024-09-16T04:25:48.403830+0000","last_peered":"2024-09-16T04:25:48.403830+0000","last_clean":"2024-09-16T04:25:48.403830+0000","last_became_active":"2024-09-16T04:11:11.970247+0000","last_became_peered":"2024-09-16T04:11:11.970247+0000","last_unstale":"2024-09-16T04:25:48.403830+0000","last_undegraded":"2024-09-16T04:25:48.403830+0000","last_fullsized":"2024-09-16T04:25:48.403830+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:37.770332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:16.298525+0000","last_change":"2024-09-16T04:11:11.973578+0000","last_active":"2024-09-16T04:24:16.298525+0000","last_peered":"2024-09-16T04:24:16.298525+0000","last_clean":"2024-09-16T04:24:16.298525+0000","last_became_active":"2024-09-16T04:11:11.973162+0000","last_became_peered":"2024-09-16T04:11:11.973162+0000","last_unstale":"2024-09-16T04:24:16.298525+0000","last_undegraded":"2024-09-16T04:24:16.298525+0000","last_fullsized":"2024-09-16T04:24:16.298525+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:31.864317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:46.399846+0000","last_change":"2024-09-16T04:11:11.975126+0000","last_active":"2024-09-16T04:24:46.399846+0000","last_peered":"2024-09-16T04:24:46.399846+0000","last_clean":"2024-09-16T04:24:46.399846+0000","last_became_active":"2024-09-16T04:11:11.974614+0000","last_became_peered":"2024-09-16T04:11:11.974614+0000","last_unstale":"2024-09-16T04:24:46.399846+0000","last_undegraded":"2024-09-16T04:24:46.399846+0000","last_fullsized":"2024-09-16T04:24:46.399846+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:25:32.801105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":232,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:33.300353+0000","last_change":"2024-09-16T04:11:11.973087+0000","last_active":"2024-09-16T04:24:33.300353+0000","last_peered":"2024-09-16T04:24:33.300353+0000","last_clean":"2024-09-16T04:24:33.300353+0000","last_became_active":"2024-09-16T04:11:11.972700+0000","last_became_peered":"2024-09-16T04:11:11.972700+0000","last_unstale":"2024-09-16T04:24:33.300353+0000","last_undegraded":"2024-09-16T04:24:33.300353+0000","last_fullsized":"2024-09-16T04:24:33.300353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:13.209774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403797+0000","last_change":"2024-09-16T04:11:11.972841+0000","last_active":"2024-09-16T04:25:48.403797+0000","last_peered":"2024-09-16T04:25:48.403797+0000","last_clean":"2024-09-16T04:25:48.403797+0000","last_became_active":"2024-09-16T04:11:11.972431+0000","last_became_peered":"2024-09-16T04:11:11.972431+0000","last_unstale":"2024-09-16T04:25:48.403797+0000","last_undegraded":"2024-09-16T04:25:48.403797+0000","last_fullsized":"2024-09-16T04:25:48.403797+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:31.140262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":179,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403820+0000","last_change":"2024-09-16T04:11:11.972928+0000","last_active":"2024-09-16T04:25:48.403820+0000","last_peered":"2024-09-16T04:25:48.403820+0000","last_clean":"2024-09-16T04:25:48.403820+0000","last_became_active":"2024-09-16T04:11:11.972513+0000","last_became_peered":"2024-09-16T04:11:11.972513+0000","last_unstale":"2024-09-16T04:25:48.403820+0000","last_undegraded":"2024-09-16T04:25:48.403820+0000","last_fullsized":"2024-09-16T04:25:48.403820+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:31.437265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":307,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:35.307984+0000","last_change":"2024-09-16T04:11:11.971270+0000","last_active":"2024-09-16T04:24:35.307984+0000","last_peered":"2024-09-16T04:24:35.307984+0000","last_clean":"2024-09-16T04:24:35.307984+0000","last_became_active":"2024-09-16T04:11:11.970869+0000","last_became_peered":"2024-09-16T04:11:11.970869+0000","last_unstale":"2024-09-16T04:24:35.307984+0000","last_undegraded":"2024-09-16T04:24:35.307984+0000","last_fullsized":"2024-09-16T04:24:35.307984+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:57:26.746389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:50.315983+0000","last_change":"2024-09-16T04:11:12.974381+0000","last_active":"2024-09-16T04:25:50.315983+0000","last_peered":"2024-09-16T04:25:50.315983+0000","last_clean":"2024-09-16T04:25:50.315983+0000","last_became_active":"2024-09-16T04:11:12.974021+0000","last_became_peered":"2024-09-16T04:11:12.974021+0000","last_unstale":"2024-09-16T04:25:50.315983+0000","last_undegraded":"2024-09-16T04:25:50.315983+0000","last_fullsized":"2024-09-16T04:25:50.315983+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:37:30.702246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:13.400389+0000","last_change":"2024-09-16T04:11:11.972888+0000","last_active":"2024-09-16T04:25:13.400389+0000","last_peered":"2024-09-16T04:25:13.400389+0000","last_clean":"2024-09-16T04:25:13.400389+0000","last_became_active":"2024-09-16T04:11:11.972527+0000","last_became_peered":"2024-09-16T04:11:11.972527+0000","last_unstale":"2024-09-16T04:25:13.400389+0000","last_undegraded":"2024-09-16T04:25:13.400389+0000","last_fullsized":"2024-09-16T04:25:13.400389+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:15.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":585,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:36.398725+0000","last_change":"2024-09-16T04:11:11.976349+0000","last_active":"2024-09-16T04:24:36.398725+0000","last_peered":"2024-09-16T04:24:36.398725+0000","last_clean":"2024-09-16T04:24:36.398725+0000","last_became_active":"2024-09-16T04:11:11.975960+0000","last_became_peered":"2024-09-16T04:11:11.975960+0000","last_unstale":"2024-09-16T04:24:36.398725+0000","last_undegraded":"2024-09-16T04:24:36.398725+0000","last_fullsized":"2024-09-16T04:24:36.398725+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:55.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":437,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308206+0000","last_change":"2024-09-16T04:11:11.978921+0000","last_active":"2024-09-16T04:25:46.308206+0000","last_peered":"2024-09-16T04:25:46.308206+0000","last_clean":"2024-09-16T04:25:46.308206+0000","last_became_active":"2024-09-16T04:11:11.971320+0000","last_became_peered":"2024-09-16T04:11:11.971320+0000","last_unstale":"2024-09-16T04:25:46.308206+0000","last_undegraded":"2024-09-16T04:25:46.308206+0000","last_fullsized":"2024-09-16T04:25:46.308206+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:13.147521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:41.399305+0000","last_change":"2024-09-16T04:11:11.975014+0000","last_active":"2024-09-16T04:24:41.399305+0000","last_peered":"2024-09-16T04:24:41.399305+0000","last_clean":"2024-09-16T04:24:41.399305+0000","last_became_active":"2024-09-16T04:11:11.974359+0000","last_became_peered":"2024-09-16T04:11:11.974359+0000","last_unstale":"2024-09-16T04:24:41.399305+0000","last_undegraded":"2024-09-16T04:24:41.399305+0000","last_fullsized":"2024-09-16T04:24:41.399305+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:29.591854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:50.315959+0000","last_change":"2024-09-16T04:11:11.973501+0000","last_active":"2024-09-16T04:25:50.315959+0000","last_peered":"2024-09-16T04:25:50.315959+0000","last_clean":"2024-09-16T04:25:50.315959+0000","last_became_active":"2024-09-16T04:11:11.973107+0000","last_became_peered":"2024-09-16T04:11:11.973107+0000","last_unstale":"2024-09-16T04:25:50.315959+0000","last_undegraded":"2024-09-16T04:25:50.315959+0000","last_fullsized":"2024-09-16T04:25:50.315959+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:14.670234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":448,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308156+0000","last_change":"2024-09-16T04:11:11.984013+0000","last_active":"2024-09-16T04:25:46.308156+0000","last_peered":"2024-09-16T04:25:46.308156+0000","last_clean":"2024-09-16T04:25:46.308156+0000","last_became_active":"2024-09-16T04:11:11.975852+0000","last_became_peered":"2024-09-16T04:11:11.975852+0000","last_unstale":"2024-09-16T04:25:46.308156+0000","last_undegraded":"2024-09-16T04:25:46.308156+0000","last_fullsized":"2024-09-16T04:25:46.308156+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:19.909892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":438,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403840+0000","last_change":"2024-09-16T04:11:12.969208+0000","last_active":"2024-09-16T04:25:48.403840+0000","last_peered":"2024-09-16T04:25:48.403840+0000","last_clean":"2024-09-16T04:25:48.403840+0000","last_became_active":"2024-09-16T04:11:12.968870+0000","last_became_peered":"2024-09-16T04:11:12.968870+0000","last_unstale":"2024-09-16T04:25:48.403840+0000","last_undegraded":"2024-09-16T04:25:48.403840+0000","last_fullsized":"2024-09-16T04:25:48.403840+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:14.254897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.405976+0000","last_change":"2024-09-16T04:11:11.976009+0000","last_active":"2024-09-16T04:25:46.405976+0000","last_peered":"2024-09-16T04:25:46.405976+0000","last_clean":"2024-09-16T04:25:46.405976+0000","last_became_active":"2024-09-16T04:11:11.975676+0000","last_became_peered":"2024-09-16T04:11:11.975676+0000","last_unstale":"2024-09-16T04:25:46.405976+0000","last_undegraded":"2024-09-16T04:25:46.405976+0000","last_fullsized":"2024-09-16T04:25:46.405976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:00:21.147770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:13.394779+0000","last_change":"2024-09-16T04:11:11.977786+0000","last_active":"2024-09-16T04:24:13.394779+0000","last_peered":"2024-09-16T04:24:13.394779+0000","last_clean":"2024-09-16T04:24:13.394779+0000","last_became_active":"2024-09-16T04:11:11.977456+0000","last_became_peered":"2024-09-16T04:11:11.977456+0000","last_unstale":"2024-09-16T04:24:13.394779+0000","last_undegraded":"2024-09-16T04:24:13.394779+0000","last_fullsized":"2024-09-16T04:24:13.394779+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:46.427728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395976+0000","last_change":"2024-09-16T04:11:11.972539+0000","last_active":"2024-09-16T04:24:11.395976+0000","last_peered":"2024-09-16T04:24:11.395976+0000","last_clean":"2024-09-16T04:24:11.395976+0000","last_became_active":"2024-09-16T04:11:11.972183+0000","last_became_peered":"2024-09-16T04:11:11.972183+0000","last_unstale":"2024-09-16T04:24:11.395976+0000","last_undegraded":"2024-09-16T04:24:11.395976+0000","last_fullsized":"2024-09-16T04:24:11.395976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:54.054660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575111,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34224,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337616,"statfs":{"total":96636764160,"available":96601718784,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34228,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337612,"statfs":{"total":96636764160,"available":96601714688,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30068,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341772,"statfs":{"total":96636764160,"available":96605974528,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640518,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30064,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341776,"statfs":{"total":96636764160,"available":96605978624,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640516,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:28:01.808 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.904+0000 7fa92ef6d640 10 monclient: get_monmap_and_config 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.904+0000 7fa92ef6d640 10 monclient: build_initial_monmap 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.904+0000 7fa92ef6d640 10 monclient: monmap: 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:01.905418+0000 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:01.905418+0000 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:01.906 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:01.907 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:01.907 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 1 Processor -- start 2024-09-16T04:28:01.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 1 -- start start 2024-09-16T04:28:01.914 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient: init 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9280a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient(hunting): picked mon.noname-b con 0x7fa92809b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient(hunting): picked mon.noname-c con 0x7fa92809b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92809e040 con 0x7fa9280a0e90 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92809d660 con 0x7fa92809b5c0 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92809c0d0 con 0x7fa92809b170 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92ef6d640 10 monclient(hunting): authenticate will time out at 2102.530805s 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa927fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa92809b170 0x7fa9281383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51940/0 (socket says 172.21.15.148:51940) 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa92cd09640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fa9280a0e90 0x7fa92809aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:40774/0 (socket says 172.21.15.148:40774) 2024-09-16T04:28:01.915 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.905+0000 7fa927fff640 1 -- 172.21.15.148:0/3851540727 learned_addr learned my addr 172.21.15.148:0/3851540727 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040667784 0 0) 0x7fa92809c0d0 con 0x7fa92809b170 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): my global_id is 154388 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa918003520 con 0x7fa92809b170 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537553093 0 0) 0x7fa91c001110 con 0x7fa9280a0e90 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient(hunting): my global_id is 174138 2024-09-16T04:28:01.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa92809c0d0 con 0x7fa9280a0e90 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 938011243 0 0) 0x7fa918003520 con 0x7fa92809b170 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 >> v1:172.21.15.171:6789/0 conn(0x7fa92809b5c0 legacy=0x7fa92813bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 >> v1:172.21.15.148:6789/0 conn(0x7fa9280a0e90 legacy=0x7fa92809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: _finish_hunting 0 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 monclient: found mon.noname-b 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92809d380 con 0x7fa92809b170 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: _finish_auth 0 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: _check_auth_tickets 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa914001540 con 0x7fa92809b170 2024-09-16T04:28:01.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: dump: 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa92ef6d640 5 monclient: authenticate success, global_id 154388 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa92ef6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa914001730 con 0x7fa92809b170 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 1 -- 172.21.15.148:0/3851540727 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa914001b40 con 0x7fa92809b170 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa9277fe640 10 monclient: dump: 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.919 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa92ef6d640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.906+0000 7fa92ef6d640 4 set_mon_vals no callback set 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 10 monclient: shutdown 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 1 -- 172.21.15.148:0/3851540727 >> v1:172.21.15.148:6790/0 conn(0x7fa92809b170 legacy=0x7fa9281383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 1 -- 172.21.15.148:0/3851540727 shutdown_connections 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 1 -- 172.21.15.148:0/3851540727 >> 172.21.15.148:0/3851540727 conn(0x7fa9280a5640 msgr2=0x7fa9280a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 1 -- 172.21.15.148:0/3851540727 shutdown_connections 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 1 -- 172.21.15.148:0/3851540727 wait complete. 2024-09-16T04:28:01.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 10 monclient: build_initial_monmap 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.907+0000 7fa92ef6d640 10 monclient: monmap: 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 1 Processor -- start 2024-09-16T04:28:01.921 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 1 -- start start 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient: init 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient(hunting): picked mon.noname-b con 0x7fa92809b170 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient(hunting): picked mon.noname-c con 0x7fa92809b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9280a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9281c5c90 con 0x7fa9280a0e90 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9281aa420 con 0x7fa92809b170 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9281acee0 con 0x7fa92809b5c0 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa92ef6d640 10 monclient(hunting): authenticate will time out at 2102.533764s 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa927fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fa92809b5c0 0x7fa9281d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:51942/0 (socket says 172.21.15.148:51942) 2024-09-16T04:28:01.922 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa927fff640 1 -- 172.21.15.148:0/1895030541 learned_addr learned my addr 172.21.15.148:0/1895030541 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802093122 0 0) 0x7fa9281acee0 con 0x7fa92809b5c0 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa9257fa640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa9257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa9257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.908+0000 7fa9257fa640 10 monclient(hunting): my global_id is 154391 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa90c003520 con 0x7fa92809b5c0 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471349976 0 0) 0x7fa9281c5c90 con 0x7fa9280a0e90 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): my global_id is 174141 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9281acee0 con 0x7fa9280a0e90 2024-09-16T04:28:01.923 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825816465 0 0) 0x7fa9281aa420 con 0x7fa92809b170 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _negotiate 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): my global_id is 164269 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9281c5c90 con 0x7fa92809b170 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2441258401 0 0) 0x7fa90c003520 con 0x7fa92809b5c0 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9281aa420 con 0x7fa92809b5c0 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa9140047e0 con 0x7fa92809b5c0 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): dump: 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.924 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3147250490 0 0) 0x7fa9281acee0 con 0x7fa9280a0e90 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa90c003520 con 0x7fa9280a0e90 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa920001af0 con 0x7fa9280a0e90 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:01.925 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient(hunting): dump: 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 766828126 0 0) 0x7fa9281aa420 con 0x7fa92809b5c0 2024-09-16T04:28:01.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 >> v1:172.21.15.171:6789/0 conn(0x7fa92809b170 legacy=0x7fa9281c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 >> v1:172.21.15.148:6789/0 conn(0x7fa9280a0e90 legacy=0x7fa9281d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: _finish_hunting 0 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 monclient: found mon.c 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9281aac10 con 0x7fa92809b5c0 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: _finish_auth 0 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: _check_auth_tickets 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 5 monclient: authenticate success, global_id 154391 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 10 monclient: _renew_subs 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9281acc00 con 0x7fa92809b5c0 2024-09-16T04:28:01.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 10 monclient: _renew_subs 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9281c7ff0 con 0x7fa92809b5c0 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa914001730 con 0x7fa92809b5c0 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fa914004dd0 con 0x7fa92809b5c0 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa926ffd640 4 set_mon_vals no callback set 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.909+0000 7fa9257fa640 10 monclient: dump: 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.928 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fa914004dd0 con 0x7fa92809b5c0 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa92ef6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa92ef6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa92ef6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa908005080 con 0x7fa92809b5c0 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fa914047960 con 0x7fa92809b5c0 2024-09-16T04:28:01.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.910+0000 7fa92cd09640 10 monclient: get_auth_request con 0x7fa90c038e60 auth_method 0 2024-09-16T04:28:01.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.915+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fa914015bc0 con 0x7fa92809b5c0 2024-09-16T04:28:01.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.915+0000 7fa9257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:01.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:01.915+0000 7fa9257fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:02.079 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.078+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa908000b90 con 0x7fa90c038e60 2024-09-16T04:28:02.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.080+0000 7fa9257fa640 1 -- 172.21.15.148:0/1895030541 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77985 (unknown 2965378022 0 2825265590) 0x7fa908000b90 con 0x7fa90c038e60 2024-09-16T04:28:02.082 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:28:02.083 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:28:02.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 >> v1:172.21.15.171:6812/7327273 conn(0x7fa90c038e60 legacy=0x7fa90c03b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 10 monclient: shutdown 2024-09-16T04:28:02.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:02.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 >> v1:172.21.15.148:6790/0 conn(0x7fa92809b5c0 legacy=0x7fa9281d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 shutdown_connections 2024-09-16T04:28:02.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 >> 172.21.15.148:0/1895030541 conn(0x7fa9280a5640 msgr2=0x7fa9280a3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:02.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 shutdown_connections 2024-09-16T04:28:02.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.082+0000 7fa92ef6d640 1 -- 172.21.15.148:0/1895030541 wait complete. 2024-09-16T04:28:02.094 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":897,"stamp":"2024-09-16T04:28:01.549244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197136,"kb_used_data":17608,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566033904,"statfs":{"total":579820584960,"available":579618717696,"internally_reserved":0,"allocated":18030592,"data_stored":15149110,"data_compressed":1198654,"data_compressed_allocated":2351104,"data_compressed_original":6012928,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003016"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":306,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:35.308006+0000","last_change":"2024-09-16T04:11:11.973056+0000","last_active":"2024-09-16T04:24:35.308006+0000","last_peered":"2024-09-16T04:24:35.308006+0000","last_clean":"2024-09-16T04:24:35.308006+0000","last_became_active":"2024-09-16T04:11:11.972654+0000","last_became_peered":"2024-09-16T04:11:11.972654+0000","last_unstale":"2024-09-16T04:24:35.308006+0000","last_undegraded":"2024-09-16T04:24:35.308006+0000","last_fullsized":"2024-09-16T04:24:35.308006+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:33.067826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298089+0000","last_change":"2024-09-16T04:11:11.971214+0000","last_active":"2024-09-16T04:24:11.298089+0000","last_peered":"2024-09-16T04:24:11.298089+0000","last_clean":"2024-09-16T04:24:11.298089+0000","last_became_active":"2024-09-16T04:11:11.970821+0000","last_became_peered":"2024-09-16T04:11:11.970821+0000","last_unstale":"2024-09-16T04:24:11.298089+0000","last_undegraded":"2024-09-16T04:24:11.298089+0000","last_fullsized":"2024-09-16T04:24:11.298089+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:04.704581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":279,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:40.308623+0000","last_change":"2024-09-16T04:11:11.972099+0000","last_active":"2024-09-16T04:24:40.308623+0000","last_peered":"2024-09-16T04:24:40.308623+0000","last_clean":"2024-09-16T04:24:40.308623+0000","last_became_active":"2024-09-16T04:11:11.971647+0000","last_became_peered":"2024-09-16T04:11:11.971647+0000","last_unstale":"2024-09-16T04:24:40.308623+0000","last_undegraded":"2024-09-16T04:24:40.308623+0000","last_fullsized":"2024-09-16T04:24:40.308623+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:24.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298065+0000","last_change":"2024-09-16T04:11:11.974362+0000","last_active":"2024-09-16T04:24:11.298065+0000","last_peered":"2024-09-16T04:24:11.298065+0000","last_clean":"2024-09-16T04:24:11.298065+0000","last_became_active":"2024-09-16T04:11:11.974022+0000","last_became_peered":"2024-09-16T04:11:11.974022+0000","last_unstale":"2024-09-16T04:24:11.298065+0000","last_undegraded":"2024-09-16T04:24:11.298065+0000","last_fullsized":"2024-09-16T04:24:11.298065+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:35.453472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:11.304629+0000","last_change":"2024-09-16T04:11:12.969110+0000","last_active":"2024-09-16T04:25:11.304629+0000","last_peered":"2024-09-16T04:25:11.304629+0000","last_clean":"2024-09-16T04:25:11.304629+0000","last_became_active":"2024-09-16T04:11:12.968688+0000","last_became_peered":"2024-09-16T04:11:12.968688+0000","last_unstale":"2024-09-16T04:25:11.304629+0000","last_undegraded":"2024-09-16T04:25:11.304629+0000","last_fullsized":"2024-09-16T04:25:11.304629+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:52.202804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":294,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:40.308632+0000","last_change":"2024-09-16T04:11:11.972073+0000","last_active":"2024-09-16T04:24:40.308632+0000","last_peered":"2024-09-16T04:24:40.308632+0000","last_clean":"2024-09-16T04:24:40.308632+0000","last_became_active":"2024-09-16T04:11:11.971583+0000","last_became_peered":"2024-09-16T04:11:11.971583+0000","last_unstale":"2024-09-16T04:24:40.308632+0000","last_undegraded":"2024-09-16T04:24:40.308632+0000","last_fullsized":"2024-09-16T04:24:40.308632+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:15.678578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395998+0000","last_change":"2024-09-16T04:11:11.975563+0000","last_active":"2024-09-16T04:24:11.395998+0000","last_peered":"2024-09-16T04:24:11.395998+0000","last_clean":"2024-09-16T04:24:11.395998+0000","last_became_active":"2024-09-16T04:11:11.975180+0000","last_became_peered":"2024-09-16T04:11:11.975180+0000","last_unstale":"2024-09-16T04:24:11.395998+0000","last_undegraded":"2024-09-16T04:24:11.395998+0000","last_fullsized":"2024-09-16T04:24:11.395998+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:27.461051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403830+0000","last_change":"2024-09-16T04:11:11.970716+0000","last_active":"2024-09-16T04:25:48.403830+0000","last_peered":"2024-09-16T04:25:48.403830+0000","last_clean":"2024-09-16T04:25:48.403830+0000","last_became_active":"2024-09-16T04:11:11.970247+0000","last_became_peered":"2024-09-16T04:11:11.970247+0000","last_unstale":"2024-09-16T04:25:48.403830+0000","last_undegraded":"2024-09-16T04:25:48.403830+0000","last_fullsized":"2024-09-16T04:25:48.403830+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:37.770332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:16.298525+0000","last_change":"2024-09-16T04:11:11.973578+0000","last_active":"2024-09-16T04:24:16.298525+0000","last_peered":"2024-09-16T04:24:16.298525+0000","last_clean":"2024-09-16T04:24:16.298525+0000","last_became_active":"2024-09-16T04:11:11.973162+0000","last_became_peered":"2024-09-16T04:11:11.973162+0000","last_unstale":"2024-09-16T04:24:16.298525+0000","last_undegraded":"2024-09-16T04:24:16.298525+0000","last_fullsized":"2024-09-16T04:24:16.298525+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:31.864317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:46.399846+0000","last_change":"2024-09-16T04:11:11.975126+0000","last_active":"2024-09-16T04:24:46.399846+0000","last_peered":"2024-09-16T04:24:46.399846+0000","last_clean":"2024-09-16T04:24:46.399846+0000","last_became_active":"2024-09-16T04:11:11.974614+0000","last_became_peered":"2024-09-16T04:11:11.974614+0000","last_unstale":"2024-09-16T04:24:46.399846+0000","last_undegraded":"2024-09-16T04:24:46.399846+0000","last_fullsized":"2024-09-16T04:24:46.399846+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:25:32.801105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":232,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:33.300353+0000","last_change":"2024-09-16T04:11:11.973087+0000","last_active":"2024-09-16T04:24:33.300353+0000","last_peered":"2024-09-16T04:24:33.300353+0000","last_clean":"2024-09-16T04:24:33.300353+0000","last_became_active":"2024-09-16T04:11:11.972700+0000","last_became_peered":"2024-09-16T04:11:11.972700+0000","last_unstale":"2024-09-16T04:24:33.300353+0000","last_undegraded":"2024-09-16T04:24:33.300353+0000","last_fullsized":"2024-09-16T04:24:33.300353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:13.209774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403797+0000","last_change":"2024-09-16T04:11:11.972841+0000","last_active":"2024-09-16T04:25:48.403797+0000","last_peered":"2024-09-16T04:25:48.403797+0000","last_clean":"2024-09-16T04:25:48.403797+0000","last_became_active":"2024-09-16T04:11:11.972431+0000","last_became_peered":"2024-09-16T04:11:11.972431+0000","last_unstale":"2024-09-16T04:25:48.403797+0000","last_undegraded":"2024-09-16T04:25:48.403797+0000","last_fullsized":"2024-09-16T04:25:48.403797+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:31.140262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":179,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403820+0000","last_change":"2024-09-16T04:11:11.972928+0000","last_active":"2024-09-16T04:25:48.403820+0000","last_peered":"2024-09-16T04:25:48.403820+0000","last_clean":"2024-09-16T04:25:48.403820+0000","last_became_active":"2024-09-16T04:11:11.972513+0000","last_became_peered":"2024-09-16T04:11:11.972513+0000","last_unstale":"2024-09-16T04:25:48.403820+0000","last_undegraded":"2024-09-16T04:25:48.403820+0000","last_fullsized":"2024-09-16T04:25:48.403820+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:31.437265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":307,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:35.307984+0000","last_change":"2024-09-16T04:11:11.971270+0000","last_active":"2024-09-16T04:24:35.307984+0000","last_peered":"2024-09-16T04:24:35.307984+0000","last_clean":"2024-09-16T04:24:35.307984+0000","last_became_active":"2024-09-16T04:11:11.970869+0000","last_became_peered":"2024-09-16T04:11:11.970869+0000","last_unstale":"2024-09-16T04:24:35.307984+0000","last_undegraded":"2024-09-16T04:24:35.307984+0000","last_fullsized":"2024-09-16T04:24:35.307984+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:57:26.746389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:50.315983+0000","last_change":"2024-09-16T04:11:12.974381+0000","last_active":"2024-09-16T04:25:50.315983+0000","last_peered":"2024-09-16T04:25:50.315983+0000","last_clean":"2024-09-16T04:25:50.315983+0000","last_became_active":"2024-09-16T04:11:12.974021+0000","last_became_peered":"2024-09-16T04:11:12.974021+0000","last_unstale":"2024-09-16T04:25:50.315983+0000","last_undegraded":"2024-09-16T04:25:50.315983+0000","last_fullsized":"2024-09-16T04:25:50.315983+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:37:30.702246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:13.400389+0000","last_change":"2024-09-16T04:11:11.972888+0000","last_active":"2024-09-16T04:25:13.400389+0000","last_peered":"2024-09-16T04:25:13.400389+0000","last_clean":"2024-09-16T04:25:13.400389+0000","last_became_active":"2024-09-16T04:11:11.972527+0000","last_became_peered":"2024-09-16T04:11:11.972527+0000","last_unstale":"2024-09-16T04:25:13.400389+0000","last_undegraded":"2024-09-16T04:25:13.400389+0000","last_fullsized":"2024-09-16T04:25:13.400389+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:15.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":585,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:36.398725+0000","last_change":"2024-09-16T04:11:11.976349+0000","last_active":"2024-09-16T04:24:36.398725+0000","last_peered":"2024-09-16T04:24:36.398725+0000","last_clean":"2024-09-16T04:24:36.398725+0000","last_became_active":"2024-09-16T04:11:11.975960+0000","last_became_peered":"2024-09-16T04:11:11.975960+0000","last_unstale":"2024-09-16T04:24:36.398725+0000","last_undegraded":"2024-09-16T04:24:36.398725+0000","last_fullsized":"2024-09-16T04:24:36.398725+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:55.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":437,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308206+0000","last_change":"2024-09-16T04:11:11.978921+0000","last_active":"2024-09-16T04:25:46.308206+0000","last_peered":"2024-09-16T04:25:46.308206+0000","last_clean":"2024-09-16T04:25:46.308206+0000","last_became_active":"2024-09-16T04:11:11.971320+0000","last_became_peered":"2024-09-16T04:11:11.971320+0000","last_unstale":"2024-09-16T04:25:46.308206+0000","last_undegraded":"2024-09-16T04:25:46.308206+0000","last_fullsized":"2024-09-16T04:25:46.308206+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:13.147521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:41.399305+0000","last_change":"2024-09-16T04:11:11.975014+0000","last_active":"2024-09-16T04:24:41.399305+0000","last_peered":"2024-09-16T04:24:41.399305+0000","last_clean":"2024-09-16T04:24:41.399305+0000","last_became_active":"2024-09-16T04:11:11.974359+0000","last_became_peered":"2024-09-16T04:11:11.974359+0000","last_unstale":"2024-09-16T04:24:41.399305+0000","last_undegraded":"2024-09-16T04:24:41.399305+0000","last_fullsized":"2024-09-16T04:24:41.399305+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:29.591854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:50.315959+0000","last_change":"2024-09-16T04:11:11.973501+0000","last_active":"2024-09-16T04:25:50.315959+0000","last_peered":"2024-09-16T04:25:50.315959+0000","last_clean":"2024-09-16T04:25:50.315959+0000","last_became_active":"2024-09-16T04:11:11.973107+0000","last_became_peered":"2024-09-16T04:11:11.973107+0000","last_unstale":"2024-09-16T04:25:50.315959+0000","last_undegraded":"2024-09-16T04:25:50.315959+0000","last_fullsized":"2024-09-16T04:25:50.315959+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:14.670234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":448,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308156+0000","last_change":"2024-09-16T04:11:11.984013+0000","last_active":"2024-09-16T04:25:46.308156+0000","last_peered":"2024-09-16T04:25:46.308156+0000","last_clean":"2024-09-16T04:25:46.308156+0000","last_became_active":"2024-09-16T04:11:11.975852+0000","last_became_peered":"2024-09-16T04:11:11.975852+0000","last_unstale":"2024-09-16T04:25:46.308156+0000","last_undegraded":"2024-09-16T04:25:46.308156+0000","last_fullsized":"2024-09-16T04:25:46.308156+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:19.909892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":438,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403840+0000","last_change":"2024-09-16T04:11:12.969208+0000","last_active":"2024-09-16T04:25:48.403840+0000","last_peered":"2024-09-16T04:25:48.403840+0000","last_clean":"2024-09-16T04:25:48.403840+0000","last_became_active":"2024-09-16T04:11:12.968870+0000","last_became_peered":"2024-09-16T04:11:12.968870+0000","last_unstale":"2024-09-16T04:25:48.403840+0000","last_undegraded":"2024-09-16T04:25:48.403840+0000","last_fullsized":"2024-09-16T04:25:48.403840+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:14.254897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.405976+0000","last_change":"2024-09-16T04:11:11.976009+0000","last_active":"2024-09-16T04:25:46.405976+0000","last_peered":"2024-09-16T04:25:46.405976+0000","last_clean":"2024-09-16T04:25:46.405976+0000","last_became_active":"2024-09-16T04:11:11.975676+0000","last_became_peered":"2024-09-16T04:11:11.975676+0000","last_unstale":"2024-09-16T04:25:46.405976+0000","last_undegraded":"2024-09-16T04:25:46.405976+0000","last_fullsized":"2024-09-16T04:25:46.405976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:00:21.147770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:13.394779+0000","last_change":"2024-09-16T04:11:11.977786+0000","last_active":"2024-09-16T04:24:13.394779+0000","last_peered":"2024-09-16T04:24:13.394779+0000","last_clean":"2024-09-16T04:24:13.394779+0000","last_became_active":"2024-09-16T04:11:11.977456+0000","last_became_peered":"2024-09-16T04:11:11.977456+0000","last_unstale":"2024-09-16T04:24:13.394779+0000","last_undegraded":"2024-09-16T04:24:13.394779+0000","last_fullsized":"2024-09-16T04:24:13.394779+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:46.427728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395976+0000","last_change":"2024-09-16T04:11:11.972539+0000","last_active":"2024-09-16T04:24:11.395976+0000","last_peered":"2024-09-16T04:24:11.395976+0000","last_clean":"2024-09-16T04:24:11.395976+0000","last_became_active":"2024-09-16T04:11:11.972183+0000","last_became_peered":"2024-09-16T04:11:11.972183+0000","last_unstale":"2024-09-16T04:24:11.395976+0000","last_undegraded":"2024-09-16T04:24:11.395976+0000","last_fullsized":"2024-09-16T04:24:11.395976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:54.054660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575111,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34224,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337616,"statfs":{"total":96636764160,"available":96601718784,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34228,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337612,"statfs":{"total":96636764160,"available":96601714688,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30068,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341772,"statfs":{"total":96636764160,"available":96605974528,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640518,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30064,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341776,"statfs":{"total":96636764160,"available":96605978624,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640516,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:28:02.096 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:28:02.096 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.193+0000 7f07fa4e6640 10 monclient: get_monmap_and_config 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.193+0000 7f07fa4e6640 10 monclient: build_initial_monmap 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.193+0000 7f07fa4e6640 10 monclient: monmap: 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:02.194361+0000 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:02.194361+0000 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:02.195 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:02.196 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:02.196 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 1 Processor -- start 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 1 -- start start 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient: init 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f07f4133910 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f07f406c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:02.205 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f07f413d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f409b0d0 con 0x7f07f406c430 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f409b340 con 0x7f07f4133910 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f409b620 con 0x7f07f413d0e0 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07fa4e6640 10 monclient(hunting): authenticate will time out at 2102.819840s 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07f8a83640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f07f413d0e0 0x7f07f413f4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38570/0 (socket says 172.21.15.148:38570) 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07f8a83640 1 -- 172.21.15.148:0/1825865367 learned_addr learned my addr 172.21.15.148:0/1825865367 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.194+0000 7f07f37fe640 1 --1- 172.21.15.148:0/1825865367 >> v1:172.21.15.148:6789/0 conn(0x7f07f406c430 0x7f07f413b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48048/0 (socket says 172.21.15.148:48048) 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841352232 0 0) 0x7f07f409b620 con 0x7f07f413d0e0 2024-09-16T04:28:02.206 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): my global_id is 154394 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07dc003520 con 0x7f07f413d0e0 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 738522871 0 0) 0x7f07f409b0d0 con 0x7f07f406c430 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): my global_id is 174144 2024-09-16T04:28:02.207 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f409b620 con 0x7f07f406c430 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874186302 0 0) 0x7f07f409b340 con 0x7f07f4133910 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient(hunting): my global_id is 164272 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f409b0d0 con 0x7f07f4133910 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 638859042 0 0) 0x7f07dc003520 con 0x7f07f413d0e0 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 >> v1:172.21.15.171:6789/0 conn(0x7f07f4133910 legacy=0x7f07f4133cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 >> v1:172.21.15.148:6789/0 conn(0x7f07f406c430 legacy=0x7f07f413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:28:02.208 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 monclient: found mon.noname-b 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f40633c0 con 0x7f07f413d0e0 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f07ec001990 con 0x7f07f413d0e0 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07fa4e6640 5 monclient: authenticate success, global_id 154394 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:02.209 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: dump: 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.210 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07fa4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07ec001bf0 con 0x7f07f413d0e0 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07fa4e6640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 1 -- 172.21.15.148:0/1825865367 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f07ec002000 con 0x7f07f413d0e0 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07fa4e6640 4 set_mon_vals no callback set 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.195+0000 7f07f2ffd640 10 monclient: dump: 2024-09-16T04:28:02.211 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 10 monclient: shutdown 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 1 -- 172.21.15.148:0/1825865367 >> v1:172.21.15.148:6790/0 conn(0x7f07f413d0e0 legacy=0x7f07f413f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 1 -- 172.21.15.148:0/1825865367 shutdown_connections 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 1 -- 172.21.15.148:0/1825865367 >> 172.21.15.148:0/1825865367 conn(0x7f07f4062630 msgr2=0x7f07f4062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 1 -- 172.21.15.148:0/1825865367 shutdown_connections 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 1 -- 172.21.15.148:0/1825865367 wait complete. 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 10 monclient: build_initial_monmap 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 10 monclient: monmap: 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:02.213 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.196+0000 7f07fa4e6640 1 Processor -- start 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 1 -- start start 2024-09-16T04:28:02.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient: init 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f07f406c430 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f07f4133910 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f07f413d0e0 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f413c540 con 0x7f07f406c430 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f40a5360 con 0x7f07f4133910 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f40a4ea0 con 0x7f07f413d0e0 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:02.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07fa4e6640 10 monclient(hunting): authenticate will time out at 2102.822677s 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f3fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f07f406c430 0x7f07f413be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48060/0 (socket says 172.21.15.148:48060) 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f3fff640 1 -- 172.21.15.148:0/822820511 learned_addr learned my addr 172.21.15.148:0/822820511 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577336741 0 0) 0x7f07f409b0d0 con 0x7f07f406c430 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): my global_id is 174147 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e0003520 con 0x7f07f406c430 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1681610264 0 0) 0x7f07f40a4ea0 con 0x7f07f413d0e0 2024-09-16T04:28:02.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 10 monclient(hunting): my global_id is 154397 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.197+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f409b0d0 con 0x7f07f413d0e0 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347399365 0 0) 0x7f07f40a5360 con 0x7f07f4133910 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): my global_id is 164275 2024-09-16T04:28:02.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f40a4ea0 con 0x7f07f4133910 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4269388782 0 0) 0x7f07e0003520 con 0x7f07f406c430 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07f40a5360 con 0x7f07f406c430 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f07e4001990 con 0x7f07f406c430 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): dump: 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.218 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2762385589 0 0) 0x7f07f409b0d0 con 0x7f07f413d0e0 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f07e0003520 con 0x7f07f413d0e0 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f07ec001da0 con 0x7f07f413d0e0 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:02.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient(hunting): dump: 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3435280807 0 0) 0x7f07f40a5360 con 0x7f07f406c430 2024-09-16T04:28:02.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 >> v1:172.21.15.148:6790/0 conn(0x7f07f413d0e0 legacy=0x7f07f41d3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 >> v1:172.21.15.171:6789/0 conn(0x7f07f4133910 legacy=0x7f07f41d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: _finish_hunting 0 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 monclient: found mon.a 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f406e000 con 0x7f07f406c430 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 5 monclient: authenticate success, global_id 174147 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 10 monclient: _renew_subs 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f07f41345c0 con 0x7f07f406c430 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 10 monclient: _renew_subs 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f07f4132410 con 0x7f07f406c430 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f07e40038c0 con 0x7f07f406c430 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f07e4003c90 con 0x7f07f406c430 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f27fc640 4 set_mon_vals no callback set 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.198+0000 7f07f0ff9640 10 monclient: dump: 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.222 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f07e4003e30 con 0x7f07f406c430 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07fa4e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07fa4e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07fa4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d4005080 con 0x7f07f406c430 2024-09-16T04:28:02.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f07e40479f0 con 0x7f07f406c430 2024-09-16T04:28:02.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.199+0000 7f07f37fe640 10 monclient: get_auth_request con 0x7f07e0038e50 auth_method 0 2024-09-16T04:28:02.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.204+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f07f413c540 con 0x7f07f406c430 2024-09-16T04:28:02.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.204+0000 7f07f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:02.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.204+0000 7f07f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:02.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.367+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07d4000b90 con 0x7f07e0038e50 2024-09-16T04:28:02.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.369+0000 7f07f0ff9640 1 -- 172.21.15.148:0/822820511 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77985 (unknown 2965378022 0 2825265590) 0x7f07d4000b90 con 0x7f07e0038e50 2024-09-16T04:28:02.371 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:28:02.372 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:28:02.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 >> v1:172.21.15.171:6812/7327273 conn(0x7f07e0038e50 legacy=0x7f07e003b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 10 monclient: shutdown 2024-09-16T04:28:02.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:02.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 >> v1:172.21.15.148:6789/0 conn(0x7f07f406c430 legacy=0x7f07f413be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 shutdown_connections 2024-09-16T04:28:02.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 >> 172.21.15.148:0/822820511 conn(0x7f07f4062630 msgr2=0x7f07f40a3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:02.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 shutdown_connections 2024-09-16T04:28:02.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.371+0000 7f07fa4e6640 1 -- 172.21.15.148:0/822820511 wait complete. 2024-09-16T04:28:02.384 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":897,"stamp":"2024-09-16T04:28:01.549244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197136,"kb_used_data":17608,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566033904,"statfs":{"total":579820584960,"available":579618717696,"internally_reserved":0,"allocated":18030592,"data_stored":15149110,"data_compressed":1198654,"data_compressed_allocated":2351104,"data_compressed_original":6012928,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003016"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":306,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:35.308006+0000","last_change":"2024-09-16T04:11:11.973056+0000","last_active":"2024-09-16T04:24:35.308006+0000","last_peered":"2024-09-16T04:24:35.308006+0000","last_clean":"2024-09-16T04:24:35.308006+0000","last_became_active":"2024-09-16T04:11:11.972654+0000","last_became_peered":"2024-09-16T04:11:11.972654+0000","last_unstale":"2024-09-16T04:24:35.308006+0000","last_undegraded":"2024-09-16T04:24:35.308006+0000","last_fullsized":"2024-09-16T04:24:35.308006+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:33.067826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298089+0000","last_change":"2024-09-16T04:11:11.971214+0000","last_active":"2024-09-16T04:24:11.298089+0000","last_peered":"2024-09-16T04:24:11.298089+0000","last_clean":"2024-09-16T04:24:11.298089+0000","last_became_active":"2024-09-16T04:11:11.970821+0000","last_became_peered":"2024-09-16T04:11:11.970821+0000","last_unstale":"2024-09-16T04:24:11.298089+0000","last_undegraded":"2024-09-16T04:24:11.298089+0000","last_fullsized":"2024-09-16T04:24:11.298089+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:04.704581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":279,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:40.308623+0000","last_change":"2024-09-16T04:11:11.972099+0000","last_active":"2024-09-16T04:24:40.308623+0000","last_peered":"2024-09-16T04:24:40.308623+0000","last_clean":"2024-09-16T04:24:40.308623+0000","last_became_active":"2024-09-16T04:11:11.971647+0000","last_became_peered":"2024-09-16T04:11:11.971647+0000","last_unstale":"2024-09-16T04:24:40.308623+0000","last_undegraded":"2024-09-16T04:24:40.308623+0000","last_fullsized":"2024-09-16T04:24:40.308623+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:24.951965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298065+0000","last_change":"2024-09-16T04:11:11.974362+0000","last_active":"2024-09-16T04:24:11.298065+0000","last_peered":"2024-09-16T04:24:11.298065+0000","last_clean":"2024-09-16T04:24:11.298065+0000","last_became_active":"2024-09-16T04:11:11.974022+0000","last_became_peered":"2024-09-16T04:11:11.974022+0000","last_unstale":"2024-09-16T04:24:11.298065+0000","last_undegraded":"2024-09-16T04:24:11.298065+0000","last_fullsized":"2024-09-16T04:24:11.298065+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:35.453472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:11.304629+0000","last_change":"2024-09-16T04:11:12.969110+0000","last_active":"2024-09-16T04:25:11.304629+0000","last_peered":"2024-09-16T04:25:11.304629+0000","last_clean":"2024-09-16T04:25:11.304629+0000","last_became_active":"2024-09-16T04:11:12.968688+0000","last_became_peered":"2024-09-16T04:11:12.968688+0000","last_unstale":"2024-09-16T04:25:11.304629+0000","last_undegraded":"2024-09-16T04:25:11.304629+0000","last_fullsized":"2024-09-16T04:25:11.304629+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:52.202804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":294,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:40.308632+0000","last_change":"2024-09-16T04:11:11.972073+0000","last_active":"2024-09-16T04:24:40.308632+0000","last_peered":"2024-09-16T04:24:40.308632+0000","last_clean":"2024-09-16T04:24:40.308632+0000","last_became_active":"2024-09-16T04:11:11.971583+0000","last_became_peered":"2024-09-16T04:11:11.971583+0000","last_unstale":"2024-09-16T04:24:40.308632+0000","last_undegraded":"2024-09-16T04:24:40.308632+0000","last_fullsized":"2024-09-16T04:24:40.308632+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:15.678578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395998+0000","last_change":"2024-09-16T04:11:11.975563+0000","last_active":"2024-09-16T04:24:11.395998+0000","last_peered":"2024-09-16T04:24:11.395998+0000","last_clean":"2024-09-16T04:24:11.395998+0000","last_became_active":"2024-09-16T04:11:11.975180+0000","last_became_peered":"2024-09-16T04:11:11.975180+0000","last_unstale":"2024-09-16T04:24:11.395998+0000","last_undegraded":"2024-09-16T04:24:11.395998+0000","last_fullsized":"2024-09-16T04:24:11.395998+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:27.461051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403830+0000","last_change":"2024-09-16T04:11:11.970716+0000","last_active":"2024-09-16T04:25:48.403830+0000","last_peered":"2024-09-16T04:25:48.403830+0000","last_clean":"2024-09-16T04:25:48.403830+0000","last_became_active":"2024-09-16T04:11:11.970247+0000","last_became_peered":"2024-09-16T04:11:11.970247+0000","last_unstale":"2024-09-16T04:25:48.403830+0000","last_undegraded":"2024-09-16T04:25:48.403830+0000","last_fullsized":"2024-09-16T04:25:48.403830+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:37.770332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:16.298525+0000","last_change":"2024-09-16T04:11:11.973578+0000","last_active":"2024-09-16T04:24:16.298525+0000","last_peered":"2024-09-16T04:24:16.298525+0000","last_clean":"2024-09-16T04:24:16.298525+0000","last_became_active":"2024-09-16T04:11:11.973162+0000","last_became_peered":"2024-09-16T04:11:11.973162+0000","last_unstale":"2024-09-16T04:24:16.298525+0000","last_undegraded":"2024-09-16T04:24:16.298525+0000","last_fullsized":"2024-09-16T04:24:16.298525+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:31.864317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:46.399846+0000","last_change":"2024-09-16T04:11:11.975126+0000","last_active":"2024-09-16T04:24:46.399846+0000","last_peered":"2024-09-16T04:24:46.399846+0000","last_clean":"2024-09-16T04:24:46.399846+0000","last_became_active":"2024-09-16T04:11:11.974614+0000","last_became_peered":"2024-09-16T04:11:11.974614+0000","last_unstale":"2024-09-16T04:24:46.399846+0000","last_undegraded":"2024-09-16T04:24:46.399846+0000","last_fullsized":"2024-09-16T04:24:46.399846+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:25:32.801105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":232,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:33.300353+0000","last_change":"2024-09-16T04:11:11.973087+0000","last_active":"2024-09-16T04:24:33.300353+0000","last_peered":"2024-09-16T04:24:33.300353+0000","last_clean":"2024-09-16T04:24:33.300353+0000","last_became_active":"2024-09-16T04:11:11.972700+0000","last_became_peered":"2024-09-16T04:11:11.972700+0000","last_unstale":"2024-09-16T04:24:33.300353+0000","last_undegraded":"2024-09-16T04:24:33.300353+0000","last_fullsized":"2024-09-16T04:24:33.300353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:13.209774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403797+0000","last_change":"2024-09-16T04:11:11.972841+0000","last_active":"2024-09-16T04:25:48.403797+0000","last_peered":"2024-09-16T04:25:48.403797+0000","last_clean":"2024-09-16T04:25:48.403797+0000","last_became_active":"2024-09-16T04:11:11.972431+0000","last_became_peered":"2024-09-16T04:11:11.972431+0000","last_unstale":"2024-09-16T04:25:48.403797+0000","last_undegraded":"2024-09-16T04:25:48.403797+0000","last_fullsized":"2024-09-16T04:25:48.403797+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:31.140262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":179,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403820+0000","last_change":"2024-09-16T04:11:11.972928+0000","last_active":"2024-09-16T04:25:48.403820+0000","last_peered":"2024-09-16T04:25:48.403820+0000","last_clean":"2024-09-16T04:25:48.403820+0000","last_became_active":"2024-09-16T04:11:11.972513+0000","last_became_peered":"2024-09-16T04:11:11.972513+0000","last_unstale":"2024-09-16T04:25:48.403820+0000","last_undegraded":"2024-09-16T04:25:48.403820+0000","last_fullsized":"2024-09-16T04:25:48.403820+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:31.437265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":307,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:35.307984+0000","last_change":"2024-09-16T04:11:11.971270+0000","last_active":"2024-09-16T04:24:35.307984+0000","last_peered":"2024-09-16T04:24:35.307984+0000","last_clean":"2024-09-16T04:24:35.307984+0000","last_became_active":"2024-09-16T04:11:11.970869+0000","last_became_peered":"2024-09-16T04:11:11.970869+0000","last_unstale":"2024-09-16T04:24:35.307984+0000","last_undegraded":"2024-09-16T04:24:35.307984+0000","last_fullsized":"2024-09-16T04:24:35.307984+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:57:26.746389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:50.315983+0000","last_change":"2024-09-16T04:11:12.974381+0000","last_active":"2024-09-16T04:25:50.315983+0000","last_peered":"2024-09-16T04:25:50.315983+0000","last_clean":"2024-09-16T04:25:50.315983+0000","last_became_active":"2024-09-16T04:11:12.974021+0000","last_became_peered":"2024-09-16T04:11:12.974021+0000","last_unstale":"2024-09-16T04:25:50.315983+0000","last_undegraded":"2024-09-16T04:25:50.315983+0000","last_fullsized":"2024-09-16T04:25:50.315983+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:37:30.702246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:13.400389+0000","last_change":"2024-09-16T04:11:11.972888+0000","last_active":"2024-09-16T04:25:13.400389+0000","last_peered":"2024-09-16T04:25:13.400389+0000","last_clean":"2024-09-16T04:25:13.400389+0000","last_became_active":"2024-09-16T04:11:11.972527+0000","last_became_peered":"2024-09-16T04:11:11.972527+0000","last_unstale":"2024-09-16T04:25:13.400389+0000","last_undegraded":"2024-09-16T04:25:13.400389+0000","last_fullsized":"2024-09-16T04:25:13.400389+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:15.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":585,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:36.398725+0000","last_change":"2024-09-16T04:11:11.976349+0000","last_active":"2024-09-16T04:24:36.398725+0000","last_peered":"2024-09-16T04:24:36.398725+0000","last_clean":"2024-09-16T04:24:36.398725+0000","last_became_active":"2024-09-16T04:11:11.975960+0000","last_became_peered":"2024-09-16T04:11:11.975960+0000","last_unstale":"2024-09-16T04:24:36.398725+0000","last_undegraded":"2024-09-16T04:24:36.398725+0000","last_fullsized":"2024-09-16T04:24:36.398725+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:55.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":437,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308206+0000","last_change":"2024-09-16T04:11:11.978921+0000","last_active":"2024-09-16T04:25:46.308206+0000","last_peered":"2024-09-16T04:25:46.308206+0000","last_clean":"2024-09-16T04:25:46.308206+0000","last_became_active":"2024-09-16T04:11:11.971320+0000","last_became_peered":"2024-09-16T04:11:11.971320+0000","last_unstale":"2024-09-16T04:25:46.308206+0000","last_undegraded":"2024-09-16T04:25:46.308206+0000","last_fullsized":"2024-09-16T04:25:46.308206+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:13.147521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:41.399305+0000","last_change":"2024-09-16T04:11:11.975014+0000","last_active":"2024-09-16T04:24:41.399305+0000","last_peered":"2024-09-16T04:24:41.399305+0000","last_clean":"2024-09-16T04:24:41.399305+0000","last_became_active":"2024-09-16T04:11:11.974359+0000","last_became_peered":"2024-09-16T04:11:11.974359+0000","last_unstale":"2024-09-16T04:24:41.399305+0000","last_undegraded":"2024-09-16T04:24:41.399305+0000","last_fullsized":"2024-09-16T04:24:41.399305+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:29.591854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:50.315959+0000","last_change":"2024-09-16T04:11:11.973501+0000","last_active":"2024-09-16T04:25:50.315959+0000","last_peered":"2024-09-16T04:25:50.315959+0000","last_clean":"2024-09-16T04:25:50.315959+0000","last_became_active":"2024-09-16T04:11:11.973107+0000","last_became_peered":"2024-09-16T04:11:11.973107+0000","last_unstale":"2024-09-16T04:25:50.315959+0000","last_undegraded":"2024-09-16T04:25:50.315959+0000","last_fullsized":"2024-09-16T04:25:50.315959+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:14.670234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":448,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308156+0000","last_change":"2024-09-16T04:11:11.984013+0000","last_active":"2024-09-16T04:25:46.308156+0000","last_peered":"2024-09-16T04:25:46.308156+0000","last_clean":"2024-09-16T04:25:46.308156+0000","last_became_active":"2024-09-16T04:11:11.975852+0000","last_became_peered":"2024-09-16T04:11:11.975852+0000","last_unstale":"2024-09-16T04:25:46.308156+0000","last_undegraded":"2024-09-16T04:25:46.308156+0000","last_fullsized":"2024-09-16T04:25:46.308156+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:19.909892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":438,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403840+0000","last_change":"2024-09-16T04:11:12.969208+0000","last_active":"2024-09-16T04:25:48.403840+0000","last_peered":"2024-09-16T04:25:48.403840+0000","last_clean":"2024-09-16T04:25:48.403840+0000","last_became_active":"2024-09-16T04:11:12.968870+0000","last_became_peered":"2024-09-16T04:11:12.968870+0000","last_unstale":"2024-09-16T04:25:48.403840+0000","last_undegraded":"2024-09-16T04:25:48.403840+0000","last_fullsized":"2024-09-16T04:25:48.403840+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:14.254897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.405976+0000","last_change":"2024-09-16T04:11:11.976009+0000","last_active":"2024-09-16T04:25:46.405976+0000","last_peered":"2024-09-16T04:25:46.405976+0000","last_clean":"2024-09-16T04:25:46.405976+0000","last_became_active":"2024-09-16T04:11:11.975676+0000","last_became_peered":"2024-09-16T04:11:11.975676+0000","last_unstale":"2024-09-16T04:25:46.405976+0000","last_undegraded":"2024-09-16T04:25:46.405976+0000","last_fullsized":"2024-09-16T04:25:46.405976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:00:21.147770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:13.394779+0000","last_change":"2024-09-16T04:11:11.977786+0000","last_active":"2024-09-16T04:24:13.394779+0000","last_peered":"2024-09-16T04:24:13.394779+0000","last_clean":"2024-09-16T04:24:13.394779+0000","last_became_active":"2024-09-16T04:11:11.977456+0000","last_became_peered":"2024-09-16T04:11:11.977456+0000","last_unstale":"2024-09-16T04:24:13.394779+0000","last_undegraded":"2024-09-16T04:24:13.394779+0000","last_fullsized":"2024-09-16T04:24:13.394779+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:46.427728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395976+0000","last_change":"2024-09-16T04:11:11.972539+0000","last_active":"2024-09-16T04:24:11.395976+0000","last_peered":"2024-09-16T04:24:11.395976+0000","last_clean":"2024-09-16T04:24:11.395976+0000","last_became_active":"2024-09-16T04:11:11.972183+0000","last_became_peered":"2024-09-16T04:11:11.972183+0000","last_unstale":"2024-09-16T04:24:11.395976+0000","last_undegraded":"2024-09-16T04:24:11.395976+0000","last_fullsized":"2024-09-16T04:24:11.395976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:54.054660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575111,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34224,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337616,"statfs":{"total":96636764160,"available":96601718784,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34228,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337612,"statfs":{"total":96636764160,"available":96601714688,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30068,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341772,"statfs":{"total":96636764160,"available":96605974528,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640518,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30064,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341776,"statfs":{"total":96636764160,"available":96605978624,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640516,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:28:02.385 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:28:02.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.480+0000 7f9df809b640 10 monclient: get_monmap_and_config 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.480+0000 7f9df809b640 10 monclient: build_initial_monmap 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.480+0000 7f9df809b640 10 monclient: monmap: 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:02.481230+0000 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:02.481230+0000 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:02.482 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.480+0000 7f9df809b640 1 Processor -- start 2024-09-16T04:28:02.490 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 1 -- start start 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient: init 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient(hunting): picked mon.noname-b con 0x7f9df009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient(hunting): picked mon.noname-a con 0x7f9df009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient(hunting): picked mon.noname-c con 0x7f9df009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df00a1010 con 0x7f9df009c660 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df009f030 con 0x7f9df009cc30 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df009dc60 con 0x7f9df009bb70 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df809b640 10 monclient(hunting): authenticate will time out at 2103.106587s 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df5e37640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f9df009bb70 0x7f9df009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38588/0 (socket says 172.21.15.148:38588) 2024-09-16T04:28:02.491 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df5636640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9df009c660 0x7f9df0138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48070/0 (socket says 172.21.15.148:48070) 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df5e37640 1 -- 172.21.15.148:0/1619161760 learned_addr learned my addr 172.21.15.148:0/1619161760 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3986029295 0 0) 0x7f9df00a1010 con 0x7f9df009c660 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): my global_id is 174150 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ddc003520 con 0x7f9df009c660 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619806709 0 0) 0x7f9df009dc60 con 0x7f9df009bb70 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 10 monclient(hunting): my global_id is 154400 2024-09-16T04:28:02.492 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.481+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df00a1010 con 0x7f9df009bb70 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648378393 0 0) 0x7f9df009f030 con 0x7f9df009cc30 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient(hunting): my global_id is 164278 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df009dc60 con 0x7f9df009cc30 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 830933852 0 0) 0x7f9df00a1010 con 0x7f9df009bb70 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 >> v1:172.21.15.171:6789/0 conn(0x7f9df009cc30 legacy=0x7f9df013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 >> v1:172.21.15.148:6789/0 conn(0x7f9df009c660 legacy=0x7f9df0138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: _finish_hunting 0 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 monclient: found mon.noname-b 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:02.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df009df40 con 0x7f9df009bb70 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: _finish_auth 0 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: _check_auth_tickets 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9dec001570 con 0x7f9df009bb70 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 5 monclient: authenticate success, global_id 154400 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: dump: 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.494 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dec0017e0 con 0x7f9df009bb70 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 1 -- 172.21.15.148:0/1619161760 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9dec001bf0 con 0x7f9df009bb70 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 4 set_mon_vals no callback set 2024-09-16T04:28:02.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df4e35640 10 monclient: dump: 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 10 monclient: shutdown 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 1 -- 172.21.15.148:0/1619161760 >> v1:172.21.15.148:6790/0 conn(0x7f9df009bb70 legacy=0x7f9df009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 1 -- 172.21.15.148:0/1619161760 shutdown_connections 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 1 -- 172.21.15.148:0/1619161760 >> 172.21.15.148:0/1619161760 conn(0x7f9df00a5640 msgr2=0x7f9df00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.482+0000 7f9df809b640 1 -- 172.21.15.148:0/1619161760 shutdown_connections 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.483+0000 7f9df809b640 1 -- 172.21.15.148:0/1619161760 wait complete. 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.483+0000 7f9df809b640 10 monclient: build_initial_monmap 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.483+0000 7f9df809b640 10 monclient: monmap: 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:02.497 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.483+0000 7f9df809b640 1 Processor -- start 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.483+0000 7f9df809b640 1 -- start start 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.483+0000 7f9df809b640 10 monclient: init 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient(hunting): picked mon.noname-a con 0x7f9df009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient(hunting): picked mon.noname-b con 0x7f9df009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:02.498 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient(hunting): picked mon.noname-c con 0x7f9df009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df01329c0 con 0x7f9df009bb70 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df009b6e0 con 0x7f9df009c660 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df5e37640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9df009bb70 0x7f9df0131ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48084/0 (socket says 172.21.15.148:48084) 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df5e37640 1 -- 172.21.15.148:0/1162491485 learned_addr learned my addr 172.21.15.148:0/1162491485 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df0134c50 con 0x7f9df009cc30 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9df809b640 10 monclient(hunting): authenticate will time out at 2103.109402s 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1867184877 0 0) 0x7f9df01329c0 con 0x7f9df009bb70 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): my global_id is 174153 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dd4003550 con 0x7f9df009bb70 2024-09-16T04:28:02.499 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409712380 0 0) 0x7f9df0134c50 con 0x7f9df009cc30 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): my global_id is 154403 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df01329c0 con 0x7f9df009cc30 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727942322 0 0) 0x7f9df009b6e0 con 0x7f9df009c660 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): my global_id is 164281 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df0134c50 con 0x7f9df009c660 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2333633420 0 0) 0x7f9dd4003550 con 0x7f9df009bb70 2024-09-16T04:28:02.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9df009b6e0 con 0x7f9df009bb70 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9dec001470 con 0x7f9df009bb70 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): dump: 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.501 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3413115436 0 0) 0x7f9df01329c0 con 0x7f9df009cc30 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9dd4003550 con 0x7f9df009cc30 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9de8001bb0 con 0x7f9df009cc30 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 10 monclient(hunting): dump: 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.502 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.484+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1397551235 0 0) 0x7f9df0134c50 con 0x7f9df009c660 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9df01329c0 con 0x7f9df009c660 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9de0001a30 con 0x7f9df009c660 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient(hunting): dump: 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.503 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3111648661 0 0) 0x7f9df009b6e0 con 0x7f9df009bb70 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 >> v1:172.21.15.148:6790/0 conn(0x7f9df009cc30 legacy=0x7f9df01d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 >> v1:172.21.15.171:6789/0 conn(0x7f9df009c660 legacy=0x7f9df01322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 monclient: found mon.a 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:02.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df01335f0 con 0x7f9df009bb70 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: _finish_auth 0 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 5 monclient: authenticate success, global_id 174153 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 10 monclient: _renew_subs 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9df013c8f0 con 0x7f9df009bb70 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 10 monclient: _renew_subs 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9df009ac70 con 0x7f9df009bb70 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9dec0017e0 con 0x7f9df009bb70 2024-09-16T04:28:02.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9dec004d90 con 0x7f9df009bb70 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de7fff640 4 set_mon_vals no callback set 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 10 monclient: dump: 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:02.506 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.485+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9dec004d90 con 0x7f9df009bb70 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.486+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f9dec047880 con 0x7f9df009bb70 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.486+0000 7f9df809b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.486+0000 7f9df809b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.486+0000 7f9df809b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.486+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd8005080 con 0x7f9df009bb70 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.486+0000 7f9df5636640 10 monclient: get_auth_request con 0x7f9dd4039090 auth_method 0 2024-09-16T04:28:02.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.490+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9dec015b40 con 0x7f9df009bb70 2024-09-16T04:28:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.490+0000 7f9de67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:02.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.490+0000 7f9de67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:02.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.657+0000 7f9df809b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:28:02.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.657+0000 7f9df809b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:28:02.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.657+0000 7f9df809b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:02.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.657+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dd8005370 con 0x7f9df009bb70 2024-09-16T04:28:02.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.658+0000 7f9de67fc640 1 -- 172.21.15.148:0/1162491485 <== mon.0 v1:172.21.15.148:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v257) ==== 74+0+20599 (unknown 2895976921 0 1630569510) 0x7f9dec015440 con 0x7f9df009bb70 2024-09-16T04:28:02.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.658+0000 7f9de67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:28:02.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.658+0000 7f9de67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:28:02.660 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:28:02.660 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":257,"fsid":"bc50e21b-589f-47d0-85d5-a865de7d0d68","created":"2024-09-16T04:06:21.146269+0000","modified":"2024-09-16T04:21:12.567797+0000","last_up_change":"2024-09-16T04:06:32.301387+0000","last_in_change":"2024-09-16T04:21:07.566576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":181,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":15,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T04:06:31.297815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T04:06:37.900873+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-16T04:09:55.435529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"barset","create_time":"2024-09-16T04:09:56.509287+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30d6186b-ce3c-4385-a2da-04f1171e31c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6808","nonce":985869260}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6809","nonce":985869260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6811","nonce":985869260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6810","nonce":985869260}]},"public_addr":"172.21.15.148:6808/985869260","cluster_addr":"172.21.15.148:6809/985869260","heartbeat_back_addr":"172.21.15.148:6811/985869260","heartbeat_front_addr":"172.21.15.148:6810/985869260","state":["exists","up"]},{"osd":1,"uuid":"d04305dd-0099-4412-a6cd-7aae325a600d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6800","nonce":3044422561}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6801","nonce":3044422561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6803","nonce":3044422561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6802","nonce":3044422561}]},"public_addr":"172.21.15.148:6800/3044422561","cluster_addr":"172.21.15.148:6801/3044422561","heartbeat_back_addr":"172.21.15.148:6803/3044422561","heartbeat_front_addr":"172.21.15.148:6802/3044422561","state":["exists","up"]},{"osd":2,"uuid":"c64f2f7b-204c-4e36-8960-822df4fbc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6804","nonce":262588452}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6805","nonce":262588452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6807","nonce":262588452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.148:6806","nonce":262588452}]},"public_addr":"172.21.15.148:6804/262588452","cluster_addr":"172.21.15.148:6805/262588452","heartbeat_back_addr":"172.21.15.148:6807/262588452","heartbeat_front_addr":"172.21.15.148:6806/262588452","state":["exists","up"]},{"osd":3,"uuid":"030ce3ae-cb8f-410f-8718-c6b1d9f10134","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6804","nonce":772155526}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6805","nonce":772155526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6807","nonce":772155526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6806","nonce":772155526}]},"public_addr":"172.21.15.171:6804/772155526","cluster_addr":"172.21.15.171:6805/772155526","heartbeat_back_addr":"172.21.15.171:6807/772155526","heartbeat_front_addr":"172.21.15.171:6806/772155526","state":["exists","up"]},{"osd":4,"uuid":"f82ddc46-d03e-415c-8b1d-0f3c98df29c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6808","nonce":4274747803}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6809","nonce":4274747803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6811","nonce":4274747803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6810","nonce":4274747803}]},"public_addr":"172.21.15.171:6808/4274747803","cluster_addr":"172.21.15.171:6809/4274747803","heartbeat_back_addr":"172.21.15.171:6811/4274747803","heartbeat_front_addr":"172.21.15.171:6810/4274747803","state":["exists","up"]},{"osd":5,"uuid":"18ff6471-ce6d-4c23-ac7c-415157ed598e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6800","nonce":466524840}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6801","nonce":466524840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6803","nonce":466524840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.171:6802","nonce":466524840}]},"public_addr":"172.21.15.171:6800/466524840","cluster_addr":"172.21.15.171:6801/466524840","heartbeat_back_addr":"172.21.15.171:6803/466524840","heartbeat_front_addr":"172.21.15.171:6802/466524840","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"6df951c8-ca70-48bb-b32a-2b7a6ec9adff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.060468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:28.867753+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140301+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.140492+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.136223+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T04:06:29.080093+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 >> v1:172.21.15.171:6812/7327273 conn(0x7f9dd4039090 legacy=0x7f9dd403b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 10 monclient: shutdown 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 >> v1:172.21.15.148:6789/0 conn(0x7f9df009bb70 legacy=0x7f9df0131ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 shutdown_connections 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 >> 172.21.15.148:0/1162491485 conn(0x7f9df00a5640 msgr2=0x7f9df00a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 shutdown_connections 2024-09-16T04:28:02.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:02.660+0000 7f9df809b640 1 -- 172.21.15.148:0/1162491485 wait complete. 2024-09-16T04:28:03.674 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:28:03.674 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:28:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.787+0000 7fe5cbef1640 10 monclient: get_monmap_and_config 2024-09-16T04:28:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.787+0000 7fe5cbef1640 10 monclient: build_initial_monmap 2024-09-16T04:28:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.787+0000 7fe5cbef1640 10 monclient: monmap: 2024-09-16T04:28:03.788 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:03.787972+0000 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:03.787972+0000 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:03.789 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.796 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.787+0000 7fe5cbef1640 1 Processor -- start 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 1 -- start start 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient: init 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c40a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c40a4580 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c413d3c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c40a0590 con 0x7fe5c40a4580 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c40a1270 con 0x7fe5c413d3c0 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c40a4fd0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:03.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5cbef1640 10 monclient(hunting): authenticate will time out at 2104.413585s 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c9c8d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe5c40a3a90 0x7fe5c40a3e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38610/0 (socket says 172.21.15.148:38610) 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c948c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe5c40a4580 0x7fe5c413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48092/0 (socket says 172.21.15.148:48092) 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c9c8d640 1 -- 172.21.15.148:0/2695560384 learned_addr learned my addr 172.21.15.148:0/2695560384 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344235773 0 0) 0x7fe5c40a4fd0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): my global_id is 154406 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5b0003520 con 0x7fe5c40a3a90 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256887812 0 0) 0x7fe5b4001110 con 0x7fe5c40a4580 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 10 monclient(hunting): my global_id is 174156 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.788+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5c40a4fd0 con 0x7fe5c40a4580 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 675398425 0 0) 0x7fe5c40a4fd0 con 0x7fe5c40a4580 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 >> v1:172.21.15.148:6790/0 conn(0x7fe5c40a3a90 legacy=0x7fe5c40a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 >> v1:172.21.15.171:6789/0 conn(0x7fe5c413d3c0 legacy=0x7fe5c413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: _finish_hunting 0 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 monclient: found mon.noname-a 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c406e170 con 0x7fe5c40a4580 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: _finish_auth 0 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: _check_auth_tickets 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:03.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5b4001760 con 0x7fe5c40a4580 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 5 monclient: authenticate success, global_id 174156 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: dump: 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.800 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5b40019c0 con 0x7fe5c40a4580 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 1 -- 172.21.15.148:0/2695560384 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5b40047e0 con 0x7fe5c40a4580 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 4 set_mon_vals no callback set 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5c8c8b640 10 monclient: dump: 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.801 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 10 monclient: shutdown 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 1 -- 172.21.15.148:0/2695560384 >> v1:172.21.15.148:6789/0 conn(0x7fe5c40a4580 legacy=0x7fe5c413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 1 -- 172.21.15.148:0/2695560384 shutdown_connections 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.789+0000 7fe5cbef1640 1 -- 172.21.15.148:0/2695560384 >> 172.21.15.148:0/2695560384 conn(0x7fe5c40a9a40 msgr2=0x7fe5c40a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 1 -- 172.21.15.148:0/2695560384 shutdown_connections 2024-09-16T04:28:03.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 1 -- 172.21.15.148:0/2695560384 wait complete. 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 10 monclient: build_initial_monmap 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 10 monclient: monmap: 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 1 Processor -- start 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 1 -- start start 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.790+0000 7fe5cbef1640 10 monclient: init 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c40a3a90 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c40a4580 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c413d3c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c409c1e0 con 0x7fe5c413d3c0 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c413ced0 con 0x7fe5c40a4580 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c40a0280 con 0x7fe5c40a3a90 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5cbef1640 10 monclient(hunting): authenticate will time out at 2104.416427s 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5c9c8d640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe5c40a3a90 0x7fe5c409b3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38626/0 (socket says 172.21.15.148:38626) 2024-09-16T04:28:03.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5c9c8d640 1 -- 172.21.15.148:0/1868338657 learned_addr learned my addr 172.21.15.148:0/1868338657 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147058064 0 0) 0x7fe5c40a0280 con 0x7fe5c40a3a90 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): my global_id is 154409 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a8003520 con 0x7fe5c40a3a90 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2843460356 0 0) 0x7fe5c409c1e0 con 0x7fe5c413d3c0 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): my global_id is 174159 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5c40a0280 con 0x7fe5c413d3c0 2024-09-16T04:28:03.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2574641430 0 0) 0x7fe5c413ced0 con 0x7fe5c40a4580 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): my global_id is 164287 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5c409c1e0 con 0x7fe5c40a4580 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2845295979 0 0) 0x7fe5a8003520 con 0x7fe5c40a3a90 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5c413ced0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5c0001a30 con 0x7fe5c40a3a90 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.806 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 335121272 0 0) 0x7fe5c40a0280 con 0x7fe5c413d3c0 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5a8003520 con 0x7fe5c413d3c0 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5bc001af0 con 0x7fe5c413d3c0 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:03.807 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.791+0000 7fe5ba7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 107864326 0 0) 0x7fe5c413ced0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.808 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 >> v1:172.21.15.171:6789/0 conn(0x7fe5c40a4580 legacy=0x7fe5c409bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 >> v1:172.21.15.148:6789/0 conn(0x7fe5c413d3c0 legacy=0x7fe5c41d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 monclient: found mon.c 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c409f190 con 0x7fe5c40a3a90 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: _finish_auth 0 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 5 monclient: authenticate success, global_id 154409 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 10 monclient: _renew_subs 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c409ebd0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.809 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 10 monclient: _renew_subs 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5c409cf60 con 0x7fe5c40a3a90 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5c0001ce0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe5c0004c30 con 0x7fe5c40a3a90 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5bbfff640 4 set_mon_vals no callback set 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 10 monclient: dump: 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.810 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.792+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe5c000fec0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5cbef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe5ac000f80 con 0x7fe5c40a3a90 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fe5c0047840 con 0x7fe5c40a3a90 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6808/985869260 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5a80469c0 con 0x7fe5a8042cf0 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (unknown 1324813924 0 0) 0x7fe5c0013bc0 con 0x7fe5c40a3a90 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5ba7fc640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T04:28:03.811 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5c948c640 10 monclient: get_auth_request con 0x7fe5a8038ea0 auth_method 0 2024-09-16T04:28:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.793+0000 7fe5ca48e640 10 monclient: get_auth_request con 0x7fe5a8042cf0 auth_method 0 2024-09-16T04:28:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.794+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== osd.0 v1:172.21.15.148:6808/985869260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5a80469c0 con 0x7fe5a8042cf0 2024-09-16T04:28:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.806+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 --> v1:172.21.15.148:6808/985869260 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe5ac002cb0 con 0x7fe5a8042cf0 2024-09-16T04:28:03.812 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.806+0000 7fe5ba7fc640 1 -- 172.21.15.148:0/1868338657 <== osd.0 v1:172.21.15.148:6808/985869260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fe5ac002cb0 con 0x7fe5a8042cf0 2024-09-16T04:28:03.813 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:28:03.813 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:28:03.813 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 >> v1:172.21.15.148:6808/985869260 conn(0x7fe5a8042cf0 legacy=0x7fe5a8045150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 >> v1:172.21.15.171:6812/7327273 conn(0x7fe5a8038ea0 legacy=0x7fe5a803b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 10 monclient: shutdown 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 >> v1:172.21.15.148:6790/0 conn(0x7fe5c40a3a90 legacy=0x7fe5c409b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5ca48e640 1 -- 172.21.15.148:0/1868338657 reap_dead start 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 shutdown_connections 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 >> 172.21.15.148:0/1868338657 conn(0x7fe5c40a9a40 msgr2=0x7fe5c413fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:03.814 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 shutdown_connections 2024-09-16T04:28:03.815 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.807+0000 7fe5cbef1640 1 -- 172.21.15.148:0/1868338657 wait complete. 2024-09-16T04:28:03.819 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:28:03.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.916+0000 7f6944511640 10 monclient: get_monmap_and_config 2024-09-16T04:28:03.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.916+0000 7f6944511640 10 monclient: build_initial_monmap 2024-09-16T04:28:03.917 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.916+0000 7f6944511640 10 monclient: monmap: 2024-09-16T04:28:03.917 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:03.916982+0000 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:03.916982+0000 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:03.918 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.926 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.916+0000 7f6944511640 1 Processor -- start 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.916+0000 7f6944511640 1 -- start start 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.916+0000 7f6944511640 10 monclient: init 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient(hunting): picked mon.noname-b con 0x7f693c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient(hunting): picked mon.noname-c con 0x7f693c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient(hunting): picked mon.noname-a con 0x7f693c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c0a0e00 con 0x7f693c13d060 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c0a10e0 con 0x7f693c0a1b10 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c0a4970 con 0x7f693c0a2410 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f6944511640 10 monclient(hunting): authenticate will time out at 2104.542417s 2024-09-16T04:28:03.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69422ad640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f693c0a2410 0x7f693c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38628/0 (socket says 172.21.15.148:38628) 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69422ad640 1 -- 172.21.15.148:0/973763102 learned_addr learned my addr 172.21.15.148:0/973763102 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2024469564 0 0) 0x7f693c0a4970 con 0x7f693c0a2410 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): my global_id is 154412 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6928003520 con 0x7f693c0a2410 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1664741250 0 0) 0x7f693c0a0e00 con 0x7f693c13d060 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 10 monclient(hunting): my global_id is 174162 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f693c0a4970 con 0x7f693c13d060 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.917+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2267124983 0 0) 0x7f693c0a4970 con 0x7f693c13d060 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 >> v1:172.21.15.148:6790/0 conn(0x7f693c0a2410 legacy=0x7f693c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 >> v1:172.21.15.171:6789/0 conn(0x7f693c0a1b10 legacy=0x7f693c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: _finish_hunting 0 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 monclient: found mon.noname-a 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c06dfd0 con 0x7f693c13d060 2024-09-16T04:28:03.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: _finish_auth 0 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: _check_auth_tickets 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6934001990 con 0x7f693c13d060 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 5 monclient: authenticate success, global_id 174162 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: dump: 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6934001bf0 con 0x7f693c13d060 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 1 -- 172.21.15.148:0/973763102 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6934002000 con 0x7f693c13d060 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 4 set_mon_vals no callback set 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f69412ab640 10 monclient: dump: 2024-09-16T04:28:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 10 monclient: shutdown 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 1 -- 172.21.15.148:0/973763102 >> v1:172.21.15.148:6789/0 conn(0x7f693c13d060 legacy=0x7f693c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 1 -- 172.21.15.148:0/973763102 shutdown_connections 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 1 -- 172.21.15.148:0/973763102 >> 172.21.15.148:0/973763102 conn(0x7f693c0a97b0 msgr2=0x7f693c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 1 -- 172.21.15.148:0/973763102 shutdown_connections 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.918+0000 7f6944511640 1 -- 172.21.15.148:0/973763102 wait complete. 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.919+0000 7f6944511640 10 monclient: build_initial_monmap 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.919+0000 7f6944511640 10 monclient: monmap: 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:03.933 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.919+0000 7f6944511640 1 Processor -- start 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.919+0000 7f6944511640 1 -- start start 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.919+0000 7f6944511640 10 monclient: init 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient(hunting): picked mon.noname-a con 0x7f693c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient(hunting): picked mon.noname-c con 0x7f693c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient(hunting): picked mon.noname-b con 0x7f693c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c138600 con 0x7f693c0a1b10 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c09f6f0 con 0x7f693c13d060 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f693c139140 con 0x7f693c0a2410 2024-09-16T04:28:03.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient(hunting): authenticate will time out at 2104.545087s 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f69422ad640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f693c0a1b10 0x7f693c136aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48108/0 (socket says 172.21.15.148:48108) 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6941aac640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f693c0a2410 0x7f693c1371b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38636/0 (socket says 172.21.15.148:38636) 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f69422ad640 1 -- 172.21.15.148:0/3592181116 learned_addr learned my addr 172.21.15.148:0/3592181116 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398408786 0 0) 0x7f693c138600 con 0x7f693c0a1b10 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): my global_id is 174165 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6920003520 con 0x7f693c0a1b10 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092646693 0 0) 0x7f693c139140 con 0x7f693c0a2410 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): my global_id is 154415 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f693c138600 con 0x7f693c0a2410 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150875352 0 0) 0x7f693c09f6f0 con 0x7f693c13d060 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): my global_id is 164293 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f693c139140 con 0x7f693c13d060 2024-09-16T04:28:03.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2694965821 0 0) 0x7f693c138600 con 0x7f693c0a2410 2024-09-16T04:28:03.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f693c09f6f0 con 0x7f693c0a2410 2024-09-16T04:28:03.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 401822556 0 0) 0x7f6920003520 con 0x7f693c0a1b10 2024-09-16T04:28:03.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f693c138600 con 0x7f693c0a1b10 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f692c0018b0 con 0x7f693c0a2410 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): dump: 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.938 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.939 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.939 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.939 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.939 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f6938001360 con 0x7f693c0a1b10 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient(hunting): dump: 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.940 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1209548175 0 0) 0x7f693c09f6f0 con 0x7f693c0a2410 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 >> v1:172.21.15.171:6789/0 conn(0x7f693c13d060 legacy=0x7f693c1d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 >> v1:172.21.15.148:6789/0 conn(0x7f693c0a1b10 legacy=0x7f693c136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 monclient: found mon.c 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c139420 con 0x7f693c0a2410 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient: _finish_auth 0 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:03.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 5 monclient: authenticate success, global_id 154415 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient: _renew_subs 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f693c13c5f0 con 0x7f693c0a2410 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient: _renew_subs 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f693c1378c0 con 0x7f693c0a2410 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f692c003610 con 0x7f693c0a2410 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f692c003960 con 0x7f693c0a2410 2024-09-16T04:28:03.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6932ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.920+0000 7f6940aaa640 4 set_mon_vals no callback set 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6932ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6932ffd640 10 monclient: dump: 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:03.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f692c003b80 con 0x7f693c0a2410 2024-09-16T04:28:03.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6944511640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:03.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6944511640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:03.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6944511640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:03.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6924005080 con 0x7f693c0a2410 2024-09-16T04:28:03.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.921+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f692c0478c0 con 0x7f693c0a2410 2024-09-16T04:28:03.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.922+0000 7f69422ad640 10 monclient: get_auth_request con 0x7f69200390f0 auth_method 0 2024-09-16T04:28:03.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.926+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f692c015c00 con 0x7f693c0a2410 2024-09-16T04:28:03.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.926+0000 7f6932ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:03.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:03.926+0000 7f6932ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:04.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.099+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f6924000b90 con 0x7f69200390f0 2024-09-16T04:28:04.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.099+0000 7f6932ffd640 1 -- 172.21.15.148:0/3592181116 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f6924000b90 con 0x7f69200390f0 2024-09-16T04:28:04.101 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:28:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.101+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 >> v1:172.21.15.171:6812/7327273 conn(0x7f69200390f0 legacy=0x7f692003b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.101+0000 7f6944511640 10 monclient: shutdown 2024-09-16T04:28:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.101+0000 7f6944511640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.101+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 >> v1:172.21.15.148:6790/0 conn(0x7f693c0a2410 legacy=0x7f693c1371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.102+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 shutdown_connections 2024-09-16T04:28:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.102+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 >> 172.21.15.148:0/3592181116 conn(0x7f693c0a97b0 msgr2=0x7f693c13f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.102+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 shutdown_connections 2024-09-16T04:28:04.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.102+0000 7f6944511640 1 -- 172.21.15.148:0/3592181116 wait complete. 2024-09-16T04:28:04.114 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:28:04.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:28:04.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.214+0000 7f8f435a6640 10 monclient: get_monmap_and_config 2024-09-16T04:28:04.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.214+0000 7f8f435a6640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.214+0000 7f8f435a6640 10 monclient: monmap: 2024-09-16T04:28:04.216 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:04.215210+0000 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:04.215210+0000 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:04.217 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.214+0000 7f8f435a6640 1 Processor -- start 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 1 -- start start 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient: init 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c0a0e00 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c0a10e0 con 0x7f8f3c13d060 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c0a4970 con 0x7f8f3c0a2410 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f435a6640 10 monclient(hunting): authenticate will time out at 2104.840558s 2024-09-16T04:28:04.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f41342640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f8f3c0a2410 0x7f8f3c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38646/0 (socket says 172.21.15.148:38646) 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f41342640 1 -- 172.21.15.148:0/3762872862 learned_addr learned my addr 172.21.15.148:0/3762872862 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277064497 0 0) 0x7f8f3c0a4970 con 0x7f8f3c0a2410 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): my global_id is 154418 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f28003520 con 0x7f8f3c0a2410 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891959434 0 0) 0x7f8f3c0a0e00 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 10 monclient(hunting): my global_id is 174168 2024-09-16T04:28:04.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.215+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f3c0a4970 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3686199472 0 0) 0x7f8f28003520 con 0x7f8f3c0a2410 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 >> v1:172.21.15.171:6789/0 conn(0x7f8f3c13d060 legacy=0x7f8f3c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 >> v1:172.21.15.148:6789/0 conn(0x7f8f3c0a1b10 legacy=0x7f8f3c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 monclient: found mon.noname-b 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c06dfd0 con 0x7f8f3c0a2410 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f38001610 con 0x7f8f3c0a2410 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: dump: 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 5 monclient: authenticate success, global_id 154418 2024-09-16T04:28:04.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f380018b0 con 0x7f8f3c0a2410 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 1 -- 172.21.15.148:0/3762872862 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f38001cc0 con 0x7f8f3c0a2410 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 4 set_mon_vals no callback set 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f33fff640 10 monclient: dump: 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.229 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 10 monclient: shutdown 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 1 -- 172.21.15.148:0/3762872862 >> v1:172.21.15.148:6790/0 conn(0x7f8f3c0a2410 legacy=0x7f8f3c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 1 -- 172.21.15.148:0/3762872862 shutdown_connections 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 1 -- 172.21.15.148:0/3762872862 >> 172.21.15.148:0/3762872862 conn(0x7f8f3c0a97b0 msgr2=0x7f8f3c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 1 -- 172.21.15.148:0/3762872862 shutdown_connections 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.216+0000 7f8f435a6640 1 -- 172.21.15.148:0/3762872862 wait complete. 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.217+0000 7f8f435a6640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.217+0000 7f8f435a6640 10 monclient: monmap: 2024-09-16T04:28:04.230 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.217+0000 7f8f435a6640 1 Processor -- start 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.217+0000 7f8f435a6640 1 -- start start 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.217+0000 7f8f435a6640 10 monclient: init 2024-09-16T04:28:04.231 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3c0a1b10 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c0a2410 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c13d060 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c137970 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c137c50 con 0x7f8f3c0a2410 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c09d9c0 con 0x7f8f3c13d060 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f435a6640 10 monclient(hunting): authenticate will time out at 2104.843446s 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f41342640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f8f3c0a1b10 0x7f8f3c09cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48122/0 (socket says 172.21.15.148:48122) 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f41342640 1 -- 172.21.15.148:0/1925620400 learned_addr learned my addr 172.21.15.148:0/1925620400 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125637674 0 0) 0x7f8f3c09d9c0 con 0x7f8f3c13d060 2024-09-16T04:28:04.232 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 154421 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f20003550 con 0x7f8f3c13d060 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2713132679 0 0) 0x7f8f3c137970 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 174171 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f3c09d9c0 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667501126 0 0) 0x7f8f3c137c50 con 0x7f8f3c0a2410 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.233 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 164299 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f3c137970 con 0x7f8f3c0a2410 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 240281172 0 0) 0x7f8f20003550 con 0x7f8f3c13d060 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f3c137c50 con 0x7f8f3c13d060 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.218+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 370341431 0 0) 0x7f8f3c09d9c0 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f20003550 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f34001b70 con 0x7f8f3c13d060 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.234 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f38001510 con 0x7f8f3c0a1b10 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.235 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4232643460 0 0) 0x7f8f3c137970 con 0x7f8f3c0a2410 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8f3c09d9c0 con 0x7f8f3c0a2410 2024-09-16T04:28:04.236 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f2c001a10 con 0x7f8f3c0a2410 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.237 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 366826285 0 0) 0x7f8f3c137c50 con 0x7f8f3c13d060 2024-09-16T04:28:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 >> v1:172.21.15.171:6789/0 conn(0x7f8f3c0a2410 legacy=0x7f8f3c09d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 >> v1:172.21.15.148:6789/0 conn(0x7f8f3c0a1b10 legacy=0x7f8f3c09cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.238 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 monclient: found mon.c 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c140340 con 0x7f8f3c13d060 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 5 monclient: authenticate success, global_id 154421 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 10 monclient: _renew_subs 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8f3c13cc60 con 0x7f8f3c13d060 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8f34003970 con 0x7f8f3c13d060 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f8f34003cc0 con 0x7f8f3c13d060 2024-09-16T04:28:04.239 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f31ffb640 10 monclient: dump: 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 10 monclient: _renew_subs 2024-09-16T04:28:04.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f337fe640 4 set_mon_vals no callback set 2024-09-16T04:28:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.219+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8f3c137690 con 0x7f8f3c13d060 2024-09-16T04:28:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f435a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.241 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8f3c09f180 con 0x7f8f3c13d060 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f8f3400ef50 con 0x7f8f3c13d060 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f8f34046850 con 0x7f8f3c13d060 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6800/3044422561 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f20046c10 con 0x7f8f20042fe0 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (unknown 1324813924 0 0) 0x7f8f34046d00 con 0x7f8f3c13d060 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f31ffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f41342640 10 monclient: get_auth_request con 0x7f8f200391b0 auth_method 0 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.220+0000 7f8f40b41640 10 monclient: get_auth_request con 0x7f8f20042fe0 auth_method 0 2024-09-16T04:28:04.242 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.221+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== osd.1 v1:172.21.15.148:6800/3044422561 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8f20046c10 con 0x7f8f20042fe0 2024-09-16T04:28:04.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.233+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 --> v1:172.21.15.148:6800/3044422561 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8f3c13c6e0 con 0x7f8f20042fe0 2024-09-16T04:28:04.243 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.233+0000 7f8f31ffb640 1 -- 172.21.15.148:0/1925620400 <== osd.1 v1:172.21.15.148:6800/3044422561 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f8f3c13c6e0 con 0x7f8f20042fe0 2024-09-16T04:28:04.244 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:28:04.244 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:28:04.244 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:28:04.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.233+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 >> v1:172.21.15.148:6800/3044422561 conn(0x7f8f20042fe0 legacy=0x7f8f20045440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.246 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 >> v1:172.21.15.171:6812/7327273 conn(0x7f8f200391b0 legacy=0x7f8f2003b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 10 monclient: shutdown 2024-09-16T04:28:04.250 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 >> v1:172.21.15.148:6790/0 conn(0x7f8f3c13d060 legacy=0x7f8f3c1d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f41b43640 1 -- 172.21.15.148:0/1925620400 reap_dead start 2024-09-16T04:28:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 shutdown_connections 2024-09-16T04:28:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 >> 172.21.15.148:0/1925620400 conn(0x7f8f3c0a97b0 msgr2=0x7f8f3c135860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 shutdown_connections 2024-09-16T04:28:04.251 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.234+0000 7f8f435a6640 1 -- 172.21.15.148:0/1925620400 wait complete. 2024-09-16T04:28:04.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:28:04.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.349+0000 7efea4f86640 10 monclient: get_monmap_and_config 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.349+0000 7efea4f86640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.349+0000 7efea4f86640 10 monclient: monmap: 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:04.350163+0000 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:04.350163+0000 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.351 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:04.352 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:04.352 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.349+0000 7efea4f86640 1 Processor -- start 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.349+0000 7efea4f86640 1 -- start start 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.349+0000 7efea4f86640 10 monclient: init 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient(hunting): picked mon.noname-c con 0x7efea009bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient(hunting): picked mon.noname-b con 0x7efea009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient(hunting): picked mon.noname-a con 0x7efea009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea00a1010 con 0x7efea009cc30 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea009f030 con 0x7efea009bb70 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea009dc60 con 0x7efea009c660 2024-09-16T04:28:04.362 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efea4f86640 10 monclient(hunting): authenticate will time out at 2104.975494s 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9dd9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efea009c660 0x7efea0138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38670/0 (socket says 172.21.15.148:38670) 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9dd9b640 1 -- 172.21.15.148:0/935418197 learned_addr learned my addr 172.21.15.148:0/935418197 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9ed9d640 1 --1- 172.21.15.148:0/935418197 >> v1:172.21.15.148:6789/0 conn(0x7efea009cc30 0x7efea013c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48132/0 (socket says 172.21.15.148:48132) 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108486046 0 0) 0x7efea009dc60 con 0x7efea009c660 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): my global_id is 154424 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe88003520 con 0x7efea009c660 2024-09-16T04:28:04.363 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824596882 0 0) 0x7efea00a1010 con 0x7efea009cc30 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 10 monclient(hunting): my global_id is 174174 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.350+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea009dc60 con 0x7efea009cc30 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1454436355 0 0) 0x7efe88003520 con 0x7efea009c660 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 >> v1:172.21.15.171:6789/0 conn(0x7efea009bb70 legacy=0x7efea009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 >> v1:172.21.15.148:6789/0 conn(0x7efea009cc30 legacy=0x7efea013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 monclient: found mon.noname-b 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.364 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea009df40 con 0x7efea009c660 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe94001610 con 0x7efea009c660 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 5 monclient: authenticate success, global_id 154424 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: dump: 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.365 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe94001880 con 0x7efea009c660 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 1 -- 172.21.15.148:0/935418197 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe94001c50 con 0x7efea009c660 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:04.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 4 set_mon_vals no callback set 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efe9d59a640 10 monclient: dump: 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 10 monclient: shutdown 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 1 -- 172.21.15.148:0/935418197 >> v1:172.21.15.148:6790/0 conn(0x7efea009c660 legacy=0x7efea0138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 1 -- 172.21.15.148:0/935418197 shutdown_connections 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 1 -- 172.21.15.148:0/935418197 >> 172.21.15.148:0/935418197 conn(0x7efea00a5640 msgr2=0x7efea00a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 1 -- 172.21.15.148:0/935418197 shutdown_connections 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.351+0000 7efea4f86640 1 -- 172.21.15.148:0/935418197 wait complete. 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.352+0000 7efea4f86640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.352+0000 7efea4f86640 10 monclient: monmap: 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:04.368 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.352+0000 7efea4f86640 1 Processor -- start 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.352+0000 7efea4f86640 1 -- start start 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.352+0000 7efea4f86640 10 monclient: init 2024-09-16T04:28:04.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient(hunting): picked mon.noname-c con 0x7efea009bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient(hunting): picked mon.noname-b con 0x7efea009c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient(hunting): picked mon.noname-a con 0x7efea009cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea0121250 con 0x7efea009cc30 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea00a41f0 con 0x7efea009c660 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea0120c90 con 0x7efea009bb70 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efea4f86640 10 monclient(hunting): authenticate will time out at 2104.978480s 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe9e59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efea009bb70 0x7efea0133890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38680/0 (socket says 172.21.15.148:38680) 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe9e59c640 1 -- 172.21.15.148:0/1235294546 learned_addr learned my addr 172.21.15.148:0/1235294546 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520632871 0 0) 0x7efea0120c90 con 0x7efea009bb70 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): my global_id is 154427 2024-09-16T04:28:04.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe80003520 con 0x7efea009bb70 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189151997 0 0) 0x7efea0121250 con 0x7efea009cc30 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): my global_id is 174177 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea0120c90 con 0x7efea009cc30 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2584955971 0 0) 0x7efea00a41f0 con 0x7efea009c660 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 10 monclient(hunting): my global_id is 164305 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea0121250 con 0x7efea009c660 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1483312000 0 0) 0x7efe80003520 con 0x7efea009bb70 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.353+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efea00a41f0 con 0x7efea009bb70 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe90001970 con 0x7efea009bb70 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:04.373 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): dump: 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:04.374 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3820883805 0 0) 0x7efea0120c90 con 0x7efea009cc30 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efe80003520 con 0x7efea009cc30 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe98001c50 con 0x7efea009cc30 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): dump: 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.375 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 62379019 0 0) 0x7efea0121250 con 0x7efea009c660 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efea0120c90 con 0x7efea009c660 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe94001bd0 con 0x7efea009c660 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient(hunting): dump: 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.376 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 516431071 0 0) 0x7efea00a41f0 con 0x7efea009bb70 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 >> v1:172.21.15.171:6789/0 conn(0x7efea009c660 legacy=0x7efea01cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 >> v1:172.21.15.148:6789/0 conn(0x7efea009cc30 legacy=0x7efea01cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 monclient: found mon.c 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea0120f70 con 0x7efea009bb70 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 5 monclient: authenticate success, global_id 154427 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 10 monclient: _renew_subs 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efea011f840 con 0x7efea009bb70 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 10 monclient: _renew_subs 2024-09-16T04:28:04.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7efea01c7250 con 0x7efea009bb70 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efe900038c0 con 0x7efea009bb70 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efe90003af0 con 0x7efea009bb70 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe9cd99640 4 set_mon_vals no callback set 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.354+0000 7efe8effd640 10 monclient: dump: 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.379 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7efe90003af0 con 0x7efea009bb70 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efea4f86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efea4f86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efea4f86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe84005080 con 0x7efea009bb70 2024-09-16T04:28:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7efe90047740 con 0x7efea009bb70 2024-09-16T04:28:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.355+0000 7efe9dd9b640 10 monclient: get_auth_request con 0x7efe80039160 auth_method 0 2024-09-16T04:28:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.359+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7efe900196d0 con 0x7efea009bb70 2024-09-16T04:28:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.359+0000 7efe8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:04.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.359+0000 7efe8effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:04.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.527+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7efe84000b90 con 0x7efe80039160 2024-09-16T04:28:04.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.528+0000 7efe8effd640 1 -- 172.21.15.148:0/1235294546 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7efe84000b90 con 0x7efe80039160 2024-09-16T04:28:04.529 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:28:04.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 >> v1:172.21.15.171:6812/7327273 conn(0x7efe80039160 legacy=0x7efe8003b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 10 monclient: shutdown 2024-09-16T04:28:04.532 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 >> v1:172.21.15.148:6790/0 conn(0x7efea009bb70 legacy=0x7efea0133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 shutdown_connections 2024-09-16T04:28:04.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 >> 172.21.15.148:0/1235294546 conn(0x7efea00a5640 msgr2=0x7efea01b9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 shutdown_connections 2024-09-16T04:28:04.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.530+0000 7efea4f86640 1 -- 172.21.15.148:0/1235294546 wait complete. 2024-09-16T04:28:04.542 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:28:04.543 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:28:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.643+0000 7f0593349640 10 monclient: get_monmap_and_config 2024-09-16T04:28:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.643+0000 7f0593349640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.643+0000 7f0593349640 10 monclient: monmap: 2024-09-16T04:28:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.645 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:04.644409+0000 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:04.644409+0000 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 1 Processor -- start 2024-09-16T04:28:04.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 1 -- start start 2024-09-16T04:28:04.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient: init 2024-09-16T04:28:04.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:04.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient(hunting): picked mon.noname-b con 0x7f058c0a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient(hunting): picked mon.noname-a con 0x7f058c09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient(hunting): picked mon.noname-c con 0x7f058c09b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c09e040 con 0x7f058c09b170 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c09d660 con 0x7f058c09b5c0 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c09c0d0 con 0x7f058c0a0e90 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f0593349640 10 monclient(hunting): authenticate will time out at 2105.269736s 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f05910e5640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f058c0a0e90 0x7f058c09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38686/0 (socket says 172.21.15.148:38686) 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f05910e5640 1 -- 172.21.15.148:0/3499714661 learned_addr learned my addr 172.21.15.148:0/3499714661 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.644+0000 7f05908e4640 1 --1- 172.21.15.148:0/3499714661 >> v1:172.21.15.148:6789/0 conn(0x7f058c09b170 0x7f058c1383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48152/0 (socket says 172.21.15.148:48152) 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531811554 0 0) 0x7f057c001110 con 0x7f058c09b170 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): my global_id is 174180 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0578003520 con 0x7f058c09b170 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245100696 0 0) 0x7f058c09c0d0 con 0x7f058c0a0e90 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): my global_id is 154430 2024-09-16T04:28:04.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f057c001110 con 0x7f058c0a0e90 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2645970630 0 0) 0x7f0584001490 con 0x7f058c09b5c0 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient(hunting): my global_id is 164308 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f058c09c0d0 con 0x7f058c09b5c0 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2803911575 0 0) 0x7f0578003520 con 0x7f058c09b170 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 >> v1:172.21.15.148:6790/0 conn(0x7f058c0a0e90 legacy=0x7f058c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 >> v1:172.21.15.171:6789/0 conn(0x7f058c09b5c0 legacy=0x7f058c13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 monclient: found mon.noname-a 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:04.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c09d380 con 0x7f058c09b170 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f057c0017e0 con 0x7f058c09b170 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 5 monclient: authenticate success, global_id 174180 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: dump: 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.662 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f057c001a90 con 0x7f058c09b170 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 1 -- 172.21.15.148:0/3499714661 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f057c0047e0 con 0x7f058c09b170 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 4 set_mon_vals no callback set 2024-09-16T04:28:04.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0583fff640 10 monclient: dump: 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 10 monclient: shutdown 2024-09-16T04:28:04.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 1 -- 172.21.15.148:0/3499714661 >> v1:172.21.15.148:6789/0 conn(0x7f058c09b170 legacy=0x7f058c1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.645+0000 7f0593349640 1 -- 172.21.15.148:0/3499714661 shutdown_connections 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 1 -- 172.21.15.148:0/3499714661 >> 172.21.15.148:0/3499714661 conn(0x7f058c0a5640 msgr2=0x7f058c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 1 -- 172.21.15.148:0/3499714661 shutdown_connections 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 1 -- 172.21.15.148:0/3499714661 wait complete. 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 10 monclient: monmap: 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.665 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 1 Processor -- start 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 1 -- start start 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.646+0000 7f0593349640 10 monclient: init 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient(hunting): picked mon.noname-a con 0x7f058c09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient(hunting): picked mon.noname-c con 0x7f058c09b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient(hunting): picked mon.noname-b con 0x7f058c0a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c1c6220 con 0x7f058c09b170 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c1aa9b0 con 0x7f058c0a0e90 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c1ad470 con 0x7f058c09b5c0 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f05910e5640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f058c09b170 0x7f058c1c5b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48164/0 (socket says 172.21.15.148:48164) 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0593349640 10 monclient(hunting): authenticate will time out at 2105.272527s 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f05910e5640 1 -- 172.21.15.148:0/1607972454 learned_addr learned my addr 172.21.15.148:0/1607972454 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072388163 0 0) 0x7f058c1ad470 con 0x7f058c09b5c0 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): my global_id is 154433 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0570003520 con 0x7f058c09b5c0 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380843835 0 0) 0x7f058c1c6220 con 0x7f058c09b170 2024-09-16T04:28:04.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): my global_id is 174183 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f058c1ad470 con 0x7f058c09b170 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279147895 0 0) 0x7f058c1aa9b0 con 0x7f058c0a0e90 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): my global_id is 164311 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f058c1c6220 con 0x7f058c0a0e90 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1733847912 0 0) 0x7f0570003520 con 0x7f058c09b5c0 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f058c1aa9b0 con 0x7f058c09b5c0 2024-09-16T04:28:04.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f057c001600 con 0x7f058c09b5c0 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.647+0000 7f0581ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 50472768 0 0) 0x7f058c1ad470 con 0x7f058c09b170 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0570003520 con 0x7f058c09b170 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f05880012b0 con 0x7f058c09b170 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.670 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3278098933 0 0) 0x7f058c1c6220 con 0x7f058c0a0e90 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f058c1ad470 con 0x7f058c0a0e90 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f0584001e00 con 0x7f058c0a0e90 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.671 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2560295528 0 0) 0x7f058c1aa9b0 con 0x7f058c09b5c0 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 >> v1:172.21.15.171:6789/0 conn(0x7f058c0a0e90 legacy=0x7f058c1d4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 >> v1:172.21.15.148:6789/0 conn(0x7f058c09b170 legacy=0x7f058c1c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 monclient: found mon.c 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c1ab1a0 con 0x7f058c09b5c0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 5 monclient: authenticate success, global_id 154433 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 10 monclient: _renew_subs 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f058c1ad190 con 0x7f058c09b5c0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f057c004d10 con 0x7f058c09b5c0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f057c005060 con 0x7f058c09b5c0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0581ffb640 10 monclient: dump: 2024-09-16T04:28:04.673 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 10 monclient: _renew_subs 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f058c1c8580 con 0x7f058c09b5c0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.648+0000 7f05837fe640 4 set_mon_vals no callback set 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f057c0102f0 con 0x7f058c09b5c0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0593349640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f058c072b60 con 0x7f058c09b5c0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f057c0478d0 con 0x7f058c09b5c0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6804/262588452 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f057c001a90 con 0x7f0570042f30 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (unknown 1324813924 0 0) 0x7f057c015200 con 0x7f058c09b5c0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f0581ffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f05910e5640 10 monclient: get_auth_request con 0x7f0570039100 auth_method 0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.649+0000 7f05918e6640 10 monclient: get_auth_request con 0x7f0570042f30 auth_method 0 2024-09-16T04:28:04.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.650+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== osd.2 v1:172.21.15.148:6804/262588452 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f057c001a90 con 0x7f0570042f30 2024-09-16T04:28:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.662+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 --> v1:172.21.15.148:6804/262588452 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f058c071da0 con 0x7f0570042f30 2024-09-16T04:28:04.676 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.662+0000 7f0581ffb640 1 -- 172.21.15.148:0/1607972454 <== osd.2 v1:172.21.15.148:6804/262588452 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f058c071da0 con 0x7f0570042f30 2024-09-16T04:28:04.677 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:28:04.677 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:28:04.677 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:28:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 >> v1:172.21.15.148:6804/262588452 conn(0x7f0570042f30 legacy=0x7f0570045390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 >> v1:172.21.15.171:6812/7327273 conn(0x7f0570039100 legacy=0x7f057003b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 10 monclient: shutdown 2024-09-16T04:28:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.678 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 >> v1:172.21.15.148:6790/0 conn(0x7f058c09b5c0 legacy=0x7f058c1d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f05918e6640 1 -- 172.21.15.148:0/1607972454 reap_dead start 2024-09-16T04:28:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 shutdown_connections 2024-09-16T04:28:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 >> 172.21.15.148:0/1607972454 conn(0x7f058c0a5640 msgr2=0x7f058c071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 shutdown_connections 2024-09-16T04:28:04.679 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.663+0000 7f0593349640 1 -- 172.21.15.148:0/1607972454 wait complete. 2024-09-16T04:28:04.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:28:04.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.776+0000 7f9b4965a640 10 monclient: get_monmap_and_config 2024-09-16T04:28:04.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.776+0000 7f9b4965a640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.776+0000 7f9b4965a640 10 monclient: monmap: 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:04.777892+0000 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:04.777892+0000 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:04.779 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.777+0000 7f9b4965a640 1 Processor -- start 2024-09-16T04:28:04.786 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.777+0000 7f9b4965a640 1 -- start start 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.777+0000 7f9b4965a640 10 monclient: init 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient(hunting): picked mon.noname-a con 0x7f9b440a0e90 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient(hunting): picked mon.noname-b con 0x7f9b4409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient(hunting): picked mon.noname-c con 0x7f9b4409b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4409e040 con 0x7f9b440a0e90 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4409d660 con 0x7f9b4409b5c0 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4409c0d0 con 0x7f9b4409b170 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient(hunting): authenticate will time out at 2105.403185s 2024-09-16T04:28:04.787 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b427fc640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f9b4409b170 0x7f9b441383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38690/0 (socket says 172.21.15.148:38690) 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b427fc640 1 -- 172.21.15.148:0/2992959935 learned_addr learned my addr 172.21.15.148:0/2992959935 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3275401854 0 0) 0x7f9b4409c0d0 con 0x7f9b4409b170 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): my global_id is 154436 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b2c003520 con 0x7f9b4409b170 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755232563 0 0) 0x7f9b34001040 con 0x7f9b440a0e90 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): my global_id is 174186 2024-09-16T04:28:04.788 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b4409c0d0 con 0x7f9b440a0e90 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4083943622 0 0) 0x7f9b3c001490 con 0x7f9b4409b5c0 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient(hunting): my global_id is 164314 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b34001040 con 0x7f9b4409b5c0 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3802759620 0 0) 0x7f9b4409c0d0 con 0x7f9b440a0e90 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 >> v1:172.21.15.148:6790/0 conn(0x7f9b4409b170 legacy=0x7f9b441383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 >> v1:172.21.15.171:6789/0 conn(0x7f9b4409b5c0 legacy=0x7f9b4413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 monclient: found mon.noname-a 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.789 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4409d380 con 0x7f9b440a0e90 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9b340016f0 con 0x7f9b440a0e90 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: dump: 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.790 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 5 monclient: authenticate success, global_id 174186 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b34001960 con 0x7f9b440a0e90 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 1 -- 172.21.15.148:0/2992959935 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9b34003610 con 0x7f9b440a0e90 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b4965a640 4 set_mon_vals no callback set 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:04.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.778+0000 7f9b41ffb640 10 monclient: dump: 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 10 monclient: shutdown 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 1 -- 172.21.15.148:0/2992959935 >> v1:172.21.15.148:6789/0 conn(0x7f9b440a0e90 legacy=0x7f9b4409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.792 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 1 -- 172.21.15.148:0/2992959935 shutdown_connections 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 1 -- 172.21.15.148:0/2992959935 >> 172.21.15.148:0/2992959935 conn(0x7f9b440a5640 msgr2=0x7f9b440a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 1 -- 172.21.15.148:0/2992959935 shutdown_connections 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.779+0000 7f9b4965a640 1 -- 172.21.15.148:0/2992959935 wait complete. 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient: build_initial_monmap 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient: monmap: 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:04.793 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:04.796 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 1 Processor -- start 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 1 -- start start 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient: init 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient(hunting): picked mon.noname-a con 0x7f9b4409b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.780+0000 7f9b4965a640 10 monclient(hunting): picked mon.noname-c con 0x7f9b4409b5c0 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient(hunting): picked mon.noname-b con 0x7f9b440a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b441a6d10 con 0x7f9b4409b170 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b441a8fe0 con 0x7f9b440a0e90 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b441a8740 con 0x7f9b4409b5c0 2024-09-16T04:28:04.797 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b42ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f9b4409b170 0x7f9b441c4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48178/0 (socket says 172.21.15.148:48178) 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b42ffd640 1 -- 172.21.15.148:0/125954508 learned_addr learned my addr 172.21.15.148:0/125954508 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient(hunting): authenticate will time out at 2105.406094s 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3117136974 0 0) 0x7f9b441a6d10 con 0x7f9b4409b170 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): my global_id is 174189 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b30003520 con 0x7f9b4409b170 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288018599 0 0) 0x7f9b441a8740 con 0x7f9b4409b5c0 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.798 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): my global_id is 154439 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b441a6d10 con 0x7f9b4409b5c0 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005779298 0 0) 0x7f9b441a8fe0 con 0x7f9b440a0e90 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): my global_id is 164317 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b441a8740 con 0x7f9b440a0e90 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1052886171 0 0) 0x7f9b30003520 con 0x7f9b4409b170 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b441a8fe0 con 0x7f9b4409b170 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9b340016b0 con 0x7f9b4409b170 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:04.799 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): dump: 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3981938045 0 0) 0x7f9b441a6d10 con 0x7f9b4409b5c0 2024-09-16T04:28:04.800 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b30003520 con 0x7f9b4409b5c0 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9b38001190 con 0x7f9b4409b5c0 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient(hunting): dump: 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.801 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 702391787 0 0) 0x7f9b441a8fe0 con 0x7f9b4409b170 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 >> v1:172.21.15.148:6790/0 conn(0x7f9b4409b5c0 legacy=0x7f9b441cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 >> v1:172.21.15.171:6789/0 conn(0x7f9b440a0e90 legacy=0x7f9b441d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 monclient: found mon.a 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b44072ac0 con 0x7f9b4409b170 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 5 monclient: authenticate success, global_id 174189 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient: _renew_subs 2024-09-16T04:28:04.802 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b441c1cd0 con 0x7f9b4409b170 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient: _renew_subs 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b441c4ea0 con 0x7f9b4409b170 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9b34001960 con 0x7f9b4409b170 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f9b34003d70 con 0x7f9b4409b170 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b417fa640 4 set_mon_vals no callback set 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.781+0000 7f9b2b7fe640 10 monclient: dump: 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:04.803 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.782+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f9b34003f90 con 0x7f9b4409b170 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.782+0000 7f9b4965a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.782+0000 7f9b4965a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.782+0000 7f9b4965a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.782+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f9b34047b90 con 0x7f9b4409b170 2024-09-16T04:28:04.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.782+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b24005080 con 0x7f9b4409b170 2024-09-16T04:28:04.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.783+0000 7f9b427fc640 10 monclient: get_auth_request con 0x7f9b30038ed0 auth_method 0 2024-09-16T04:28:04.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.787+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f9b4409e040 con 0x7f9b4409b170 2024-09-16T04:28:04.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.787+0000 7f9b2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:04.805 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.787+0000 7f9b2b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:04.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.952+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f9b24000b90 con 0x7f9b30038ed0 2024-09-16T04:28:04.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.952+0000 7f9b2b7fe640 1 -- 172.21.15.148:0/125954508 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f9b24000b90 con 0x7f9b30038ed0 2024-09-16T04:28:04.954 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:28:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 >> v1:172.21.15.171:6812/7327273 conn(0x7f9b30038ed0 legacy=0x7f9b3003b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 10 monclient: shutdown 2024-09-16T04:28:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:04.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 >> v1:172.21.15.148:6789/0 conn(0x7f9b4409b170 legacy=0x7f9b441c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:04.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 shutdown_connections 2024-09-16T04:28:04.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 >> 172.21.15.148:0/125954508 conn(0x7f9b440a5640 msgr2=0x7f9b440a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:04.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 shutdown_connections 2024-09-16T04:28:04.957 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:04.954+0000 7f9b4965a640 1 -- 172.21.15.148:0/125954508 wait complete. 2024-09-16T04:28:04.968 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:28:04.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:28:05.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.069+0000 7fc5cc949640 10 monclient: get_monmap_and_config 2024-09-16T04:28:05.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.070+0000 7fc5cc949640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.070+0000 7fc5cc949640 10 monclient: monmap: 2024-09-16T04:28:05.071 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.071 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:05.070932+0000 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:05.070932+0000 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:05.072 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.070+0000 7fc5cc949640 1 Processor -- start 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.070+0000 7fc5cc949640 1 -- start start 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.070+0000 7fc5cc949640 10 monclient: init 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c809c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.080 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c809cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c80a1010 con 0x7fc5c809c660 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c809f030 con 0x7fc5c809cc30 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c809dc60 con 0x7fc5c809bb70 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient(hunting): authenticate will time out at 2105.696165s 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c659c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc5c809bb70 0x7fc5c809bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38702/0 (socket says 172.21.15.148:38702) 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5bdd9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fc5c809c660 0x7fc5c8138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48190/0 (socket says 172.21.15.148:48190) 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c659c640 1 -- 172.21.15.148:0/308969664 learned_addr learned my addr 172.21.15.148:0/308969664 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 76580871 0 0) 0x7fc5c80a1010 con 0x7fc5c809c660 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.081 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): my global_id is 174192 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b4003520 con 0x7fc5c809c660 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382161282 0 0) 0x7fc5c809dc60 con 0x7fc5c809bb70 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): my global_id is 154442 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c80a1010 con 0x7fc5c809bb70 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788077769 0 0) 0x7fc5c809f030 con 0x7fc5c809cc30 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient(hunting): my global_id is 164320 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c809dc60 con 0x7fc5c809cc30 2024-09-16T04:28:05.082 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2509844874 0 0) 0x7fc5b4003520 con 0x7fc5c809c660 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 >> v1:172.21.15.148:6790/0 conn(0x7fc5c809bb70 legacy=0x7fc5c809bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 >> v1:172.21.15.171:6789/0 conn(0x7fc5c809cc30 legacy=0x7fc5c813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c809df40 con 0x7fc5c809c660 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5b0001610 con 0x7fc5c809c660 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:05.083 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: dump: 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 5 monclient: authenticate success, global_id 174192 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.084 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5b0001840 con 0x7fc5c809c660 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 1 -- 172.21.15.148:0/308969664 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5b0001c50 con 0x7fc5c809c660 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5cc949640 4 set_mon_vals no callback set 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.071+0000 7fc5c5d9b640 10 monclient: dump: 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.085 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 10 monclient: shutdown 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 1 -- 172.21.15.148:0/308969664 >> v1:172.21.15.148:6789/0 conn(0x7fc5c809c660 legacy=0x7fc5c8138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 1 -- 172.21.15.148:0/308969664 shutdown_connections 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 1 -- 172.21.15.148:0/308969664 >> 172.21.15.148:0/308969664 conn(0x7fc5c80a5640 msgr2=0x7fc5c80a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 1 -- 172.21.15.148:0/308969664 shutdown_connections 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.072+0000 7fc5cc949640 1 -- 172.21.15.148:0/308969664 wait complete. 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.073+0000 7fc5cc949640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.073+0000 7fc5cc949640 10 monclient: monmap: 2024-09-16T04:28:05.086 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.073+0000 7fc5cc949640 1 Processor -- start 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.073+0000 7fc5cc949640 1 -- start start 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.073+0000 7fc5cc949640 10 monclient: init 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.073+0000 7fc5cc949640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.087 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c809bb70 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c809c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c809cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c8120340 con 0x7fc5c809cc30 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c80740a0 con 0x7fc5c809c660 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c811edd0 con 0x7fc5c809bb70 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient(hunting): authenticate will time out at 2105.699104s 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5c659c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fc5c809bb70 0x7fc5c8131970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38716/0 (socket says 172.21.15.148:38716) 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5c659c640 1 -- 172.21.15.148:0/12399102 learned_addr learned my addr 172.21.15.148:0/12399102 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1049613798 0 0) 0x7fc5c8120340 con 0x7fc5c809cc30 2024-09-16T04:28:05.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): my global_id is 174195 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5a8003520 con 0x7fc5c809cc30 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247903966 0 0) 0x7fc5c811edd0 con 0x7fc5c809bb70 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.089 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): my global_id is 154445 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c8120340 con 0x7fc5c809bb70 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332433488 0 0) 0x7fc5c80740a0 con 0x7fc5c809c660 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): my global_id is 164323 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5c811edd0 con 0x7fc5c809c660 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 196109954 0 0) 0x7fc5c8120340 con 0x7fc5c809bb70 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5c80740a0 con 0x7fc5c809bb70 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4128871376 0 0) 0x7fc5a8003520 con 0x7fc5c809cc30 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5c8120340 con 0x7fc5c809cc30 2024-09-16T04:28:05.090 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5b80011b0 con 0x7fc5c809bb70 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): dump: 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5c0001bb0 con 0x7fc5c809cc30 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): dump: 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.092 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1442875786 0 0) 0x7fc5c811edd0 con 0x7fc5c809c660 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5a8003520 con 0x7fc5c809c660 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5b0001c50 con 0x7fc5c809c660 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient(hunting): dump: 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.093 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 406805468 0 0) 0x7fc5c8120340 con 0x7fc5c809cc30 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 >> v1:172.21.15.148:6790/0 conn(0x7fc5c809bb70 legacy=0x7fc5c8131970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 >> v1:172.21.15.171:6789/0 conn(0x7fc5c809c660 legacy=0x7fc5c81cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 monclient: found mon.a 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c811e050 con 0x7fc5c809cc30 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 5 monclient: authenticate success, global_id 174195 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient: _renew_subs 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c8120060 con 0x7fc5c809cc30 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient: _renew_subs 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5c81c77d0 con 0x7fc5c809cc30 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.074+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5c0003990 con 0x7fc5c809cc30 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fc5c0003ce0 con 0x7fc5c809cc30 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.095 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 10 monclient: dump: 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5c559a640 4 set_mon_vals no callback set 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5cc949640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.148:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fc5ac000f80 con 0x7fc5c809cc30 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fc5c000ef70 con 0x7fc5c809cc30 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.075+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fc5c0046770 con 0x7fc5c809cc30 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.076+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.171:6804/772155526 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5a8046ba0 con 0x7fc5a8042f70 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.076+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (unknown 1324813924 0 0) 0x7fc5c0046be0 con 0x7fc5c809cc30 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.076+0000 7fc5bf7fe640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.076+0000 7fc5c659c640 10 monclient: get_auth_request con 0x7fc5a8039120 auth_method 0 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.076+0000 7fc5bdd9b640 10 monclient: get_auth_request con 0x7fc5a8042f70 auth_method 0 2024-09-16T04:28:05.097 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.077+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== osd.3 v1:172.21.15.171:6804/772155526 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc5a8046ba0 con 0x7fc5a8042f70 2024-09-16T04:28:05.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.089+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 --> v1:172.21.15.171:6804/772155526 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc5ac002cb0 con 0x7fc5a8042f70 2024-09-16T04:28:05.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5bf7fe640 1 -- 172.21.15.148:0/12399102 <== osd.3 v1:172.21.15.171:6804/772155526 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fc5ac002cb0 con 0x7fc5a8042f70 2024-09-16T04:28:05.098 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:28:05.098 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:28:05.098 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:28:05.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 >> v1:172.21.15.171:6804/772155526 conn(0x7fc5a8042f70 legacy=0x7fc5a80453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 >> v1:172.21.15.171:6812/7327273 conn(0x7fc5a8039120 legacy=0x7fc5a803b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 10 monclient: shutdown 2024-09-16T04:28:05.099 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 >> v1:172.21.15.148:6789/0 conn(0x7fc5c809cc30 legacy=0x7fc5c81cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5c6d9d640 1 -- 172.21.15.148:0/12399102 reap_dead start 2024-09-16T04:28:05.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 shutdown_connections 2024-09-16T04:28:05.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 >> 172.21.15.148:0/12399102 conn(0x7fc5c80a5640 msgr2=0x7fc5c81b9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 shutdown_connections 2024-09-16T04:28:05.100 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.090+0000 7fc5cc949640 1 -- 172.21.15.148:0/12399102 wait complete. 2024-09-16T04:28:05.102 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:28:05.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.199+0000 7efd0b637640 10 monclient: get_monmap_and_config 2024-09-16T04:28:05.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.199+0000 7efd0b637640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.204 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.199+0000 7efd0b637640 10 monclient: monmap: 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:05.200589+0000 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:05.200589+0000 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:05.205 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 1 Processor -- start 2024-09-16T04:28:05.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 1 -- start start 2024-09-16T04:28:05.212 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 10 monclient: init 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 10 monclient(hunting): picked mon.noname-a con 0x7efd0409bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 10 monclient(hunting): picked mon.noname-b con 0x7efd0409c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 10 monclient(hunting): picked mon.noname-c con 0x7efd0409cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd040a1010 con 0x7efd0409bb70 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0409f030 con 0x7efd0409cc30 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.200+0000 7efd0b637640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0409dc60 con 0x7efd0409c660 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 10 monclient(hunting): authenticate will time out at 2105.826040s 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd08bd2640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7efd0409c660 0x7efd04138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38724/0 (socket says 172.21.15.148:38724) 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd08bd2640 1 -- 172.21.15.148:0/2487951962 learned_addr learned my addr 172.21.15.148:0/2487951962 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.213 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd093d3640 1 --1- 172.21.15.148:0/2487951962 >> v1:172.21.15.148:6789/0 conn(0x7efd0409bb70 0x7efd0409bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48208/0 (socket says 172.21.15.148:48208) 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2560919274 0 0) 0x7efd0409dc60 con 0x7efd0409c660 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): my global_id is 154448 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcf0003520 con 0x7efd0409c660 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3398079465 0 0) 0x7efd040a1010 con 0x7efd0409bb70 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient(hunting): my global_id is 174198 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd0409dc60 con 0x7efd0409bb70 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2771205615 0 0) 0x7efcf0003520 con 0x7efd0409c660 2024-09-16T04:28:05.214 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 >> v1:172.21.15.171:6789/0 conn(0x7efd0409cc30 legacy=0x7efd0413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 >> v1:172.21.15.148:6789/0 conn(0x7efd0409bb70 legacy=0x7efd0409bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 monclient: found mon.noname-b 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0409df40 con 0x7efd0409c660 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcf4001610 con 0x7efd0409c660 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 5 monclient: authenticate success, global_id 154448 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.215 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: dump: 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efcf4001870 con 0x7efd0409c660 2024-09-16T04:28:05.216 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 1 -- 172.21.15.148:0/2487951962 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcf4001bc0 con 0x7efd0409c660 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efd0b637640 4 set_mon_vals no callback set 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.201+0000 7efcfbfff640 10 monclient: dump: 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.217 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 10 monclient: shutdown 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 1 -- 172.21.15.148:0/2487951962 >> v1:172.21.15.148:6790/0 conn(0x7efd0409c660 legacy=0x7efd04138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 1 -- 172.21.15.148:0/2487951962 shutdown_connections 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 1 -- 172.21.15.148:0/2487951962 >> 172.21.15.148:0/2487951962 conn(0x7efd040a5640 msgr2=0x7efd040a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 1 -- 172.21.15.148:0/2487951962 shutdown_connections 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 1 -- 172.21.15.148:0/2487951962 wait complete. 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.202+0000 7efd0b637640 10 monclient: monmap: 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.218 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 1 Processor -- start 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 1 -- start start 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient: init 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.219 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient(hunting): picked mon.noname-b con 0x7efd0409bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient(hunting): picked mon.noname-a con 0x7efd0409c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient(hunting): picked mon.noname-c con 0x7efd0409cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0413ba50 con 0x7efd0409c660 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd04134ae0 con 0x7efd0409bb70 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd04132a60 con 0x7efd0409cc30 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd0b637640 10 monclient(hunting): authenticate will time out at 2105.828712s 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd08bd2640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efd0409c660 0x7efd04132350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48212/0 (socket says 172.21.15.148:48212) 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efd08bd2640 1 -- 172.21.15.148:0/1210188231 learned_addr learned my addr 172.21.15.148:0/1210188231 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516319747 0 0) 0x7efd0413ba50 con 0x7efd0409c660 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.220 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): my global_id is 174201 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efce8003520 con 0x7efd0409c660 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729097746 0 0) 0x7efd04132a60 con 0x7efd0409cc30 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.203+0000 7efcf9ffb640 10 monclient(hunting): my global_id is 154451 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd0413ba50 con 0x7efd0409cc30 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 778666394 0 0) 0x7efce8003520 con 0x7efd0409c660 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd04132a60 con 0x7efd0409c660 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcf40015d0 con 0x7efd0409c660 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.221 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 226811168 0 0) 0x7efd0413ba50 con 0x7efd0409cc30 2024-09-16T04:28:05.222 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efce8003520 con 0x7efd0409cc30 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcfc001770 con 0x7efd0409cc30 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.223 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414821011 0 0) 0x7efd04134ae0 con 0x7efd0409bb70 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient(hunting): my global_id is 164329 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd0413ba50 con 0x7efd0409bb70 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2725719168 0 0) 0x7efd04132a60 con 0x7efd0409c660 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 >> v1:172.21.15.148:6790/0 conn(0x7efd0409cc30 legacy=0x7efd041d1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 >> v1:172.21.15.171:6789/0 conn(0x7efd0409bb70 legacy=0x7efd04131c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 monclient: found mon.a 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.224 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd041333c0 con 0x7efd0409c660 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 5 monclient: authenticate success, global_id 174201 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 10 monclient: _renew_subs 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd04134dc0 con 0x7efd0409c660 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efcf4004a90 con 0x7efd0409c660 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7efcf4004e60 con 0x7efd0409c660 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcf9ffb640 10 monclient: dump: 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 10 monclient: _renew_subs 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efcfb7fe640 4 set_mon_vals no callback set 2024-09-16T04:28:05.226 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.204+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efd04133140 con 0x7efd0409c660 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7efcf4010310 con 0x7efd0409c660 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7efcf40479b0 con 0x7efd0409c660 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efd0b637640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efd0b637640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efd0b637640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd04073960 con 0x7efd0409c660 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.205+0000 7efd093d3640 10 monclient: get_auth_request con 0x7efce80390c0 auth_method 0 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.210+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7efd04073960 con 0x7efd0409c660 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.210+0000 7efcf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:05.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.210+0000 7efcf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:05.377 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.375+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7efd0406bc20 con 0x7efce80390c0 2024-09-16T04:28:05.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.376+0000 7efcf9ffb640 1 -- 172.21.15.148:0/1210188231 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7efd0406bc20 con 0x7efce80390c0 2024-09-16T04:28:05.378 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 >> v1:172.21.15.171:6812/7327273 conn(0x7efce80390c0 legacy=0x7efce803b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 10 monclient: shutdown 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 >> v1:172.21.15.148:6789/0 conn(0x7efd0409c660 legacy=0x7efd04132350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 shutdown_connections 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 >> 172.21.15.148:0/1210188231 conn(0x7efd040a5640 msgr2=0x7efd040a38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.380 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 shutdown_connections 2024-09-16T04:28:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.378+0000 7efd0b637640 1 -- 172.21.15.148:0/1210188231 wait complete. 2024-09-16T04:28:05.392 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:28:05.392 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:28:05.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.492+0000 7faaab52c640 10 monclient: get_monmap_and_config 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.492+0000 7faaab52c640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.492+0000 7faaab52c640 10 monclient: monmap: 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:05.493239+0000 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:05.493239+0000 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:05.494 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:05.495 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.503 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.492+0000 7faaab52c640 1 Processor -- start 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 1 -- start start 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient: init 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient(hunting): picked mon.noname-c con 0x7faaa40a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient(hunting): picked mon.noname-b con 0x7faaa409b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient(hunting): picked mon.noname-a con 0x7faaa409b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa409e040 con 0x7faaa409b5c0 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa409d660 con 0x7faaa40a0e90 2024-09-16T04:28:05.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa409c0d0 con 0x7faaa409b170 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaab52c640 10 monclient(hunting): authenticate will time out at 2106.118594s 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaa9ac9640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7faaa409b5c0 0x7faaa413bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48214/0 (socket says 172.21.15.148:48214) 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faaa9ac9640 1 -- 172.21.15.148:0/84416159 learned_addr learned my addr 172.21.15.148:0/84416159 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450941524 0 0) 0x7faa9c001490 con 0x7faaa409b5c0 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 10 monclient(hunting): my global_id is 174204 2024-09-16T04:28:05.505 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa90003520 con 0x7faaa409b5c0 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.493+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369924536 0 0) 0x7faaa409c0d0 con 0x7faaa409b170 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient(hunting): my global_id is 154454 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa9c001490 con 0x7faaa409b170 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1237445967 0 0) 0x7faa90003520 con 0x7faaa409b5c0 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 >> v1:172.21.15.148:6790/0 conn(0x7faaa409b170 legacy=0x7faaa41383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 >> v1:172.21.15.171:6789/0 conn(0x7faaa40a0e90 legacy=0x7faaa409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.506 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 monclient: found mon.noname-a 2024-09-16T04:28:05.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa409d380 con 0x7faaa409b5c0 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa9c001b60 con 0x7faaa409b5c0 2024-09-16T04:28:05.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 5 monclient: authenticate success, global_id 174204 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: dump: 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.509 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa9c001d50 con 0x7faaa409b5c0 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 1 -- 172.21.15.148:0/84416159 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa9c004c20 con 0x7faaa409b5c0 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 4 set_mon_vals no callback set 2024-09-16T04:28:05.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faa9bfff640 10 monclient: dump: 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.511 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 10 monclient: shutdown 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 1 -- 172.21.15.148:0/84416159 >> v1:172.21.15.148:6789/0 conn(0x7faaa409b5c0 legacy=0x7faaa413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 1 -- 172.21.15.148:0/84416159 shutdown_connections 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 1 -- 172.21.15.148:0/84416159 >> 172.21.15.148:0/84416159 conn(0x7faaa40a5640 msgr2=0x7faaa40a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 1 -- 172.21.15.148:0/84416159 shutdown_connections 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.494+0000 7faaab52c640 1 -- 172.21.15.148:0/84416159 wait complete. 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.495+0000 7faaab52c640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.495+0000 7faaab52c640 10 monclient: monmap: 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:05.513 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.495+0000 7faaab52c640 1 Processor -- start 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.495+0000 7faaab52c640 1 -- start start 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.495+0000 7faaab52c640 10 monclient: init 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient(hunting): picked mon.noname-a con 0x7faaa409b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient(hunting): picked mon.noname-b con 0x7faaa409b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient(hunting): picked mon.noname-c con 0x7faaa40a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa41cc930 con 0x7faaa409b170 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa41ace40 con 0x7faaa409b5c0 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa41ad120 con 0x7faaa40a0e90 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaa92c8640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7faaa409b170 0x7faaa41bd740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48218/0 (socket says 172.21.15.148:48218) 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaa92c8640 1 -- 172.21.15.148:0/3143076261 learned_addr learned my addr 172.21.15.148:0/3143076261 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faaab52c640 10 monclient(hunting): authenticate will time out at 2106.121454s 2024-09-16T04:28:05.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373439405 0 0) 0x7faaa41ad120 con 0x7faaa40a0e90 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): my global_id is 154457 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa88003520 con 0x7faaa40a0e90 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609493434 0 0) 0x7faaa41cc930 con 0x7faaa409b170 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): my global_id is 174207 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaa41ad120 con 0x7faaa409b170 2024-09-16T04:28:05.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736598534 0 0) 0x7faaa41ace40 con 0x7faaa409b5c0 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): my global_id is 164335 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaa41cc930 con 0x7faaa409b5c0 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4170487794 0 0) 0x7faa88003520 con 0x7faaa40a0e90 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faaa41ace40 con 0x7faaa40a0e90 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa9c001b20 con 0x7faaa40a0e90 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.517 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3932940052 0 0) 0x7faaa41ad120 con 0x7faaa409b170 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa88003520 con 0x7faaa409b170 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faaa0001a00 con 0x7faaa409b170 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.496+0000 7faa99ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2402537282 0 0) 0x7faaa41cc930 con 0x7faaa409b5c0 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faaa41ad120 con 0x7faaa409b5c0 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa940013d0 con 0x7faaa409b5c0 2024-09-16T04:28:05.519 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2071880322 0 0) 0x7faaa41ace40 con 0x7faaa40a0e90 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 >> v1:172.21.15.171:6789/0 conn(0x7faaa409b5c0 legacy=0x7faaa41d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 >> v1:172.21.15.148:6789/0 conn(0x7faaa409b170 legacy=0x7faaa41bd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 monclient: found mon.c 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa40740a0 con 0x7faaa40a0e90 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 5 monclient: authenticate success, global_id 154457 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 10 monclient: _renew_subs 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faaa41acb60 con 0x7faaa40a0e90 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa9c004f70 con 0x7faaa40a0e90 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7faa9c0052c0 con 0x7faaa40a0e90 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:05.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 10 monclient: dump: 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 10 monclient: _renew_subs 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7faaa41cba10 con 0x7faaa40a0e90 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa9b7fe640 4 set_mon_vals no callback set 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.497+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7faa9c010550 con 0x7faaa40a0e90 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faaab52c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7faaa409ef00 con 0x7faaa40a0e90 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7faa9c047b40 con 0x7faaa40a0e90 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.171:6808/4274747803 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa9c001d50 con 0x7faa88043030 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (unknown 1324813924 0 0) 0x7faa9c0154d0 con 0x7faaa40a0e90 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faa99ffb640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faaa92c8640 10 monclient: get_auth_request con 0x7faa88039200 auth_method 0 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.498+0000 7faaa8ac7640 10 monclient: get_auth_request con 0x7faa88043030 auth_method 0 2024-09-16T04:28:05.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.499+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== osd.4 v1:172.21.15.171:6808/4274747803 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faa9c001d50 con 0x7faa88043030 2024-09-16T04:28:05.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.511+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 --> v1:172.21.15.171:6808/4274747803 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faaa4072da0 con 0x7faa88043030 2024-09-16T04:28:05.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faa99ffb640 1 -- 172.21.15.148:0/3143076261 <== osd.4 v1:172.21.15.171:6808/4274747803 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7faaa4072da0 con 0x7faa88043030 2024-09-16T04:28:05.525 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:28:05.526 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:28:05.526 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:28:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 >> v1:172.21.15.171:6808/4274747803 conn(0x7faa88043030 legacy=0x7faa88045490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.527 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 >> v1:172.21.15.171:6812/7327273 conn(0x7faa88039200 legacy=0x7faa8803b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 10 monclient: shutdown 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 >> v1:172.21.15.148:6790/0 conn(0x7faaa40a0e90 legacy=0x7faaa41d4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaa9ac9640 1 -- 172.21.15.148:0/3143076261 reap_dead start 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 shutdown_connections 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 >> 172.21.15.148:0/3143076261 conn(0x7faaa40a5640 msgr2=0x7faaa4070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 shutdown_connections 2024-09-16T04:28:05.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.512+0000 7faaab52c640 1 -- 172.21.15.148:0/3143076261 wait complete. 2024-09-16T04:28:05.529 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.625+0000 7f2893ea6640 10 monclient: get_monmap_and_config 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.625+0000 7f2893ea6640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.625+0000 7f2893ea6640 10 monclient: monmap: 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:05.626545+0000 2024-09-16T04:28:05.627 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:05.626545+0000 2024-09-16T04:28:05.628 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.628 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.628 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.628 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:05.628 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:05.628 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 1 Processor -- start 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 1 -- start start 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient: init 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient(hunting): picked mon.noname-a con 0x7f288c134870 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient(hunting): picked mon.noname-b con 0x7f288c09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient(hunting): picked mon.noname-c con 0x7f288c13ae80 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.637 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c135360 con 0x7f288c134870 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c134440 con 0x7f288c13ae80 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c09e650 con 0x7f288c09e240 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.626+0000 7f2893ea6640 10 monclient(hunting): authenticate will time out at 2106.251976s 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2891c42640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f288c134870 0x7f288c134c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48224/0 (socket says 172.21.15.148:48224) 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2891c42640 1 -- 172.21.15.148:0/2408442554 learned_addr learned my addr 172.21.15.148:0/2408442554 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2891441640 1 --1- 172.21.15.148:0/2408442554 >> v1:172.21.15.148:6790/0 conn(0x7f288c09e240 0x7f288c139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38766/0 (socket says 172.21.15.148:38766) 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3437568855 0 0) 0x7f288c135360 con 0x7f288c134870 2024-09-16T04:28:05.638 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): my global_id is 174210 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2878003520 con 0x7f288c134870 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3491362709 0 0) 0x7f288c09e650 con 0x7f288c09e240 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient(hunting): my global_id is 154460 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f288c135360 con 0x7f288c09e240 2024-09-16T04:28:05.639 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2574606770 0 0) 0x7f2878003520 con 0x7f288c134870 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 >> v1:172.21.15.148:6790/0 conn(0x7f288c09e240 legacy=0x7f288c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 >> v1:172.21.15.171:6789/0 conn(0x7f288c13ae80 legacy=0x7f288c13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 monclient: found mon.noname-a 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288c073230 con 0x7f288c134870 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.640 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2893ea6640 5 monclient: authenticate success, global_id 174210 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2888001610 con 0x7f288c134870 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: dump: 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.641 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.642 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2893ea6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f28880018f0 con 0x7f288c134870 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 1 -- 172.21.15.148:0/2408442554 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2888001d00 con 0x7f288c134870 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2890c40640 10 monclient: dump: 2024-09-16T04:28:05.643 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.644 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.644 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.645 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2893ea6640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.627+0000 7f2893ea6640 4 set_mon_vals no callback set 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 10 monclient: shutdown 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 1 -- 172.21.15.148:0/2408442554 >> v1:172.21.15.148:6789/0 conn(0x7f288c134870 legacy=0x7f288c134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 1 -- 172.21.15.148:0/2408442554 shutdown_connections 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 1 -- 172.21.15.148:0/2408442554 >> 172.21.15.148:0/2408442554 conn(0x7f288c0a5640 msgr2=0x7f288c0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 1 -- 172.21.15.148:0/2408442554 shutdown_connections 2024-09-16T04:28:05.646 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 1 -- 172.21.15.148:0/2408442554 wait complete. 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.628+0000 7f2893ea6640 10 monclient: monmap: 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.647 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 1 Processor -- start 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 1 -- start start 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient: init 2024-09-16T04:28:05.648 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient(hunting): picked mon.noname-c con 0x7f288c09e240 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient(hunting): picked mon.noname-b con 0x7f288c134870 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient(hunting): picked mon.noname-a con 0x7f288c13ae80 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c120860 con 0x7f288c13ae80 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c133930 con 0x7f288c134870 2024-09-16T04:28:05.649 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c09d6d0 con 0x7f288c09e240 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2893ea6640 10 monclient(hunting): authenticate will time out at 2106.254631s 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2891c42640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f288c09e240 0x7f288c132b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38776/0 (socket says 172.21.15.148:38776) 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f2891c42640 1 -- 172.21.15.148:0/214854836 learned_addr learned my addr 172.21.15.148:0/214854836 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620045874 0 0) 0x7f288c09d6d0 con 0x7f288c09e240 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.650 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): my global_id is 154463 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2870003520 con 0x7f288c09e240 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2919139053 0 0) 0x7f288c120860 con 0x7f288c13ae80 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 10 monclient(hunting): my global_id is 174213 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.629+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f288c09d6d0 con 0x7f288c13ae80 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230369665 0 0) 0x7f288c133930 con 0x7f288c134870 2024-09-16T04:28:05.651 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): my global_id is 164341 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f288c120860 con 0x7f288c134870 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1979816620 0 0) 0x7f2870003520 con 0x7f288c09e240 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f288c133930 con 0x7f288c09e240 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2888001c40 con 0x7f288c09e240 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.652 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): dump: 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.653 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4244341038 0 0) 0x7f288c09d6d0 con 0x7f288c13ae80 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2870003520 con 0x7f288c13ae80 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2884001b50 con 0x7f288c13ae80 2024-09-16T04:28:05.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): dump: 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.655 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.656 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.656 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.656 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.656 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 404217045 0 0) 0x7f288c120860 con 0x7f288c134870 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f288c09d6d0 con 0x7f288c134870 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f287c001070 con 0x7f288c134870 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient(hunting): dump: 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.657 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1290667170 0 0) 0x7f288c133930 con 0x7f288c09e240 2024-09-16T04:28:05.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 >> v1:172.21.15.171:6789/0 conn(0x7f288c134870 legacy=0x7f288c133220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 >> v1:172.21.15.148:6789/0 conn(0x7f288c13ae80 legacy=0x7f288c1d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 monclient: found mon.c 2024-09-16T04:28:05.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288c0a43d0 con 0x7f288c09e240 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 5 monclient: authenticate success, global_id 154463 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 10 monclient: _renew_subs 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f288c120580 con 0x7f288c09e240 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 10 monclient: _renew_subs 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f288c09cd60 con 0x7f288c09e240 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2888004cc0 con 0x7f288c09e240 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f2888005010 con 0x7f288c09e240 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f2883fff640 4 set_mon_vals no callback set 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.630+0000 7f28827fc640 10 monclient: dump: 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.661 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f28880102a0 con 0x7f288c09e240 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f2888047880 con 0x7f288c09e240 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f2893ea6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:05.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f2893ea6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:05.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f2893ea6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2874005080 con 0x7f288c09e240 2024-09-16T04:28:05.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.631+0000 7f2891441640 10 monclient: get_auth_request con 0x7f28700391f0 auth_method 0 2024-09-16T04:28:05.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.636+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f2888015be0 con 0x7f288c09e240 2024-09-16T04:28:05.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.636+0000 7f28827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:05.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.636+0000 7f28827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:05.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.799+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f2874000b90 con 0x7f28700391f0 2024-09-16T04:28:05.801 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.800+0000 7f28827fc640 1 -- 172.21.15.148:0/214854836 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f2874000b90 con 0x7f28700391f0 2024-09-16T04:28:05.802 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 >> v1:172.21.15.171:6812/7327273 conn(0x7f28700391f0 legacy=0x7f287003b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 10 monclient: shutdown 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 >> v1:172.21.15.148:6790/0 conn(0x7f288c09e240 legacy=0x7f288c132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 shutdown_connections 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 >> 172.21.15.148:0/214854836 conn(0x7f288c0a5640 msgr2=0x7f288c0a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 shutdown_connections 2024-09-16T04:28:05.804 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.802+0000 7f2893ea6640 1 -- 172.21.15.148:0/214854836 wait complete. 2024-09-16T04:28:05.814 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:28:05.815 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:28:05.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.914+0000 7f7636fe4640 10 monclient: get_monmap_and_config 2024-09-16T04:28:05.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.914+0000 7f7636fe4640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.916 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.914+0000 7f7636fe4640 10 monclient: monmap: 2024-09-16T04:28:05.916 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.916 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:05.915847+0000 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:05.915847+0000 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:05.917 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.915+0000 7f7636fe4640 1 Processor -- start 2024-09-16T04:28:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.915+0000 7f7636fe4640 1 -- start start 2024-09-16T04:28:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.915+0000 7f7636fe4640 10 monclient: init 2024-09-16T04:28:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.927 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient(hunting): picked mon.noname-a con 0x7f763009bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient(hunting): picked mon.noname-b con 0x7f763009c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient(hunting): picked mon.noname-c con 0x7f763009cc30 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76300a1010 con 0x7f763009bb70 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763009f030 con 0x7f763009cc30 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763009dc60 con 0x7f763009c660 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient(hunting): authenticate will time out at 2106.541190s 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7627fff640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f763009c660 0x7f7630138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38778/0 (socket says 172.21.15.148:38778) 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7627fff640 1 -- 172.21.15.148:0/1336884185 learned_addr learned my addr 172.21.15.148:0/1336884185 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7634d80640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f763009bb70 0x7f763009bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48250/0 (socket says 172.21.15.148:48250) 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679276243 0 0) 0x7f763009dc60 con 0x7f763009c660 2024-09-16T04:28:05.928 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): my global_id is 154466 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7618003520 con 0x7f763009c660 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211487265 0 0) 0x7f76300a1010 con 0x7f763009bb70 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): my global_id is 174216 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f763009dc60 con 0x7f763009bb70 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3687300905 0 0) 0x7f763009f030 con 0x7f763009cc30 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.929 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient(hunting): my global_id is 164344 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76300a1010 con 0x7f763009cc30 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 127848071 0 0) 0x7f7618003520 con 0x7f763009c660 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 >> v1:172.21.15.171:6789/0 conn(0x7f763009cc30 legacy=0x7f763013c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 >> v1:172.21.15.148:6789/0 conn(0x7f763009bb70 legacy=0x7f763009bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 monclient: found mon.noname-b 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763009df40 con 0x7f763009c660 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7620001610 con 0x7f763009c660 2024-09-16T04:28:05.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 5 monclient: authenticate success, global_id 154466 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: dump: 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.931 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7620001840 con 0x7f763009c660 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f7636fe4640 4 set_mon_vals no callback set 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 1 -- 172.21.15.148:0/1336884185 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7620001c50 con 0x7f763009c660 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.916+0000 7f76277fe640 10 monclient: dump: 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.932 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 10 monclient: shutdown 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 1 -- 172.21.15.148:0/1336884185 >> v1:172.21.15.148:6790/0 conn(0x7f763009c660 legacy=0x7f7630138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 1 -- 172.21.15.148:0/1336884185 shutdown_connections 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 1 -- 172.21.15.148:0/1336884185 >> 172.21.15.148:0/1336884185 conn(0x7f76300a5640 msgr2=0x7f76300a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 1 -- 172.21.15.148:0/1336884185 shutdown_connections 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.917+0000 7f7636fe4640 1 -- 172.21.15.148:0/1336884185 wait complete. 2024-09-16T04:28:05.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient: build_initial_monmap 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient: monmap: 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 1 Processor -- start 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 1 -- start start 2024-09-16T04:28:05.934 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient: init 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient(hunting): picked mon.noname-b con 0x7f763009bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient(hunting): picked mon.noname-a con 0x7f763009c660 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient(hunting): picked mon.noname-c con 0x7f763009cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76300a3d70 con 0x7f763009c660 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763013ba20 con 0x7f763009bb70 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763013b740 con 0x7f763009cc30 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.918+0000 7f7636fe4640 10 monclient(hunting): authenticate will time out at 2106.543966s 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7627fff640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f763009c660 0x7f76301d09a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48258/0 (socket says 172.21.15.148:48258) 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7627fff640 1 -- 172.21.15.148:0/1119913202 learned_addr learned my addr 172.21.15.148:0/1119913202 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:05.935 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2507005784 0 0) 0x7f763013b740 con 0x7f763009cc30 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): my global_id is 154469 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f761c003520 con 0x7f763009cc30 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333746377 0 0) 0x7f763009df40 con 0x7f763009c660 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): my global_id is 174219 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f763013b740 con 0x7f763009c660 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344758582 0 0) 0x7f763013ba20 con 0x7f763009bb70 2024-09-16T04:28:05.936 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _negotiate 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): my global_id is 164347 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f763009df40 con 0x7f763009bb70 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 87754859 0 0) 0x7f761c003520 con 0x7f763009cc30 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f763013ba20 con 0x7f763009cc30 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7628001c30 con 0x7f763009cc30 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): dump: 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.937 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3333154063 0 0) 0x7f763013b740 con 0x7f763009c660 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f761c003520 con 0x7f763009c660 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7620001c50 con 0x7f763009c660 2024-09-16T04:28:05.938 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): dump: 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 355199720 0 0) 0x7f763009df40 con 0x7f763009bb70 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f763013b740 con 0x7f763009bb70 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f762c0010b0 con 0x7f763009bb70 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient(hunting): dump: 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.940 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4126190838 0 0) 0x7f763013ba20 con 0x7f763009cc30 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 >> v1:172.21.15.171:6789/0 conn(0x7f763009bb70 legacy=0x7f76301cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 >> v1:172.21.15.148:6789/0 conn(0x7f763009c660 legacy=0x7f76301d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: _finish_hunting 0 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 monclient: found mon.c 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763013bd00 con 0x7f763009cc30 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: _finish_auth 0 2024-09-16T04:28:05.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: _check_auth_tickets 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 5 monclient: authenticate success, global_id 154469 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 10 monclient: _renew_subs 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f763013c990 con 0x7f763009cc30 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 10 monclient: _renew_subs 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f763006efb0 con 0x7f763009cc30 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7628003990 con 0x7f763009cc30 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f7628003ce0 con 0x7f763009cc30 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f76257fa640 10 monclient: dump: 2024-09-16T04:28:05.942 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.919+0000 7f7626ffd640 4 set_mon_vals no callback set 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f762800ef70 con 0x7f763009cc30 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f7636fe4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.148:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f763006f2a0 con 0x7f763009cc30 2024-09-16T04:28:05.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f76280468a0 con 0x7f763009cc30 2024-09-16T04:28:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.171:6800/466524840 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f761c046d50 con 0x7f761c043080 2024-09-16T04:28:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (unknown 1324813924 0 0) 0x7f76280141e0 con 0x7f763009cc30 2024-09-16T04:28:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f76257fa640 10 monclient: handle_get_version_reply finishing 1 version 257 2024-09-16T04:28:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.920+0000 7f7634d80640 10 monclient: get_auth_request con 0x7f761c039250 auth_method 0 2024-09-16T04:28:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.921+0000 7f7627fff640 10 monclient: get_auth_request con 0x7f761c043080 auth_method 0 2024-09-16T04:28:05.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.921+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== osd.5 v1:172.21.15.171:6800/466524840 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f761c046d50 con 0x7f761c043080 2024-09-16T04:28:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.934+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 --> v1:172.21.15.171:6800/466524840 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f763006f4a0 con 0x7f761c043080 2024-09-16T04:28:05.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.934+0000 7f76257fa640 1 -- 172.21.15.148:0/1119913202 <== osd.5 v1:172.21.15.171:6800/466524840 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f763006f4a0 con 0x7f761c043080 2024-09-16T04:28:05.945 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-16T04:28:05.945 INFO:teuthology.orchestra.run.smithi148.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:28:05.945 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.934+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 >> v1:172.21.15.171:6800/466524840 conn(0x7f761c043080 legacy=0x7f761c0454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 >> v1:172.21.15.171:6812/7327273 conn(0x7f761c039250 legacy=0x7f761c03b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 10 monclient: shutdown 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 >> v1:172.21.15.148:6790/0 conn(0x7f763009cc30 legacy=0x7f76301d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7635581640 1 -- 172.21.15.148:0/1119913202 reap_dead start 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 shutdown_connections 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 >> 172.21.15.148:0/1119913202 conn(0x7f76300a5640 msgr2=0x7f763013ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 shutdown_connections 2024-09-16T04:28:05.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:05.935+0000 7f7636fe4640 1 -- 172.21.15.148:0/1119913202 wait complete. 2024-09-16T04:28:05.948 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:28:06.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.045+0000 7fe6536d0640 10 monclient: get_monmap_and_config 2024-09-16T04:28:06.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.045+0000 7fe6536d0640 10 monclient: build_initial_monmap 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.045+0000 7fe6536d0640 10 monclient: monmap: 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:06.046072+0000 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:06.046072+0000 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:06.047 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.045+0000 7fe6536d0640 1 Processor -- start 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.045+0000 7fe6536d0640 1 -- start start 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.045+0000 7fe6536d0640 10 monclient: init 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe64c0a2410 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe64c0a1b10 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe64c13d060 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c0a0e00 con 0x7fe64c0a2410 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c0a10e0 con 0x7fe64c13d060 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c0a4970 con 0x7fe64c0a1b10 2024-09-16T04:28:06.055 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe6536d0640 10 monclient(hunting): authenticate will time out at 2106.671481s 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe650c6b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe64c0a1b10 0x7fe64c13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38788/0 (socket says 172.21.15.148:38788) 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe65146c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fe64c0a2410 0x7fe64c0a27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48268/0 (socket says 172.21.15.148:48268) 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe650c6b640 1 -- 172.21.15.148:0/1280985528 learned_addr learned my addr 172.21.15.148:0/1280985528 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326845971 0 0) 0x7fe64c0a0e00 con 0x7fe64c0a2410 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): my global_id is 174222 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe638003520 con 0x7fe64c0a2410 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271798600 0 0) 0x7fe64c0a4970 con 0x7fe64c0a1b10 2024-09-16T04:28:06.056 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 10 monclient(hunting): my global_id is 154472 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c0a0e00 con 0x7fe64c0a1b10 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.046+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3363177607 0 0) 0x7fe64c0a10e0 con 0x7fe64c13d060 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient(hunting): my global_id is 164350 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c0a4970 con 0x7fe64c13d060 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4240975139 0 0) 0x7fe64c0a0e00 con 0x7fe64c0a1b10 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 >> v1:172.21.15.171:6789/0 conn(0x7fe64c13d060 legacy=0x7fe64c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 >> v1:172.21.15.148:6789/0 conn(0x7fe64c0a2410 legacy=0x7fe64c0a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.057 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: _finish_hunting 0 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 monclient: found mon.noname-b 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c06dfd0 con 0x7fe64c0a1b10 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: _finish_auth 0 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: _check_auth_tickets 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 5 monclient: authenticate success, global_id 154472 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe63c0015f0 con 0x7fe64c0a1b10 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: dump: 2024-09-16T04:28:06.058 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe63c0018d0 con 0x7fe64c0a1b10 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:06.059 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 1 -- 172.21.15.148:0/1280985528 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe63c001ce0 con 0x7fe64c0a1b10 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 4 set_mon_vals no callback set 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe643fff640 10 monclient: dump: 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.060 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 10 monclient: shutdown 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 1 -- 172.21.15.148:0/1280985528 >> v1:172.21.15.148:6790/0 conn(0x7fe64c0a1b10 legacy=0x7fe64c13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 1 -- 172.21.15.148:0/1280985528 shutdown_connections 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 1 -- 172.21.15.148:0/1280985528 >> 172.21.15.148:0/1280985528 conn(0x7fe64c0a97b0 msgr2=0x7fe64c0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 1 -- 172.21.15.148:0/1280985528 shutdown_connections 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.047+0000 7fe6536d0640 1 -- 172.21.15.148:0/1280985528 wait complete. 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.048+0000 7fe6536d0640 10 monclient: build_initial_monmap 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.048+0000 7fe6536d0640 10 monclient: monmap: 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:06.061 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.048+0000 7fe6536d0640 1 Processor -- start 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.048+0000 7fe6536d0640 1 -- start start 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.048+0000 7fe6536d0640 10 monclient: init 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient(hunting): picked mon.noname-b con 0x7fe64c0a1b10 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:06.062 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient(hunting): picked mon.noname-c con 0x7fe64c0a2410 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient(hunting): picked mon.noname-a con 0x7fe64c13d060 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c137760 con 0x7fe64c13d060 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c09d840 con 0x7fe64c0a1b10 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c09f1a0 con 0x7fe64c0a2410 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 10 monclient(hunting): authenticate will time out at 2106.674220s 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe650c6b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7fe64c0a2410 0x7fe64c09d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38798/0 (socket says 172.21.15.148:38798) 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe650c6b640 1 -- 172.21.15.148:0/1988613637 learned_addr learned my addr 172.21.15.148:0/1988613637 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe651c6d640 1 --1- 172.21.15.148:0/1988613637 >> v1:172.21.15.148:6789/0 conn(0x7fe64c13d060 0x7fe64c1d3b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48272/0 (socket says 172.21.15.148:48272) 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660040178 0 0) 0x7fe64c137760 con 0x7fe64c13d060 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.063 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): my global_id is 174225 2024-09-16T04:28:06.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe630003520 con 0x7fe64c13d060 2024-09-16T04:28:06.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3296469824 0 0) 0x7fe64c09f1a0 con 0x7fe64c0a2410 2024-09-16T04:28:06.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.064 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): my global_id is 154475 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c137760 con 0x7fe64c0a2410 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329975392 0 0) 0x7fe64c09d840 con 0x7fe64c0a1b10 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.065 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): my global_id is 164353 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c09f1a0 con 0x7fe64c0a1b10 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1025229307 0 0) 0x7fe630003520 con 0x7fe64c13d060 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe64c09d840 con 0x7fe64c13d060 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1775525474 0 0) 0x7fe64c137760 con 0x7fe64c0a2410 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe630003520 con 0x7fe64c0a2410 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe648001c30 con 0x7fe64c13d060 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:06.066 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:06.067 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:06.067 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.067 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.067 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.067 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.067 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe63c001c20 con 0x7fe64c0a2410 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:06.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.1 v1:172.21.15.171:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1506534002 0 0) 0x7fe64c09f1a0 con 0x7fe64c0a1b10 2024-09-16T04:28:06.069 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.171:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe64c137760 con 0x7fe64c0a1b10 2024-09-16T04:28:06.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.1 v1:172.21.15.171:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe644001840 con 0x7fe64c0a1b10 2024-09-16T04:28:06.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:06.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-16T04:28:06.070 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient(hunting): dump: 2024-09-16T04:28:06.070 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.070 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1624077728 0 0) 0x7fe630003520 con 0x7fe64c0a2410 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 >> v1:172.21.15.171:6789/0 conn(0x7fe64c0a1b10 legacy=0x7fe64c09ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.071 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 >> v1:172.21.15.148:6789/0 conn(0x7fe64c13d060 legacy=0x7fe64c1d3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 monclient: found mon.c 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c13cae0 con 0x7fe64c0a2410 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient: _finish_auth 0 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:06.072 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.049+0000 7fe6536d0640 5 monclient: authenticate success, global_id 154475 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6536d0640 10 monclient: _renew_subs 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6536d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c0a0130 con 0x7fe64c0a2410 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe63c004cc0 con 0x7fe64c0a2410 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7fe63c005010 con 0x7fe64c0a2410 2024-09-16T04:28:06.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 10 monclient: dump: 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.074 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6536d0640 10 monclient: _renew_subs 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6536d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe64c09ef20 con 0x7fe64c0a2410 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe6437fe640 4 set_mon_vals no callback set 2024-09-16T04:28:06.075 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.050+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7fe63c0102a0 con 0x7fe64c0a2410 2024-09-16T04:28:06.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.051+0000 7fe6536d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:06.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.051+0000 7fe6536d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:06.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.051+0000 7fe6536d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:06.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.051+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe64c0008d0 con 0x7fe64c0a2410 2024-09-16T04:28:06.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.051+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7fe63c047800 con 0x7fe64c0a2410 2024-09-16T04:28:06.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.051+0000 7fe65146c640 10 monclient: get_auth_request con 0x7fe630039220 auth_method 0 2024-09-16T04:28:06.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.055+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7fe63c015b20 con 0x7fe64c0a2410 2024-09-16T04:28:06.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.055+0000 7fe641ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:06.078 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.055+0000 7fe641ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:06.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.223+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fe64c13c260 con 0x7fe630039220 2024-09-16T04:28:06.225 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.223+0000 7fe641ffb640 1 -- 172.21.15.148:0/1988613637 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7fe64c13c260 con 0x7fe630039220 2024-09-16T04:28:06.225 INFO:teuthology.orchestra.run.smithi148.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:28:06.227 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.225+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 >> v1:172.21.15.171:6812/7327273 conn(0x7fe630039220 legacy=0x7fe63003b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.225+0000 7fe6536d0640 10 monclient: shutdown 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.225+0000 7fe6536d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.226+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 >> v1:172.21.15.148:6790/0 conn(0x7fe64c0a2410 legacy=0x7fe64c09d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.226+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 shutdown_connections 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.226+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 >> 172.21.15.148:0/1988613637 conn(0x7fe64c0a97b0 msgr2=0x7fe64c135860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.226+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 shutdown_connections 2024-09-16T04:28:06.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.226+0000 7fe6536d0640 1 -- 172.21.15.148:0/1988613637 wait complete. 2024-09-16T04:28:06.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:28:06.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.333+0000 7f50f0d26640 10 monclient: get_monmap_and_config 2024-09-16T04:28:06.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.333+0000 7f50f0d26640 10 monclient: build_initial_monmap 2024-09-16T04:28:06.335 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.333+0000 7f50f0d26640 10 monclient: monmap: 2024-09-16T04:28:06.335 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:06.334537+0000 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:06.334537+0000 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 1 Processor -- start 2024-09-16T04:28:06.336 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 1 -- start start 2024-09-16T04:28:06.337 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient: init 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient(hunting): picked mon.noname-b con 0x7f50ec0a0e90 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient(hunting): picked mon.noname-a con 0x7f50ec09b170 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient(hunting): picked mon.noname-c con 0x7f50ec09b5c0 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec09e040 con 0x7f50ec09b170 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec09d660 con 0x7f50ec09b5c0 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec09c0d0 con 0x7f50ec0a0e90 2024-09-16T04:28:06.345 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50f0d26640 10 monclient(hunting): authenticate will time out at 2106.959684s 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50ea59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f50ec0a0e90 0x7f50ec09aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38802/0 (socket says 172.21.15.148:38802) 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50e1d9b640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f50ec09b170 0x7f50ec1383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:48276/0 (socket says 172.21.15.148:48276) 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50ea59c640 1 -- 172.21.15.148:0/182391987 learned_addr learned my addr 172.21.15.148:0/182391987 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.334+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323286441 0 0) 0x7f50d4001110 con 0x7f50ec09b170 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): my global_id is 174228 2024-09-16T04:28:06.346 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d8003440 con 0x7f50ec09b170 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3882904178 0 0) 0x7f50ec09c0d0 con 0x7f50ec0a0e90 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient(hunting): my global_id is 154478 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d4001110 con 0x7f50ec0a0e90 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1931571 0 0) 0x7f50d8003440 con 0x7f50ec09b170 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 >> v1:172.21.15.148:6790/0 conn(0x7f50ec0a0e90 legacy=0x7f50ec09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 >> v1:172.21.15.171:6789/0 conn(0x7f50ec09b5c0 legacy=0x7f50ec13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:28:06.347 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 monclient: found mon.noname-a 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec09d380 con 0x7f50ec09b170 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: _finish_auth 0 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 5 monclient: authenticate success, global_id 174228 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50d40017e0 con 0x7f50ec09b170 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:06.348 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: dump: 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.349 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50d4001a50 con 0x7f50ec09b170 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 1 -- 172.21.15.148:0/182391987 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50d40047e0 con 0x7f50ec09b170 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 4 set_mon_vals no callback set 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50e9d9b640 10 monclient: dump: 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.350 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 10 monclient: shutdown 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 1 -- 172.21.15.148:0/182391987 >> v1:172.21.15.148:6789/0 conn(0x7f50ec09b170 legacy=0x7f50ec1383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 1 -- 172.21.15.148:0/182391987 shutdown_connections 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 1 -- 172.21.15.148:0/182391987 >> 172.21.15.148:0/182391987 conn(0x7f50ec0a5640 msgr2=0x7f50ec0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:06.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 1 -- 172.21.15.148:0/182391987 shutdown_connections 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.335+0000 7f50f0d26640 1 -- 172.21.15.148:0/182391987 wait complete. 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.336+0000 7f50f0d26640 10 monclient: build_initial_monmap 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.336+0000 7f50f0d26640 10 monclient: monmap: 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:06.352 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 1 Processor -- start 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 1 -- start start 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient: init 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient(hunting): picked mon.noname-c con 0x7f50ec09b170 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient(hunting): picked mon.noname-a con 0x7f50ec09b5c0 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient(hunting): picked mon.noname-b con 0x7f50ec0a0e90 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:06.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec1c1880 con 0x7f50ec09b5c0 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec1a6010 con 0x7f50ec0a0e90 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50ec1a8ad0 con 0x7f50ec09b170 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50f0d26640 10 monclient(hunting): authenticate will time out at 2106.962587s 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50ea59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f50ec09b170 0x7f50ec1c1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:38804/0 (socket says 172.21.15.148:38804) 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50ea59c640 1 -- 172.21.15.148:0/3321886827 learned_addr learned my addr 172.21.15.148:0/3321886827 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 86669603 0 0) 0x7f50ec1a8ad0 con 0x7f50ec09b170 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): my global_id is 154481 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50cc003520 con 0x7f50ec09b170 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355242133 0 0) 0x7f50ec1c1880 con 0x7f50ec09b5c0 2024-09-16T04:28:06.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 10 monclient(hunting): my global_id is 174231 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.337+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50ec1a8ad0 con 0x7f50ec09b5c0 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099028313 0 0) 0x7f50ec1a6010 con 0x7f50ec0a0e90 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): _negotiate 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): my global_id is 164359 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50ec1c1880 con 0x7f50ec0a0e90 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1718167600 0 0) 0x7f50ec1a8ad0 con 0x7f50ec09b5c0 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50ec1a6010 con 0x7f50ec09b5c0 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1904784895 0 0) 0x7f50cc003520 con 0x7f50ec09b170 2024-09-16T04:28:06.355 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50ec1a8ad0 con 0x7f50ec09b170 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50d4000fe0 con 0x7f50ec09b5c0 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): dump: 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.356 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50dc001890 con 0x7f50ec09b170 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient(hunting): dump: 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.357 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1057502548 0 0) 0x7f50ec1a6010 con 0x7f50ec09b5c0 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 >> v1:172.21.15.148:6790/0 conn(0x7f50ec09b170 legacy=0x7f50ec1c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 >> v1:172.21.15.171:6789/0 conn(0x7f50ec0a0e90 legacy=0x7f50ec1cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 monclient: found mon.a 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec1a6800 con 0x7f50ec09b5c0 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: _finish_auth 0 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 5 monclient: authenticate success, global_id 174231 2024-09-16T04:28:06.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 10 monclient: _renew_subs 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec1a87f0 con 0x7f50ec09b5c0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 10 monclient: _renew_subs 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50ec1c3be0 con 0x7f50ec09b5c0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50d4001240 con 0x7f50ec09b5c0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f50d4004f00 con 0x7f50ec09b5c0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e959a640 4 set_mon_vals no callback set 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.338+0000 7f50e37fe640 10 monclient: dump: 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:06.359 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f50d4010330 con 0x7f50ec09b5c0 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50f0d26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50f0d26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50f0d26640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d0005080 con 0x7f50ec09b5c0 2024-09-16T04:28:06.360 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f50d4047880 con 0x7f50ec09b5c0 2024-09-16T04:28:06.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.339+0000 7f50ea59c640 10 monclient: get_auth_request con 0x7f50cc039110 auth_method 0 2024-09-16T04:28:06.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.344+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f50ec09e040 con 0x7f50ec09b5c0 2024-09-16T04:28:06.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.344+0000 7f50e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:06.361 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.344+0000 7f50e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:06.508 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.507+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f50d0000b90 con 0x7f50cc039110 2024-09-16T04:28:06.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.509+0000 7f50e37fe640 1 -- 172.21.15.148:0/3321886827 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77837 (unknown 2965378022 0 4059319985) 0x7f50d0000b90 con 0x7f50cc039110 2024-09-16T04:28:06.511 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:28:06.511 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 >> v1:172.21.15.171:6812/7327273 conn(0x7f50cc039110 legacy=0x7f50cc03b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 10 monclient: shutdown 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 >> v1:172.21.15.148:6789/0 conn(0x7f50ec09b5c0 legacy=0x7f50ec1cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 shutdown_connections 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 >> 172.21.15.148:0/3321886827 conn(0x7f50ec0a5640 msgr2=0x7f50ec0a3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 shutdown_connections 2024-09-16T04:28:06.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:06.511+0000 7f50f0d26640 1 -- 172.21.15.148:0/3321886827 wait complete. 2024-09-16T04:28:06.524 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":899,"stamp":"2024-09-16T04:28:05.550205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197136,"kb_used_data":17608,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566033904,"statfs":{"total":579820584960,"available":579618717696,"internally_reserved":0,"allocated":18030592,"data_stored":15149110,"data_compressed":1198654,"data_compressed_allocated":2351104,"data_compressed_original":6012928,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":307,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:26:40.321350+0000","last_change":"2024-09-16T04:11:11.973056+0000","last_active":"2024-09-16T04:26:40.321350+0000","last_peered":"2024-09-16T04:26:40.321350+0000","last_clean":"2024-09-16T04:26:40.321350+0000","last_became_active":"2024-09-16T04:11:11.972654+0000","last_became_peered":"2024-09-16T04:11:11.972654+0000","last_unstale":"2024-09-16T04:26:40.321350+0000","last_undegraded":"2024-09-16T04:26:40.321350+0000","last_fullsized":"2024-09-16T04:26:40.321350+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298089+0000","last_change":"2024-09-16T04:11:11.971214+0000","last_active":"2024-09-16T04:24:11.298089+0000","last_peered":"2024-09-16T04:24:11.298089+0000","last_clean":"2024-09-16T04:24:11.298089+0000","last_became_active":"2024-09-16T04:11:11.970821+0000","last_became_peered":"2024-09-16T04:11:11.970821+0000","last_unstale":"2024-09-16T04:24:11.298089+0000","last_undegraded":"2024-09-16T04:24:11.298089+0000","last_fullsized":"2024-09-16T04:24:11.298089+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:04.704581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":288,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:05.057488+0000","last_change":"2024-09-16T04:28:05.057522+0000","last_active":"2024-09-16T04:28:05.057488+0000","last_peered":"2024-09-16T04:28:05.057488+0000","last_clean":"2024-09-16T04:28:05.057488+0000","last_became_active":"2024-09-16T04:11:11.971647+0000","last_became_peered":"2024-09-16T04:11:11.971647+0000","last_unstale":"2024-09-16T04:28:05.057488+0000","last_undegraded":"2024-09-16T04:28:05.057488+0000","last_fullsized":"2024-09-16T04:28:05.057488+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:05.057386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:05.057386+0000","last_clean_scrub_stamp":"2024-09-16T04:28:05.057386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.670217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":175,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.298065+0000","last_change":"2024-09-16T04:11:11.974362+0000","last_active":"2024-09-16T04:24:11.298065+0000","last_peered":"2024-09-16T04:24:11.298065+0000","last_clean":"2024-09-16T04:24:11.298065+0000","last_became_active":"2024-09-16T04:11:11.974022+0000","last_became_peered":"2024-09-16T04:11:11.974022+0000","last_unstale":"2024-09-16T04:24:11.298065+0000","last_undegraded":"2024-09-16T04:24:11.298065+0000","last_fullsized":"2024-09-16T04:24:11.298065+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:35.453472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":224,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:11.304629+0000","last_change":"2024-09-16T04:11:12.969110+0000","last_active":"2024-09-16T04:25:11.304629+0000","last_peered":"2024-09-16T04:25:11.304629+0000","last_clean":"2024-09-16T04:25:11.304629+0000","last_became_active":"2024-09-16T04:11:12.968688+0000","last_became_peered":"2024-09-16T04:11:12.968688+0000","last_unstale":"2024-09-16T04:25:11.304629+0000","last_undegraded":"2024-09-16T04:25:11.304629+0000","last_fullsized":"2024-09-16T04:25:11.304629+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:52.202804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":295,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:26:45.321905+0000","last_change":"2024-09-16T04:11:11.972073+0000","last_active":"2024-09-16T04:26:45.321905+0000","last_peered":"2024-09-16T04:26:45.321905+0000","last_clean":"2024-09-16T04:26:45.321905+0000","last_became_active":"2024-09-16T04:11:11.971583+0000","last_became_peered":"2024-09-16T04:11:11.971583+0000","last_unstale":"2024-09-16T04:26:45.321905+0000","last_undegraded":"2024-09-16T04:26:45.321905+0000","last_fullsized":"2024-09-16T04:26:45.321905+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395998+0000","last_change":"2024-09-16T04:11:11.975563+0000","last_active":"2024-09-16T04:24:11.395998+0000","last_peered":"2024-09-16T04:24:11.395998+0000","last_clean":"2024-09-16T04:24:11.395998+0000","last_became_active":"2024-09-16T04:11:11.975180+0000","last_became_peered":"2024-09-16T04:11:11.975180+0000","last_unstale":"2024-09-16T04:24:11.395998+0000","last_undegraded":"2024-09-16T04:24:11.395998+0000","last_fullsized":"2024-09-16T04:24:11.395998+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:27.461051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403830+0000","last_change":"2024-09-16T04:11:11.970716+0000","last_active":"2024-09-16T04:25:48.403830+0000","last_peered":"2024-09-16T04:25:48.403830+0000","last_clean":"2024-09-16T04:25:48.403830+0000","last_became_active":"2024-09-16T04:11:11.970247+0000","last_became_peered":"2024-09-16T04:11:11.970247+0000","last_unstale":"2024-09-16T04:25:48.403830+0000","last_undegraded":"2024-09-16T04:25:48.403830+0000","last_fullsized":"2024-09-16T04:25:48.403830+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:37:37.770332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:16.298525+0000","last_change":"2024-09-16T04:11:11.973578+0000","last_active":"2024-09-16T04:24:16.298525+0000","last_peered":"2024-09-16T04:24:16.298525+0000","last_clean":"2024-09-16T04:24:16.298525+0000","last_became_active":"2024-09-16T04:11:11.973162+0000","last_became_peered":"2024-09-16T04:11:11.973162+0000","last_unstale":"2024-09-16T04:24:16.298525+0000","last_undegraded":"2024-09-16T04:24:16.298525+0000","last_fullsized":"2024-09-16T04:24:16.298525+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:31.864317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":208,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:46.399846+0000","last_change":"2024-09-16T04:11:11.975126+0000","last_active":"2024-09-16T04:24:46.399846+0000","last_peered":"2024-09-16T04:24:46.399846+0000","last_clean":"2024-09-16T04:24:46.399846+0000","last_became_active":"2024-09-16T04:11:11.974614+0000","last_became_peered":"2024-09-16T04:11:11.974614+0000","last_unstale":"2024-09-16T04:24:46.399846+0000","last_undegraded":"2024-09-16T04:24:46.399846+0000","last_fullsized":"2024-09-16T04:24:46.399846+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:25:32.801105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":232,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:33.300353+0000","last_change":"2024-09-16T04:11:11.973087+0000","last_active":"2024-09-16T04:24:33.300353+0000","last_peered":"2024-09-16T04:24:33.300353+0000","last_clean":"2024-09-16T04:24:33.300353+0000","last_became_active":"2024-09-16T04:11:11.972700+0000","last_became_peered":"2024-09-16T04:11:11.972700+0000","last_unstale":"2024-09-16T04:24:33.300353+0000","last_undegraded":"2024-09-16T04:24:33.300353+0000","last_fullsized":"2024-09-16T04:24:33.300353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:13.209774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403797+0000","last_change":"2024-09-16T04:11:11.972841+0000","last_active":"2024-09-16T04:25:48.403797+0000","last_peered":"2024-09-16T04:25:48.403797+0000","last_clean":"2024-09-16T04:25:48.403797+0000","last_became_active":"2024-09-16T04:11:11.972431+0000","last_became_peered":"2024-09-16T04:11:11.972431+0000","last_unstale":"2024-09-16T04:25:48.403797+0000","last_undegraded":"2024-09-16T04:25:48.403797+0000","last_fullsized":"2024-09-16T04:25:48.403797+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:31.140262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":179,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403820+0000","last_change":"2024-09-16T04:11:11.972928+0000","last_active":"2024-09-16T04:25:48.403820+0000","last_peered":"2024-09-16T04:25:48.403820+0000","last_clean":"2024-09-16T04:25:48.403820+0000","last_became_active":"2024-09-16T04:11:11.972513+0000","last_became_peered":"2024-09-16T04:11:11.972513+0000","last_unstale":"2024-09-16T04:25:48.403820+0000","last_undegraded":"2024-09-16T04:25:48.403820+0000","last_fullsized":"2024-09-16T04:25:48.403820+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:31.437265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":316,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:04.457894+0000","last_change":"2024-09-16T04:28:04.457943+0000","last_active":"2024-09-16T04:28:04.457894+0000","last_peered":"2024-09-16T04:28:04.457894+0000","last_clean":"2024-09-16T04:28:04.457894+0000","last_became_active":"2024-09-16T04:11:11.970869+0000","last_became_peered":"2024-09-16T04:11:11.970869+0000","last_unstale":"2024-09-16T04:28:04.457894+0000","last_undegraded":"2024-09-16T04:28:04.457894+0000","last_fullsized":"2024-09-16T04:28:04.457894+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:04.457813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:04.457813+0000","last_clean_scrub_stamp":"2024-09-16T04:28:04.457813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:05.805336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":176,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:27:55.329292+0000","last_change":"2024-09-16T04:11:12.974381+0000","last_active":"2024-09-16T04:27:55.329292+0000","last_peered":"2024-09-16T04:27:55.329292+0000","last_clean":"2024-09-16T04:27:55.329292+0000","last_became_active":"2024-09-16T04:11:12.974021+0000","last_became_peered":"2024-09-16T04:11:12.974021+0000","last_unstale":"2024-09-16T04:27:55.329292+0000","last_undegraded":"2024-09-16T04:27:55.329292+0000","last_fullsized":"2024-09-16T04:27:55.329292+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:57.116331+0000","last_clean_scrub_stamp":"2024-09-16T04:09:57.116331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":178,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:13.400389+0000","last_change":"2024-09-16T04:11:11.972888+0000","last_active":"2024-09-16T04:25:13.400389+0000","last_peered":"2024-09-16T04:25:13.400389+0000","last_clean":"2024-09-16T04:25:13.400389+0000","last_became_active":"2024-09-16T04:11:11.972527+0000","last_became_peered":"2024-09-16T04:11:11.972527+0000","last_unstale":"2024-09-16T04:25:13.400389+0000","last_undegraded":"2024-09-16T04:25:13.400389+0000","last_fullsized":"2024-09-16T04:25:13.400389+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:09:56.111868+0000","last_clean_scrub_stamp":"2024-09-16T04:09:56.111868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:56:15.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":585,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:36.398725+0000","last_change":"2024-09-16T04:11:11.976349+0000","last_active":"2024-09-16T04:24:36.398725+0000","last_peered":"2024-09-16T04:24:36.398725+0000","last_clean":"2024-09-16T04:24:36.398725+0000","last_became_active":"2024-09-16T04:11:11.975960+0000","last_became_peered":"2024-09-16T04:11:11.975960+0000","last_unstale":"2024-09-16T04:24:36.398725+0000","last_undegraded":"2024-09-16T04:24:36.398725+0000","last_fullsized":"2024-09-16T04:24:36.398725+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:31.297948+0000","last_clean_scrub_stamp":"2024-09-16T04:06:31.297948+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:55.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":437,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308206+0000","last_change":"2024-09-16T04:11:11.978921+0000","last_active":"2024-09-16T04:25:46.308206+0000","last_peered":"2024-09-16T04:25:46.308206+0000","last_clean":"2024-09-16T04:25:46.308206+0000","last_became_active":"2024-09-16T04:11:11.971320+0000","last_became_peered":"2024-09-16T04:11:11.971320+0000","last_unstale":"2024-09-16T04:25:46.308206+0000","last_undegraded":"2024-09-16T04:25:46.308206+0000","last_fullsized":"2024-09-16T04:25:46.308206+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:13.147521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:41.399305+0000","last_change":"2024-09-16T04:11:11.975014+0000","last_active":"2024-09-16T04:24:41.399305+0000","last_peered":"2024-09-16T04:24:41.399305+0000","last_clean":"2024-09-16T04:24:41.399305+0000","last_became_active":"2024-09-16T04:11:11.974359+0000","last_became_peered":"2024-09-16T04:11:11.974359+0000","last_unstale":"2024-09-16T04:24:41.399305+0000","last_undegraded":"2024-09-16T04:24:41.399305+0000","last_fullsized":"2024-09-16T04:24:41.399305+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:29.591854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:27:55.329259+0000","last_change":"2024-09-16T04:11:11.973501+0000","last_active":"2024-09-16T04:27:55.329259+0000","last_peered":"2024-09-16T04:27:55.329259+0000","last_clean":"2024-09-16T04:27:55.329259+0000","last_became_active":"2024-09-16T04:11:11.973107+0000","last_became_peered":"2024-09-16T04:11:11.973107+0000","last_unstale":"2024-09-16T04:27:55.329259+0000","last_undegraded":"2024-09-16T04:27:55.329259+0000","last_fullsized":"2024-09-16T04:27:55.329259+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":448,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.308156+0000","last_change":"2024-09-16T04:11:11.984013+0000","last_active":"2024-09-16T04:25:46.308156+0000","last_peered":"2024-09-16T04:25:46.308156+0000","last_clean":"2024-09-16T04:25:46.308156+0000","last_became_active":"2024-09-16T04:11:11.975852+0000","last_became_peered":"2024-09-16T04:11:11.975852+0000","last_unstale":"2024-09-16T04:25:46.308156+0000","last_undegraded":"2024-09-16T04:25:46.308156+0000","last_fullsized":"2024-09-16T04:25:46.308156+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:19.909892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":438,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:48.403840+0000","last_change":"2024-09-16T04:11:12.969208+0000","last_active":"2024-09-16T04:25:48.403840+0000","last_peered":"2024-09-16T04:25:48.403840+0000","last_clean":"2024-09-16T04:25:48.403840+0000","last_became_active":"2024-09-16T04:11:12.968870+0000","last_became_peered":"2024-09-16T04:11:12.968870+0000","last_unstale":"2024-09-16T04:25:48.403840+0000","last_undegraded":"2024-09-16T04:25:48.403840+0000","last_fullsized":"2024-09-16T04:25:48.403840+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:14.254897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:25:46.405976+0000","last_change":"2024-09-16T04:11:11.976009+0000","last_active":"2024-09-16T04:25:46.405976+0000","last_peered":"2024-09-16T04:25:46.405976+0000","last_clean":"2024-09-16T04:25:46.405976+0000","last_became_active":"2024-09-16T04:11:11.975676+0000","last_became_peered":"2024-09-16T04:11:11.975676+0000","last_unstale":"2024-09-16T04:25:46.405976+0000","last_undegraded":"2024-09-16T04:25:46.405976+0000","last_fullsized":"2024-09-16T04:25:46.405976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:00:21.147770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:13.394779+0000","last_change":"2024-09-16T04:11:11.977786+0000","last_active":"2024-09-16T04:24:13.394779+0000","last_peered":"2024-09-16T04:24:13.394779+0000","last_clean":"2024-09-16T04:24:13.394779+0000","last_became_active":"2024-09-16T04:11:11.977456+0000","last_became_peered":"2024-09-16T04:11:11.977456+0000","last_unstale":"2024-09-16T04:24:13.394779+0000","last_undegraded":"2024-09-16T04:24:13.394779+0000","last_fullsized":"2024-09-16T04:24:13.394779+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:46.427728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":171,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:24:11.395976+0000","last_change":"2024-09-16T04:11:11.972539+0000","last_active":"2024-09-16T04:24:11.395976+0000","last_peered":"2024-09-16T04:24:11.395976+0000","last_clean":"2024-09-16T04:24:11.395976+0000","last_became_active":"2024-09-16T04:11:11.972183+0000","last_became_peered":"2024-09-16T04:11:11.972183+0000","last_unstale":"2024-09-16T04:24:11.395976+0000","last_undegraded":"2024-09-16T04:24:11.395976+0000","last_fullsized":"2024-09-16T04:24:11.395976+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:06:38.328980+0000","last_clean_scrub_stamp":"2024-09-16T04:06:38.328980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:54.054660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575112,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34224,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337616,"statfs":{"total":96636764160,"available":96601718784,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575113,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34228,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337612,"statfs":{"total":96636764160,"available":96601714688,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30068,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341772,"statfs":{"total":96636764160,"available":96605974528,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640518,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30064,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341776,"statfs":{"total":96636764160,"available":96605978624,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640517,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:28:06.530 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.530 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.530 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.530 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.530 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.531 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.532 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-16T04:09:57.116331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=57, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.532 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-16T04:09:56.111868+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=9, tm_sec=56, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.532 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T04:06:31.297948+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.532 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.532 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.532 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.533 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.533 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.533 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.533 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.533 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T04:06:38.328980+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=6, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=28, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:28:06.533 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:28:26.534 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:28:26.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.642+0000 7ff9fad9d640 10 monclient: get_monmap_and_config 2024-09-16T04:28:26.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.642+0000 7ff9fad9d640 10 monclient: build_initial_monmap 2024-09-16T04:28:26.644 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.642+0000 7ff9fad9d640 10 monclient: monmap: 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:26.643801+0000 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:26.643801+0000 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:26.645 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.643+0000 7ff9fad9d640 1 Processor -- start 2024-09-16T04:28:26.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.643+0000 7ff9fad9d640 1 -- start start 2024-09-16T04:28:26.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.643+0000 7ff9fad9d640 10 monclient: init 2024-09-16T04:28:26.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:26.654 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f4136940 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f4139700 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f413d210 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f40a0090 con 0x7ff9f4139700 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f406ef40 con 0x7ff9f4136940 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f406c1d0 con 0x7ff9f413d210 2024-09-16T04:28:26.655 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 10 monclient(hunting): authenticate will time out at 2127.269332s 2024-09-16T04:28:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fa59c640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff9f413d210 0x7ff9f413f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39170/0 (socket says 172.21.15.148:39170) 2024-09-16T04:28:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fa59c640 1 -- 172.21.15.148:0/1177942144 learned_addr learned my addr 172.21.15.148:0/1177942144 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f959a640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff9f4139700 0x7ff9f413baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52902/0 (socket says 172.21.15.148:52902) 2024-09-16T04:28:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2858688531 0 0) 0x7ff9f406c1d0 con 0x7ff9f413d210 2024-09-16T04:28:26.656 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): my global_id is 154484 2024-09-16T04:28:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9e0003520 con 0x7ff9f413d210 2024-09-16T04:28:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375152115 0 0) 0x7ff9f40a0090 con 0x7ff9f4139700 2024-09-16T04:28:26.657 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient(hunting): my global_id is 174234 2024-09-16T04:28:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9f406c1d0 con 0x7ff9f4139700 2024-09-16T04:28:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 157748653 0 0) 0x7ff9e0003520 con 0x7ff9f413d210 2024-09-16T04:28:26.658 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 >> v1:172.21.15.171:6789/0 conn(0x7ff9f4136940 legacy=0x7ff9f4136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 >> v1:172.21.15.148:6789/0 conn(0x7ff9f4139700 legacy=0x7ff9f413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient: _finish_hunting 0 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 monclient: found mon.noname-b 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.148:6790/0 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f406c650 con 0x7ff9f413d210 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient: _finish_auth 0 2024-09-16T04:28:26.659 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 10 monclient: _check_auth_tickets 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 5 monclient: authenticate success, global_id 154484 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff9ec001990 con 0x7ff9f413d210 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.644+0000 7ff9fad9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:28:26.660 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: dump: 2024-09-16T04:28:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.661 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.662 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:26.662 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 <== mon.2 v1:172.21.15.148:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9ec001b80 con 0x7ff9f413d210 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 1 -- 172.21.15.148:0/1177942144 <== mon.2 v1:172.21.15.148:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff9ec001f90 con 0x7ff9f413d210 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 4 set_mon_vals no callback set 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9f8d99640 10 monclient: dump: 2024-09-16T04:28:26.663 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 10 monclient: shutdown 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 1 -- 172.21.15.148:0/1177942144 >> v1:172.21.15.148:6790/0 conn(0x7ff9f413d210 legacy=0x7ff9f413f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.664 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 1 -- 172.21.15.148:0/1177942144 shutdown_connections 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 1 -- 172.21.15.148:0/1177942144 >> 172.21.15.148:0/1177942144 conn(0x7ff9f40a62f0 msgr2=0x7ff9f40a2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 1 -- 172.21.15.148:0/1177942144 shutdown_connections 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.645+0000 7ff9fad9d640 1 -- 172.21.15.148:0/1177942144 wait complete. 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.646+0000 7ff9fad9d640 10 monclient: build_initial_monmap 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.646+0000 7ff9fad9d640 10 monclient: monmap: 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:26.665 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.646+0000 7ff9fad9d640 1 Processor -- start 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.646+0000 7ff9fad9d640 1 -- start start 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.646+0000 7ff9fad9d640 10 monclient: init 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f4136940 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f4139700 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f413d210 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:26.666 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f413c170 con 0x7ff9f4139700 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f411ebb0 con 0x7ff9f413d210 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f411dd40 con 0x7ff9f4136940 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9fad9d640 10 monclient(hunting): authenticate will time out at 2127.272348s 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9f9d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7ff9f4136940 0x7ff9f41cb8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39180/0 (socket says 172.21.15.148:39180) 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9f9d9b640 1 -- 172.21.15.148:0/3721626928 learned_addr learned my addr 172.21.15.148:0/3721626928 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 853991422 0 0) 0x7ff9f413c170 con 0x7ff9f4139700 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.667 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): my global_id is 174237 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9d8003520 con 0x7ff9f4139700 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220919621 0 0) 0x7ff9f411dd40 con 0x7ff9f4136940 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): my global_id is 154487 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9f413c170 con 0x7ff9f4136940 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 329623735 0 0) 0x7ff9d8003520 con 0x7ff9f4139700 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9f411dd40 con 0x7ff9f4139700 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff9e4001310 con 0x7ff9f4139700 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:26.668 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4242658486 0 0) 0x7ff9f413c170 con 0x7ff9f4136940 2024-09-16T04:28:26.669 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9d8003520 con 0x7ff9f4136940 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff9f00018b0 con 0x7ff9f4136940 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): dump: 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.670 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236477883 0 0) 0x7ff9f411ebb0 con 0x7ff9f413d210 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 10 monclient(hunting): my global_id is 164365 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.647+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9f413c170 con 0x7ff9f413d210 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2537246904 0 0) 0x7ff9f411dd40 con 0x7ff9f4139700 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 >> v1:172.21.15.148:6790/0 conn(0x7ff9f4136940 legacy=0x7ff9f41cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 >> v1:172.21.15.171:6789/0 conn(0x7ff9f413d210 legacy=0x7ff9f41d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 monclient: found mon.a 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:26.671 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f411e8d0 con 0x7ff9f4139700 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: _finish_auth 0 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 5 monclient: authenticate success, global_id 174237 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 10 monclient: _renew_subs 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f4133840 con 0x7ff9f4139700 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 10 monclient: _renew_subs 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9f41cbfd0 con 0x7ff9f4139700 2024-09-16T04:28:26.672 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff9e4001c70 con 0x7ff9f4139700 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7ff9e4004a00 con 0x7ff9f4139700 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ebfff640 4 set_mon_vals no callback set 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 10 monclient: dump: 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.673 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.648+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7ff9e400fe30 con 0x7ff9f4139700 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.649+0000 7ff9fad9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.649+0000 7ff9fad9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.649+0000 7ff9fad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.148:6789/0 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.649+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.148:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9dc005080 con 0x7ff9f4139700 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.649+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7ff9e4047950 con 0x7ff9f4139700 2024-09-16T04:28:26.674 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.649+0000 7ff9f9d9b640 10 monclient: get_auth_request con 0x7ff9d80390e0 auth_method 0 2024-09-16T04:28:26.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.653+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mon.0 v1:172.21.15.148:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7ff9e4015b50 con 0x7ff9f4139700 2024-09-16T04:28:26.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.653+0000 7ff9ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:26.675 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.653+0000 7ff9ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:26.817 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.816+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 --> v1:172.21.15.171:6812/7327273 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9dc000b90 con 0x7ff9d80390e0 2024-09-16T04:28:26.820 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.819+0000 7ff9ea7fc640 1 -- 172.21.15.148:0/3721626928 <== mgr.4104 v1:172.21.15.171:6812/7327273 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+78004 (unknown 2965378022 0 444087724) 0x7ff9dc000b90 con 0x7ff9d80390e0 2024-09-16T04:28:26.820 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-16T04:28:26.821 INFO:teuthology.orchestra.run.smithi148.stderr:dumped all 2024-09-16T04:28:26.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.820+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 >> v1:172.21.15.171:6812/7327273 conn(0x7ff9d80390e0 legacy=0x7ff9d803b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.822 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.820+0000 7ff9fad9d640 10 monclient: shutdown 2024-09-16T04:28:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.820+0000 7ff9fad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.820+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 >> v1:172.21.15.148:6789/0 conn(0x7ff9f4139700 legacy=0x7ff9f41d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.820+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 shutdown_connections 2024-09-16T04:28:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.821+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 >> 172.21.15.148:0/3721626928 conn(0x7ff9f40a62f0 msgr2=0x7ff9f4135e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.821+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 shutdown_connections 2024-09-16T04:28:26.823 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.821+0000 7ff9fad9d640 1 -- 172.21.15.148:0/3721626928 wait complete. 2024-09-16T04:28:26.833 INFO:teuthology.orchestra.run.smithi148.stdout:{"pg_ready":true,"pg_map":{"version":909,"stamp":"2024-09-16T04:28:25.555134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":196392,"kb_used_data":17608,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566034648,"statfs":{"total":579820584960,"available":579619479552,"internally_reserved":0,"allocated":18030592,"data_stored":15149110,"data_compressed":1198654,"data_compressed_allocated":2351104,"data_compressed_original":6012928,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002873"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":315,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:06.403546+0000","last_change":"2024-09-16T04:28:06.403584+0000","last_active":"2024-09-16T04:28:06.403546+0000","last_peered":"2024-09-16T04:28:06.403546+0000","last_clean":"2024-09-16T04:28:06.403546+0000","last_became_active":"2024-09-16T04:11:11.972654+0000","last_became_peered":"2024-09-16T04:11:11.972654+0000","last_unstale":"2024-09-16T04:28:06.403546+0000","last_undegraded":"2024-09-16T04:28:06.403546+0000","last_fullsized":"2024-09-16T04:28:06.403546+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:06.403456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:06.403456+0000","last_clean_scrub_stamp":"2024-09-16T04:28:06.403456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:01.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":184,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:07.054684+0000","last_change":"2024-09-16T04:28:07.054710+0000","last_active":"2024-09-16T04:28:07.054684+0000","last_peered":"2024-09-16T04:28:07.054684+0000","last_clean":"2024-09-16T04:28:07.054684+0000","last_became_active":"2024-09-16T04:11:11.970821+0000","last_became_peered":"2024-09-16T04:11:11.970821+0000","last_unstale":"2024-09-16T04:28:07.054684+0000","last_undegraded":"2024-09-16T04:28:07.054684+0000","last_fullsized":"2024-09-16T04:28:07.054684+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:07.054596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:07.054596+0000","last_clean_scrub_stamp":"2024-09-16T04:28:07.054596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:33.537354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":288,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:05.057488+0000","last_change":"2024-09-16T04:28:05.057522+0000","last_active":"2024-09-16T04:28:05.057488+0000","last_peered":"2024-09-16T04:28:05.057488+0000","last_clean":"2024-09-16T04:28:05.057488+0000","last_became_active":"2024-09-16T04:11:11.971647+0000","last_became_peered":"2024-09-16T04:11:11.971647+0000","last_unstale":"2024-09-16T04:28:05.057488+0000","last_undegraded":"2024-09-16T04:28:05.057488+0000","last_fullsized":"2024-09-16T04:28:05.057488+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:05.057386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:05.057386+0000","last_clean_scrub_stamp":"2024-09-16T04:28:05.057386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.670217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":184,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:09.112146+0000","last_change":"2024-09-16T04:28:09.112171+0000","last_active":"2024-09-16T04:28:09.112146+0000","last_peered":"2024-09-16T04:28:09.112146+0000","last_clean":"2024-09-16T04:28:09.112146+0000","last_became_active":"2024-09-16T04:11:11.974022+0000","last_became_peered":"2024-09-16T04:11:11.974022+0000","last_unstale":"2024-09-16T04:28:09.112146+0000","last_undegraded":"2024-09-16T04:28:09.112146+0000","last_fullsized":"2024-09-16T04:28:09.112146+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:09.112048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:09.112048+0000","last_clean_scrub_stamp":"2024-09-16T04:28:09.112048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:58.500923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":233,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:12.081378+0000","last_change":"2024-09-16T04:28:12.081403+0000","last_active":"2024-09-16T04:28:12.081378+0000","last_peered":"2024-09-16T04:28:12.081378+0000","last_clean":"2024-09-16T04:28:12.081378+0000","last_became_active":"2024-09-16T04:11:12.968688+0000","last_became_peered":"2024-09-16T04:11:12.968688+0000","last_unstale":"2024-09-16T04:28:12.081378+0000","last_undegraded":"2024-09-16T04:28:12.081378+0000","last_fullsized":"2024-09-16T04:28:12.081378+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:12.081289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:12.081289+0000","last_clean_scrub_stamp":"2024-09-16T04:28:12.081289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:37:32.383001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":303,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:05.421358+0000","last_change":"2024-09-16T04:28:05.421379+0000","last_active":"2024-09-16T04:28:05.421358+0000","last_peered":"2024-09-16T04:28:05.421358+0000","last_clean":"2024-09-16T04:28:05.421358+0000","last_became_active":"2024-09-16T04:11:11.971583+0000","last_became_peered":"2024-09-16T04:11:11.971583+0000","last_unstale":"2024-09-16T04:28:05.421358+0000","last_undegraded":"2024-09-16T04:28:05.421358+0000","last_fullsized":"2024-09-16T04:28:05.421358+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:05.421293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:05.421293+0000","last_clean_scrub_stamp":"2024-09-16T04:28:05.421293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:39:01.087168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":181,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:09.148106+0000","last_change":"2024-09-16T04:28:09.148129+0000","last_active":"2024-09-16T04:28:09.148106+0000","last_peered":"2024-09-16T04:28:09.148106+0000","last_clean":"2024-09-16T04:28:09.148106+0000","last_became_active":"2024-09-16T04:11:11.975180+0000","last_became_peered":"2024-09-16T04:11:11.975180+0000","last_unstale":"2024-09-16T04:28:09.148106+0000","last_undegraded":"2024-09-16T04:28:09.148106+0000","last_fullsized":"2024-09-16T04:28:09.148106+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:09.147987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:09.147987+0000","last_clean_scrub_stamp":"2024-09-16T04:28:09.147987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:29.386438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":187,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:04.603738+0000","last_change":"2024-09-16T04:28:04.603759+0000","last_active":"2024-09-16T04:28:04.603738+0000","last_peered":"2024-09-16T04:28:04.603738+0000","last_clean":"2024-09-16T04:28:04.603738+0000","last_became_active":"2024-09-16T04:11:11.970247+0000","last_became_peered":"2024-09-16T04:11:11.970247+0000","last_unstale":"2024-09-16T04:28:04.603738+0000","last_undegraded":"2024-09-16T04:28:04.603738+0000","last_fullsized":"2024-09-16T04:28:04.603738+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:04.603661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:04.603661+0000","last_clean_scrub_stamp":"2024-09-16T04:28:04.603661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:09.197451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":218,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:08.100391+0000","last_change":"2024-09-16T04:28:08.100391+0000","last_active":"2024-09-16T04:28:08.100391+0000","last_peered":"2024-09-16T04:28:08.100391+0000","last_clean":"2024-09-16T04:28:08.100391+0000","last_became_active":"2024-09-16T04:11:11.973162+0000","last_became_peered":"2024-09-16T04:11:11.973162+0000","last_unstale":"2024-09-16T04:28:08.100391+0000","last_undegraded":"2024-09-16T04:28:08.100391+0000","last_fullsized":"2024-09-16T04:28:08.100391+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:08.100279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:08.100279+0000","last_clean_scrub_stamp":"2024-09-16T04:28:08.100279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:39:25.096872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":217,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:08.124004+0000","last_change":"2024-09-16T04:28:08.124027+0000","last_active":"2024-09-16T04:28:08.124004+0000","last_peered":"2024-09-16T04:28:08.124004+0000","last_clean":"2024-09-16T04:28:08.124004+0000","last_became_active":"2024-09-16T04:11:11.974614+0000","last_became_peered":"2024-09-16T04:11:11.974614+0000","last_unstale":"2024-09-16T04:28:08.124004+0000","last_undegraded":"2024-09-16T04:28:08.124004+0000","last_fullsized":"2024-09-16T04:28:08.124004+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:08.123923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:08.123923+0000","last_clean_scrub_stamp":"2024-09-16T04:28:08.123923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:10:01.450120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":241,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:06.488516+0000","last_change":"2024-09-16T04:28:06.488545+0000","last_active":"2024-09-16T04:28:06.488516+0000","last_peered":"2024-09-16T04:28:06.488516+0000","last_clean":"2024-09-16T04:28:06.488516+0000","last_became_active":"2024-09-16T04:11:11.972700+0000","last_became_peered":"2024-09-16T04:11:11.972700+0000","last_unstale":"2024-09-16T04:28:06.488516+0000","last_undegraded":"2024-09-16T04:28:06.488516+0000","last_fullsized":"2024-09-16T04:28:06.488516+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:06.488427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:06.488427+0000","last_clean_scrub_stamp":"2024-09-16T04:28:06.488427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:22:41.074117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":187,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:05.555888+0000","last_change":"2024-09-16T04:28:05.555924+0000","last_active":"2024-09-16T04:28:05.555888+0000","last_peered":"2024-09-16T04:28:05.555888+0000","last_clean":"2024-09-16T04:28:05.555888+0000","last_became_active":"2024-09-16T04:11:11.972431+0000","last_became_peered":"2024-09-16T04:11:11.972431+0000","last_unstale":"2024-09-16T04:28:05.555888+0000","last_undegraded":"2024-09-16T04:28:05.555888+0000","last_fullsized":"2024-09-16T04:28:05.555888+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:05.555822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:05.555822+0000","last_clean_scrub_stamp":"2024-09-16T04:28:05.555822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:51.935524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":188,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:07.563014+0000","last_change":"2024-09-16T04:28:07.563040+0000","last_active":"2024-09-16T04:28:07.563014+0000","last_peered":"2024-09-16T04:28:07.563014+0000","last_clean":"2024-09-16T04:28:07.563014+0000","last_became_active":"2024-09-16T04:11:11.972513+0000","last_became_peered":"2024-09-16T04:11:11.972513+0000","last_unstale":"2024-09-16T04:28:07.563014+0000","last_undegraded":"2024-09-16T04:28:07.563014+0000","last_fullsized":"2024-09-16T04:28:07.563014+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:07.562920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:07.562920+0000","last_clean_scrub_stamp":"2024-09-16T04:28:07.562920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:19.609888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":316,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:04.457894+0000","last_change":"2024-09-16T04:28:04.457943+0000","last_active":"2024-09-16T04:28:04.457894+0000","last_peered":"2024-09-16T04:28:04.457894+0000","last_clean":"2024-09-16T04:28:04.457894+0000","last_became_active":"2024-09-16T04:11:11.970869+0000","last_became_peered":"2024-09-16T04:11:11.970869+0000","last_unstale":"2024-09-16T04:28:04.457894+0000","last_undegraded":"2024-09-16T04:28:04.457894+0000","last_fullsized":"2024-09-16T04:28:04.457894+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:04.457813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:04.457813+0000","last_clean_scrub_stamp":"2024-09-16T04:28:04.457813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:12:05.805336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":184,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:06.986207+0000","last_change":"2024-09-16T04:28:06.986232+0000","last_active":"2024-09-16T04:28:06.986207+0000","last_peered":"2024-09-16T04:28:06.986207+0000","last_clean":"2024-09-16T04:28:06.986207+0000","last_became_active":"2024-09-16T04:11:12.974021+0000","last_became_peered":"2024-09-16T04:11:12.974021+0000","last_unstale":"2024-09-16T04:28:06.986207+0000","last_undegraded":"2024-09-16T04:28:06.986207+0000","last_fullsized":"2024-09-16T04:28:06.986207+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:06.986127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:06.986127+0000","last_clean_scrub_stamp":"2024-09-16T04:28:06.986127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:31:37.799540+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":187,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:06.531499+0000","last_change":"2024-09-16T04:28:06.531532+0000","last_active":"2024-09-16T04:28:06.531499+0000","last_peered":"2024-09-16T04:28:06.531499+0000","last_clean":"2024-09-16T04:28:06.531499+0000","last_became_active":"2024-09-16T04:11:11.972527+0000","last_became_peered":"2024-09-16T04:11:11.972527+0000","last_unstale":"2024-09-16T04:28:06.531499+0000","last_undegraded":"2024-09-16T04:28:06.531499+0000","last_fullsized":"2024-09-16T04:28:06.531499+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:06.531390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:06.531390+0000","last_clean_scrub_stamp":"2024-09-16T04:28:06.531390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:34:51.584647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"257'119","reported_seq":596,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:11.221183+0000","last_change":"2024-09-16T04:28:11.221227+0000","last_active":"2024-09-16T04:28:11.221183+0000","last_peered":"2024-09-16T04:28:11.221183+0000","last_clean":"2024-09-16T04:28:11.221183+0000","last_became_active":"2024-09-16T04:11:11.975960+0000","last_became_peered":"2024-09-16T04:11:11.975960+0000","last_unstale":"2024-09-16T04:28:11.221183+0000","last_undegraded":"2024-09-16T04:28:11.221183+0000","last_fullsized":"2024-09-16T04:28:11.221183+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'119","last_scrub_stamp":"2024-09-16T04:28:11.221089+0000","last_deep_scrub":"257'119","last_deep_scrub_stamp":"2024-09-16T04:28:11.221089+0000","last_clean_scrub_stamp":"2024-09-16T04:28:11.221089+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:28.123600+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"257'2","reported_seq":447,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:10.113449+0000","last_change":"2024-09-16T04:28:10.113497+0000","last_active":"2024-09-16T04:28:10.113449+0000","last_peered":"2024-09-16T04:28:10.113449+0000","last_clean":"2024-09-16T04:28:10.113449+0000","last_became_active":"2024-09-16T04:11:11.971320+0000","last_became_peered":"2024-09-16T04:11:11.971320+0000","last_unstale":"2024-09-16T04:28:10.113449+0000","last_undegraded":"2024-09-16T04:28:10.113449+0000","last_fullsized":"2024-09-16T04:28:10.113449+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'2","last_scrub_stamp":"2024-09-16T04:28:10.113344+0000","last_deep_scrub":"257'2","last_deep_scrub_stamp":"2024-09-16T04:28:10.113344+0000","last_clean_scrub_stamp":"2024-09-16T04:28:10.113344+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:01:47.656814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00049426699999999995,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":495,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:07.087137+0000","last_change":"2024-09-16T04:28:07.087160+0000","last_active":"2024-09-16T04:28:07.087137+0000","last_peered":"2024-09-16T04:28:07.087137+0000","last_clean":"2024-09-16T04:28:07.087137+0000","last_became_active":"2024-09-16T04:11:11.974359+0000","last_became_peered":"2024-09-16T04:11:11.974359+0000","last_unstale":"2024-09-16T04:28:07.087137+0000","last_undegraded":"2024-09-16T04:28:07.087137+0000","last_fullsized":"2024-09-16T04:28:07.087137+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:07.087047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:07.087047+0000","last_clean_scrub_stamp":"2024-09-16T04:28:07.087047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:33:07.834914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000387735,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:06.031302+0000","last_change":"2024-09-16T04:28:06.031344+0000","last_active":"2024-09-16T04:28:06.031302+0000","last_peered":"2024-09-16T04:28:06.031302+0000","last_clean":"2024-09-16T04:28:06.031302+0000","last_became_active":"2024-09-16T04:11:11.973107+0000","last_became_peered":"2024-09-16T04:11:11.973107+0000","last_unstale":"2024-09-16T04:28:06.031302+0000","last_undegraded":"2024-09-16T04:28:06.031302+0000","last_fullsized":"2024-09-16T04:28:06.031302+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:06.031207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:06.031207+0000","last_clean_scrub_stamp":"2024-09-16T04:28:06.031207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:41:13.877003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049557900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"257'3","reported_seq":458,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:11.084023+0000","last_change":"2024-09-16T04:28:11.084076+0000","last_active":"2024-09-16T04:28:11.084023+0000","last_peered":"2024-09-16T04:28:11.084023+0000","last_clean":"2024-09-16T04:28:11.084023+0000","last_became_active":"2024-09-16T04:11:11.975852+0000","last_became_peered":"2024-09-16T04:11:11.975852+0000","last_unstale":"2024-09-16T04:28:11.084023+0000","last_undegraded":"2024-09-16T04:28:11.084023+0000","last_fullsized":"2024-09-16T04:28:11.084023+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'3","last_scrub_stamp":"2024-09-16T04:28:11.083914+0000","last_deep_scrub":"257'3","last_deep_scrub_stamp":"2024-09-16T04:28:11.083914+0000","last_clean_scrub_stamp":"2024-09-16T04:28:11.083914+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:06.988506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00049720999999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:09.573100+0000","last_change":"2024-09-16T04:28:09.573125+0000","last_active":"2024-09-16T04:28:09.573100+0000","last_peered":"2024-09-16T04:28:09.573100+0000","last_clean":"2024-09-16T04:28:09.573100+0000","last_became_active":"2024-09-16T04:11:12.968870+0000","last_became_peered":"2024-09-16T04:11:12.968870+0000","last_unstale":"2024-09-16T04:28:09.573100+0000","last_undegraded":"2024-09-16T04:28:09.573100+0000","last_fullsized":"2024-09-16T04:28:09.573100+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:09.572992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:09.572992+0000","last_clean_scrub_stamp":"2024-09-16T04:28:09.572992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:24:06.092354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000420233,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":181,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:10.168276+0000","last_change":"2024-09-16T04:28:10.168299+0000","last_active":"2024-09-16T04:28:10.168276+0000","last_peered":"2024-09-16T04:28:10.168276+0000","last_clean":"2024-09-16T04:28:10.168276+0000","last_became_active":"2024-09-16T04:11:11.975676+0000","last_became_peered":"2024-09-16T04:11:11.975676+0000","last_unstale":"2024-09-16T04:28:10.168276+0000","last_undegraded":"2024-09-16T04:28:10.168276+0000","last_fullsized":"2024-09-16T04:28:10.168276+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:10.168197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:10.168197+0000","last_clean_scrub_stamp":"2024-09-16T04:28:10.168197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:16:26.365935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057685699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":186,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:08.567710+0000","last_change":"2024-09-16T04:28:08.567736+0000","last_active":"2024-09-16T04:28:08.567710+0000","last_peered":"2024-09-16T04:28:08.567710+0000","last_clean":"2024-09-16T04:28:08.567710+0000","last_became_active":"2024-09-16T04:11:11.977456+0000","last_became_peered":"2024-09-16T04:11:11.977456+0000","last_unstale":"2024-09-16T04:28:08.567710+0000","last_undegraded":"2024-09-16T04:28:08.567710+0000","last_fullsized":"2024-09-16T04:28:08.567710+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:08.567618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:08.567618+0000","last_clean_scrub_stamp":"2024-09-16T04:28:08.567618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:51:27.240685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040592100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":180,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-16T04:28:06.102430+0000","last_change":"2024-09-16T04:28:06.102456+0000","last_active":"2024-09-16T04:28:06.102430+0000","last_peered":"2024-09-16T04:28:06.102430+0000","last_clean":"2024-09-16T04:28:06.102430+0000","last_became_active":"2024-09-16T04:11:11.972183+0000","last_became_peered":"2024-09-16T04:11:11.972183+0000","last_unstale":"2024-09-16T04:28:06.102430+0000","last_undegraded":"2024-09-16T04:28:06.102430+0000","last_fullsized":"2024-09-16T04:28:06.102430+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:28:06.102324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:28:06.102324+0000","last_clean_scrub_stamp":"2024-09-16T04:28:06.102324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:04.542358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000384943,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10816,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575116,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34100,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337740,"statfs":{"total":96636764160,"available":96601845760,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34104,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337736,"statfs":{"total":96636764160,"available":96601841664,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29944,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341896,"statfs":{"total":96636764160,"available":96606101504,"internally_reserved":0,"allocated":2990080,"data_stored":2348555,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640522,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34152,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337688,"statfs":{"total":96636764160,"available":96601792512,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29940,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341900,"statfs":{"total":96636764160,"available":96606105600,"internally_reserved":0,"allocated":2985984,"data_stored":2348536,"data_compressed":197973,"data_compressed_allocated":380928,"data_compressed_original":831488,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640521,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34152,"kb_used_data":2968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337688,"statfs":{"total":96636764160,"available":96601792512,"internally_reserved":0,"allocated":3039232,"data_stored":2877464,"data_compressed":203381,"data_compressed_allocated":413696,"data_compressed_original":1343488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5408,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:28:26.835 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.931+0000 7f99d0fa6640 10 monclient: get_monmap_and_config 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.931+0000 7f99d0fa6640 10 monclient: build_initial_monmap 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.931+0000 7f99d0fa6640 10 monclient: monmap: 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:28:26.932528+0000 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:28:26.932528+0000 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.933 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:26.934 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-c 2024-09-16T04:28:26.934 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-b 2024-09-16T04:28:26.934 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 1 Processor -- start 2024-09-16T04:28:26.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 1 -- start start 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient: init 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient(hunting): picked mon.noname-c con 0x7f99cc09bb70 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient(hunting): picked mon.noname-b con 0x7f99cc09c660 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient(hunting): picked mon.noname-a con 0x7f99cc09cc30 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc0a1010 con 0x7f99cc09cc30 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc09f030 con 0x7f99cc09bb70 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc09dc60 con 0x7f99cc09c660 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:26.942 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99d0fa6640 10 monclient(hunting): authenticate will time out at 2127.557902s 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.932+0000 7f99c9d9b640 1 --1- >> v1:172.21.15.148:6790/0 conn(0x7f99cc09c660 0x7f99cc138960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6790/0 says I am v1:172.21.15.148:39182/0 (socket says 172.21.15.148:39182) 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c9d9b640 1 -- 172.21.15.148:0/780432332 learned_addr learned my addr 172.21.15.148:0/780432332 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99cad9d640 1 --1- 172.21.15.148:0/780432332 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09cc30 0x7f99cc13c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52930/0 (socket says 172.21.15.148:52930) 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238808666 0 0) 0x7f99cc09dc60 con 0x7f99cc09c660 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): my global_id is 154490 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b4003520 con 0x7f99cc09c660 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.0 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618179501 0 0) 0x7f99cc0a1010 con 0x7f99cc09cc30 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.943 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): my global_id is 174240 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99cc09dc60 con 0x7f99cc09cc30 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.1 v1:172.21.15.171:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727830282 0 0) 0x7f99cc09f030 con 0x7f99cc09bb70 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient(hunting): my global_id is 164368 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 --> v1:172.21.15.171:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99cc0a1010 con 0x7f99cc09bb70 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.0 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2841932345 0 0) 0x7f99cc09dc60 con 0x7f99cc09cc30 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 >> v1:172.21.15.148:6790/0 conn(0x7f99cc09c660 legacy=0x7f99cc138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 >> v1:172.21.15.171:6789/0 conn(0x7f99cc09bb70 legacy=0x7f99cc09bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: _finish_hunting 0 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 monclient: found mon.noname-a 2024-09-16T04:28:26.944 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.148:6789/0 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc09df40 con 0x7f99cc09cc30 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: _finish_auth 0 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: _check_auth_tickets 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.0 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99c40018f0 con 0x7f99cc09cc30 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99d0fa6640 5 monclient: authenticate success, global_id 174240 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99d0fa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: dump: 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.945 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99d0fa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.0 v1:172.21.15.148:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f99c4001b50 con 0x7f99cc09cc30 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99d0fa6640 10 monclient: get_monmap_and_config success 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 1 -- 172.21.15.148:0/780432332 <== mon.0 v1:172.21.15.148:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99c4001f20 con 0x7f99cc09cc30 2024-09-16T04:28:26.946 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99d0fa6640 4 set_mon_vals no callback set 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.933+0000 7f99c959a640 10 monclient: dump: 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.947 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 10 monclient: shutdown 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 1 -- 172.21.15.148:0/780432332 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09cc30 legacy=0x7f99cc13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 1 -- 172.21.15.148:0/780432332 shutdown_connections 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 1 -- 172.21.15.148:0/780432332 >> 172.21.15.148:0/780432332 conn(0x7f99cc0a5640 msgr2=0x7f99cc0a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 1 -- 172.21.15.148:0/780432332 shutdown_connections 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 1 -- 172.21.15.148:0/780432332 wait complete. 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 10 monclient: build_initial_monmap 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.934+0000 7f99d0fa6640 10 monclient: monmap: 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 0 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 0.000000 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:created 0.000000 2024-09-16T04:28:26.948 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 0 (unknown) 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.noname-a 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.noname-b 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.noname-c 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 1 Processor -- start 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 1 -- start start 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient: init 2024-09-16T04:28:26.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient: _reopen_session rank -1 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient(hunting): picked mon.noname-a con 0x7f99cc09bb70 addr v1:172.21.15.148:6789/0 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient(hunting): picked mon.noname-b con 0x7f99cc09c660 addr v1:172.21.15.171:6789/0 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient(hunting): picked mon.noname-c con 0x7f99cc09cc30 addr v1:172.21.15.148:6790/0 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc11e810 con 0x7f99cc09bb70 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 1 -- --> v1:172.21.15.171:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc11c060 con 0x7f99cc09c660 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 1 -- --> v1:172.21.15.148:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc11eaf0 con 0x7f99cc09cc30 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient(hunting): _renew_subs 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99d0fa6640 10 monclient(hunting): authenticate will time out at 2127.560739s 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99ca59c640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f99cc09bb70 0x7f99cc11b240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:52944/0 (socket says 172.21.15.148:52944) 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99ca59c640 1 -- 172.21.15.148:0/297746911 learned_addr learned my addr 172.21.15.148:0/297746911 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99ca59c640 1 -- 172.21.15.148:0/297746911 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09bb70 legacy=0x7f99cc11b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99ca59c640 1 -- 172.21.15.148:0/297746911 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09bb70 legacy=0x7f99cc11b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:28:26.950 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99ca59c640 1 --1- 172.21.15.148:0/297746911 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09bb70 0x7f99cc11b240 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99ca59c640 1 --1- 172.21.15.148:0/297746911 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09bb70 0x7f99cc11b240 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99baffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.148:6789/0 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2721013379 0 0) 0x7f99cc11eaf0 con 0x7f99cc09cc30 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99baffd640 10 monclient(hunting): _negotiate 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.935+0000 7f99baffd640 10 monclient(hunting): my global_id is 154493 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99ac003520 con 0x7f99cc09cc30 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 294176267 0 0) 0x7f99ac003520 con 0x7f99cc09cc30 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99cc11eaf0 con 0x7f99cc09cc30 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99c40017f0 con 0x7f99cc09cc30 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:28:26.951 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient(hunting): dump: 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.952 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 412426115 0 0) 0x7f99cc11eaf0 con 0x7f99cc09cc30 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 >> v1:172.21.15.171:6789/0 conn(0x7f99cc09c660 legacy=0x7f99cc11b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 >> v1:172.21.15.148:6789/0 conn(0x7f99cc09bb70 legacy=0x7f99cc11b240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: _finish_hunting 0 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 monclient: found mon.c 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc13cc20 con 0x7f99cc09cc30 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: _finish_auth 0 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: _check_auth_tickets 2024-09-16T04:28:26.953 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 5 monclient: authenticate success, global_id 154493 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 10 monclient: _renew_subs 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc0a3d90 con 0x7f99cc09cc30 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 10 monclient: _renew_subs 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f99cc11e530 con 0x7f99cc09cc30 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f99c4001b50 con 0x7f99cc09cc30 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2815707521 0 0) 0x7f99c4004fb0 con 0x7f99cc09cc30 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:28:26.954 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99baffd640 10 monclient: dump: 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:epoch 1 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:fsid bc50e21b-589f-47d0-85d5-a865de7d0d68 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:last_changed 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:created 2024-09-16T04:05:49.094408+0000 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:min_mon_release 19 (squid) 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:election_strategy: 1 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:0: v1:172.21.15.148:6789/0 mon.a 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:1: v1:172.21.15.171:6789/0 mon.b 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2: v1:172.21.15.148:6790/0 mon.c 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.936+0000 7f99c8d99640 4 set_mon_vals no callback set 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 7 ==== mgrmap(e 5) ==== 45702+0+0 (unknown 2829788778 0 0) 0x7f99c40103e0 con 0x7f99cc09cc30 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99d0fa6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:26.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99d0fa6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99d0fa6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b0005080 con 0x7f99cc09cc30 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 8 ==== osd_map(257..257 src has 232..257) ==== 6817+0+0 (unknown 769199190 0 0) 0x7f99c4047d50 con 0x7f99cc09cc30 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.937+0000 7f99ca59c640 10 monclient: get_auth_request con 0x7f99ac038ec0 auth_method 0 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.942+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (unknown 1092875540 0 2179354791) 0x7f99c4015fb0 con 0x7f99cc09cc30 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.942+0000 7f99baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:28:26.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:26.942+0000 7f99baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:28:27.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.106+0000 7f99d0fa6640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:28:27.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.106+0000 7f99d0fa6640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:28:27.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.106+0000 7f99d0fa6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.148:6790/0 2024-09-16T04:28:27.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.106+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 --> v1:172.21.15.148:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f99b0005370 con 0x7f99cc09cc30 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f99c40158b0 con 0x7f99cc09cc30 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99baffd640 10 monclient: handle_config config(1 keys) 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99c8d99640 4 set_mon_vals no callback set 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99c8d99640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99baffd640 1 -- 172.21.15.148:0/297746911 <== mon.2 v1:172.21.15.148:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (unknown 1766731237 0 0) 0x7f99c401e460 con 0x7f99cc09cc30 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:28:27.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.110+0000 7f99baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:28:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.112+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 >> v1:172.21.15.171:6812/7327273 conn(0x7f99ac038ec0 legacy=0x7f99ac03b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.112+0000 7f99d0fa6640 10 monclient: shutdown 2024-09-16T04:28:27.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.112+0000 7f99d0fa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:28:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.112+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 >> v1:172.21.15.148:6790/0 conn(0x7f99cc09cc30 legacy=0x7f99cc1d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:28:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.112+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 shutdown_connections 2024-09-16T04:28:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.113+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 >> 172.21.15.148:0/297746911 conn(0x7f99cc0a5640 msgr2=0x7f99cc13b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:28:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.113+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 shutdown_connections 2024-09-16T04:28:27.115 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-16T04:28:27.113+0000 7f99d0fa6640 1 -- 172.21.15.148:0/297746911 wait complete. 2024-09-16T04:28:27.125 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:28:27.125 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:28:27.126 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:28:27.126 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.160 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:28:27.160 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:28:27.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.236 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:28:27.236 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:28:27.236 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.301 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:28:27.301 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:28:27.302 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.340 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:28:27.340 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:28:27.340 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.408 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:28:27.408 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:28:27.408 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.479 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:28:27.480 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:28:27.480 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:28:27.480 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.550 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:28:27.550 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:28:27.551 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:28:27.551 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.561 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:28:27.561 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:28:27.561 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.613 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:28:27.614 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:28:27.614 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:28:27.628 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:28:27.628 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:28:27.628 DEBUG:teuthology.orchestra.run.smithi148:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-16T04:28:27.684 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:28:27.684 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:28:27.837 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:28:27.837 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:28:27.940 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:28:27.940 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:28:28.000 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:28:28.000 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:28:28.150 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:28:28.151 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:28:28.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:28:28.230 DEBUG:teuthology.orchestra.run.smithi171:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:28:28.285 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:28:28.286 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/data/mon.a.tgz 2024-09-16T04:28:28.286 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-09-16T04:28:28.302 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.IkqZlTAaoQ 2024-09-16T04:28:28.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IkqZlTAaoQ 2024-09-16T04:28:28.430 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.IkqZlTAaoQ 2024-09-16T04:28:28.513 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.IkqZlTAaoQ is 334KB 2024-09-16T04:28:28.608 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.IkqZlTAaoQ 2024-09-16T04:28:28.623 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/data/mon.c.tgz 2024-09-16T04:28:28.624 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-09-16T04:28:28.678 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.TO8T4rsmQn 2024-09-16T04:28:28.679 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.TO8T4rsmQn 2024-09-16T04:28:28.792 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.TO8T4rsmQn 2024-09-16T04:28:28.874 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.TO8T4rsmQn is 339KB 2024-09-16T04:28:28.969 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.TO8T4rsmQn 2024-09-16T04:28:28.984 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/data/mon.b.tgz 2024-09-16T04:28:28.984 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-09-16T04:28:29.003 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.MLaT2MfRiN 2024-09-16T04:28:29.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.MLaT2MfRiN 2024-09-16T04:28:29.135 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.MLaT2MfRiN 2024-09-16T04:28:29.220 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.MLaT2MfRiN is 339KB 2024-09-16T04:28:29.316 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.MLaT2MfRiN 2024-09-16T04:28:29.332 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:28:29.332 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:28:29.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:28:29.415 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:28:29.454 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:28:29.455 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:28:29.456 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/remote/smithi148/crash 2024-09-16T04:28:29.457 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:28:29.483 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/remote/smithi171/crash 2024-09-16T04:28:29.483 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:28:29.518 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:28:29.518 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:28:29.526 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:28:29.695 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:28:29.695 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39740.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35106.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42674.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40390.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39740.log.gz 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35106.log.gz 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38526.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42674.log.gz 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40390.log.gz -5 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42489.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38652.log 2024-09-16T04:28:29.696 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42489.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38526.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36998.log 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42489.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35359.log 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36998.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36998.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37394.log 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39050.log 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35359.log.gz 2024-09-16T04:28:29.697 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37351.log 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39050.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37394.log.gz 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35650.log 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39050.log.gz 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35650.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37351.log.gz 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35099.log 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36435.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35650.log.gzgzip 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35099.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-16T04:28:29.698 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41606.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35099.log.gz 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43221.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42593.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37604.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42593.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43221.log.gz 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37750.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37604.log: /var/log/ceph/ceph-client.admin.34923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37834.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37750.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37604.log.gz 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39993.log 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-09-16T04:28:29.699 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37834.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37750.log.gz 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35847.log 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31887.log 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39993.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37834.log.gz 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37792.log 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39993.log.gz 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.31887.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35847.log.gz 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42194.log 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37792.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31887.log.gz 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37792.log.gz 2024-09-16T04:28:29.700 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42194.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35107.log 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37267.log 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37267.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35107.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37082.log 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39344.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37267.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42405.log 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37082.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-09-16T04:28:29.701 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40999.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37082.log.gz 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42405.log.gz 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40999.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41293.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38820.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40999.log.gz 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39866.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-09-16T04:28:29.702 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40096.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41293.log.gz 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40138.log: gzip -5 --verbose 89.1% -- -- replaced with /var/log/ceph/ceph-client.admin.39866.log.gz /var/log/ceph/ceph-client.admin.39218.log 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37225.log 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35356.log 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42885.log 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37225.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39218.log.gz 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37225.log.gz 2024-09-16T04:28:29.703 INFO:teuthology.orchestra.run.smithi148.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35356.log.gz 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40432.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41858.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42885.log.gz 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41521.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38778.log: /var/log/ceph/ceph-client.admin.43095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43557.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41521.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz --verbose 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.38568.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41900.log 2024-09-16T04:28:29.704 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41521.log.gz 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38568.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43557.log.gz 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42363.log 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41900.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38568.log.gz 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41690.log 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41900.log.gz 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42363.log: /var/log/ceph/ceph-client.admin.41690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2024-09-16T04:28:29.705 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41690.log.gz 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36620.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37520.log 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43011.log 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37520.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41209.log 2024-09-16T04:28:29.706 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43011.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37520.log.gz 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39176.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43011.log.gz 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36141.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41209.log.gz 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39656.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39176.log.gz 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40348.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36141.log.gz/var/log/ceph/ceph-client.admin.41083.log: 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41479.log 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39656.log.gz 2024-09-16T04:28:29.707 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43053.log 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41479.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40348.log.gz 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43053.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41479.log.gz -5 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36662.log 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr: 87.4%/var/log/ceph/ceph-client.admin.41563.log: -- replaced with /var/log/ceph/ceph-client.admin.43053.log.gz 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27516.log 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36351.log 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.tmp-client.admin.27516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35973.log 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27516.log.gz 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36662.log.gz 2024-09-16T04:28:29.708 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35531.log 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35889.log 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35973.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36351.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35805.log 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35973.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35889.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35531.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37646.log 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35889.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35805.log: /var/log/ceph/ceph-client.admin.37646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42321.log 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35805.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 88.6% 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37646.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42321.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-09-16T04:28:29.709 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42758.log 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43305.log 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40957.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42321.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36704.log 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43305.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42758.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42927.log 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36704.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43305.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36057.log 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr: 92.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36704.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42927.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz 2024-09-16T04:28:29.710 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38170.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42927.log.gz 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36309.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36057.log: /var/log/ceph/ceph-client.admin.38170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38212.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36057.log.gz 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36309.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38170.log.gz 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38212.log: 87.2% -- /var/log/ceph/ceph-client.admin.41942.log -- replaced with /var/log/ceph/ceph-client.admin.36309.log.gz 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38610.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38212.log.gz 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35640.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41942.log: /var/log/ceph/ceph-client.admin.38610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32116.log 2024-09-16T04:28:29.711 INFO:teuthology.orchestra.run.smithi148.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41942.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38442.log 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38610.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.32116.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35640.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32116.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41356.log 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37040.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34749.log 2024-09-16T04:28:29.712 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41356.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.34749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39302.log 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41356.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:gzip/var/log/ceph/ceph-client.admin.38315.log: -5 --verbose -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34749.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/ceph-client.admin.35357.log 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39302.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37144.log 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35357.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39302.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42110.log 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35357.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41167.log 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42110.log: gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.37144.log.gz 2024-09-16T04:28:29.713 INFO:teuthology.orchestra.run.smithi148.stderr: /var/log/ceph/ceph-client.admin.34965.log 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35358.log 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41167.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42110.log.gz 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.34965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40516.log 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41167.log.gz 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35358.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34965.log.gz 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36015.log 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35358.log.gz 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36578.log 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36015.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40516.log.gz 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38926.log 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36578.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36015.log.gz 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:28:29.714 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36183.log 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38926.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36578.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39951.log 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38926.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38736.log 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39951.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37960.log 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36183.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38736.log: gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.36267.log -- replaced with /var/log/ceph/ceph-client.admin.39951.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38736.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr: --verbose/var/log/ceph/ceph-client.admin.37960.log: -- /var/log/ceph/ceph-client.admin.39092.log 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36267.log: gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37960.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr: -- /var/log/ceph/ceph-client.admin.37309.log 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36267.log.gz 2024-09-16T04:28:29.715 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43640.log 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37309.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39092.log.gz 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43347.log 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43640.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41816.log 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43640.log.gz 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38862.log 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43347.log.gz 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38862.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41816.log.gz 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37562.log 2024-09-16T04:28:29.716 INFO:teuthology.orchestra.run.smithi148.stderr:gzip 88.4% -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.34646.log: -- replaced with /var/log/ceph/ceph-client.admin.38862.log.gz 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35592.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37562.log.gz 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.35592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36099.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35592.log.gz 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40642.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log 2024-09-16T04:28:29.717 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40707.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38400.log 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36099.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41041.log: /var/log/ceph/ceph-client.admin.38400.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38400.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36746.log 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42236.log 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36746.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42236.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42236.log.gz 2024-09-16T04:28:29.718 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35578.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39260.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35578.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32191.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35578.log.gz 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42279.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.32191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42279.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32191.log.gz 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42279.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40915.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38484.log 2024-09-16T04:28:29.719 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38694.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34881.log 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.38002.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38484.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42531.log 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.34881.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34881.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39007.log 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42716.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42531.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39134.log 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gz 2024-09-16T04:28:29.720 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39007.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39007.log.gz 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39614.log 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42026.log 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39614.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39134.log.gz 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39614.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41984.log 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42026.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42026.log.gz 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40264.log 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41984.log.gz 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42447.log 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40180.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40264.log.gz 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr:gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-09-16T04:28:29.721 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35097.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42447.log: -- /var/log/ceph/ceph-client.admin.42843.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35097.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42447.log.gz 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42068.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35097.log.gz 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.42843.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42843.log.gz 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35104.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42068.log: -- /var/log/ceph/ceph-client.admin.43431.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43389.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42068.log.gz 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35104.log.gz 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35355.log 2024-09-16T04:28:29.722 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40474.log 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43389.log.gz 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35355.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40474.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35355.log.gz 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40474.log.gz 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35354.log 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40306.log: -- /var/log/ceph/ceph-client.admin.42969.log 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36496.log 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35354.log.gz/var/log/ceph/ceph-client.admin.42969.log: 2024-09-16T04:28:29.723 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36788.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36496.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42969.log.gz 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36496.log.gz 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35763.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40600.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36788.log.gz 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37436.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35763.log.gz 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.41437.log: /var/log/ceph/ceph-client.admin.40600.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40600.log.gz 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39572.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39530.log 2024-09-16T04:28:29.724 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43263.log 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37436.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43515.log 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39530.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39572.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40873.log 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39530.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43515.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43263.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36225.log 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43515.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34805.log 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.40873.log: /var/log/ceph/ceph-client.admin.36225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.34805.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40873.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36225.log.gz 2024-09-16T04:28:29.725 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34805.log.gz 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37918.log 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35511.log/var/log/ceph/ceph-client.admin.42152.log: 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gz 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3%/var/log/ceph/ceph-client.admin.35511.log: -- replaced with /var/log/ceph/ceph-client.admin.37918.log.gz 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34830.log 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35511.log.gz 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36956.log: /var/log/ceph/ceph-client.admin.37876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37478.log 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.34830.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-09-16T04:28:29.726 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36914.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34830.log.gz 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43473.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37478.log: /var/log/ceph/ceph-client.admin.36914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.43473.log: 87.5% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37478.log.gz 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36914.log.gz 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.43473.log.gz 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35105.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39782.log: -- /var/log/ceph/ceph-client.admin.35931.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39824.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35105.log.gz 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.35931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36830.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.39824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36393.log 2024-09-16T04:28:29.727 INFO:teuthology.orchestra.run.smithi148.stderr: 87.7%/var/log/ceph/ceph-client.admin.36830.log: -- replaced with /var/log/ceph/ceph-client.admin.35931.log.gz 2024-09-16T04:28:29.728 INFO:teuthology.orchestra.run.smithi148.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39824.log.gz 2024-09-16T04:28:29.728 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39386.log 2024-09-16T04:28:29.728 INFO:teuthology.orchestra.run.smithi148.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36830.log.gz 2024-09-16T04:28:29.728 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.36393.log: /var/log/ceph/ceph-client.admin.39386.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36393.log.gz 2024-09-16T04:28:29.728 INFO:teuthology.orchestra.run.smithi148.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39386.log.gz 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42172.log 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35451.log 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bar.42172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41341.log 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.bar.42172.log.gz 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40404.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35451.log.gz 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35427.log 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42457.log 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-09-16T04:28:29.800 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35427.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41341.log.gz 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39506.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34865.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35427.log.gz 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40572.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42457.log.gz 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38324.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39506.log.gz 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41620.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35268.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40572.log.gz 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37982.log 2024-09-16T04:28:29.801 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38324.log.gz 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35268.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41620.log.gz 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38588.log 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37982.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35268.log.gz 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36000.log 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37982.log.gz 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38804.log 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38588.log.gz 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40308.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.38372.log 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36000.log.gz 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41692.log 2024-09-16T04:28:29.802 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40981.log 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38804.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41692.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38372.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41692.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38754.log 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40981.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40524.log 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-09-16T04:28:29.803 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36025.log 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38754.log.gz 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40524.log.gz 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36025.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz /var/log/ceph/ceph-client.admin.35926.log 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42086.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36025.log.gzgzip 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31640.log 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40164.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35926.log.gz 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38200.log 2024-09-16T04:28:29.804 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.31640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41925.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.41973.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31640.log.gz 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40068.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38200.log.gz 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.foo.41973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35824.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41925.log.gz 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40068.log: 88.2% -- replaced with /var/log/ceph/ceph-client.foo.41973.log.gz 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42238.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35038.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bar.42238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41668.log 2024-09-16T04:28:29.805 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35824.log.gz 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37476.log 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.bar.42238.log.gz 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bazar.41874.log 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35038.log.gz 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37476.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41668.log.gz 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37330.log 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37476.log.gz 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bazar.41874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38348.log 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39005.log 2024-09-16T04:28:29.806 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37330.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37642.log 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38348.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42124.log 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39005.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.bazar.41874.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37642.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37642.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36221.log 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42124.log.gz 2024-09-16T04:28:29.807 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42019.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36221.log.gz 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40020.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.foo.42019.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41053.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36097.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.foo.42019.log.gz 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38396.log 2024-09-16T04:28:29.808 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40020.log.gz 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36097.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.31476.log -- replaced with /var/log/ceph/ceph-client.admin.41053.log.gz 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42110.log 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36097.log.gz 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.31476.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38396.log.gz 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36559.log 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.foo.42110.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.31476.log.gz /var/log/ceph/ceph-client.admin.40789.log 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36949.log 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36559.log.gz 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35560.log 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.foo.42110.log.gz 2024-09-16T04:28:29.809 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36949.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41293.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36949.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34685.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35560.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41293.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38054.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34685.log.gz 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31516.log 2024-09-16T04:28:29.810 INFO:teuthology.orchestra.run.smithi171.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38054.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz /var/log/ceph/ceph-client.admin.40476.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.31516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36369.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38054.log.gz 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35849.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31516.log.gz 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40476.log.gz 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37117.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38252.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36369.log.gz 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35849.log.gz 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38252.log: gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.39755.log 2024-09-16T04:28:29.811 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37117.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37043.log 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38252.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35068.log 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41716.log 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39755.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37043.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35068.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39707.log 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41716.log.gz 2024-09-16T04:28:29.812 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3%/var/log/ceph/ceph-client.admin.40428.log: -- replaced with /var/log/ceph/ceph-client.admin.39707.log.gz 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37522.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40188.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42406.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40188.log: gzip -5 --verbose 87.1% -- -- replaced with /var/log/ceph/ceph-client.admin.37522.log.gz /var/log/ceph/ceph-client.admin.35608.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37189.log 2024-09-16T04:28:29.813 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40188.log.gz 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35608.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.42406.log.gz -5 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36535.log 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40716.log 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41644.log 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37189.log.gz 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40716.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36535.log.gzgzip 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36825.log 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41644.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40716.log.gz 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36825.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.35202.log 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41644.log.gz 2024-09-16T04:28:29.814 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36825.log.gz/var/log/ceph/ceph-osd.4.log: 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42260.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35202.log: /var/log/ceph/ceph-client.bar.42260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39431.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39431.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39032.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.bar.42260.log.gz 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39431.log.gz 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39875.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39032.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35289.log 2024-09-16T04:28:29.815 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39032.log.gz 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37836.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39875.log.gz 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35656.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37836.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35289.log.gz 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36049.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37836.log.gz 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.41949.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35656.log: /var/log/ceph/ceph-client.admin.36049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36073.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35656.log.gz 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36049.log.gz 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.foo.41949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37404.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39176.log 2024-09-16T04:28:29.816 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36073.log: 88.6% -- replaced with /var/log/ceph/ceph-client.foo.41949.log.gz 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35704.log 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36073.log.gz 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37404.log.gz 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39899.log 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39176.log: /var/log/ceph/ceph-client.admin.35704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39080.log 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39176.log.gz 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35704.log.gz 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39353.log 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39080.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39899.log.gz 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42216.log 2024-09-16T04:28:29.817 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39080.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39353.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39803.log 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bar.42216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36799.log 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39353.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.bar.42216.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39851.log 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36799.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39803.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38662.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36799.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr: 91.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39851.log: -- /var/log/ceph/ceph-client.admin.39996.log 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-09-16T04:28:29.818 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36999.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39851.log.gz 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39996.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34910.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35379.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39996.log.gz 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36999.log: /var/log/ceph/ceph-client.admin.34910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42311.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36999.log.gz 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35379.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34910.log.gz 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bar.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42481.log 2024-09-16T04:28:29.819 INFO:teuthology.orchestra.run.smithi171.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.bar.42311.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35403.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41173.log 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42481.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41173.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42481.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39152.log 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31556.log 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41173.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39152.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36703.log 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.31556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36197.log 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39152.log.gz 2024-09-16T04:28:29.820 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35752.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31556.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36703.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35752.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36197.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40957.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35752.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38102.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37860.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38102.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35124.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37860.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38102.log.gz 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36925.log 2024-09-16T04:28:29.821 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37067.log 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37860.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35124.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37021.log 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36925.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36851.log 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37067.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37812.log 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37021.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39200.log 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37812.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36851.log.gz 2024-09-16T04:28:29.822 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37428.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39200.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37812.log.gz 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35632.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37594.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39200.log.gz 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37428.log.gz 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37594.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35632.log.gz 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36419.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34773.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37594.log.gz 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36419.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz -5 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38442.log 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40909.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36419.log.gz 2024-09-16T04:28:29.823 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34636.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40644.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38442.log.gz -5 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36321.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34636.log.gz 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34843.log 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36321.log.gz 2024-09-16T04:28:29.824 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37690.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37213.log 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34843.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41533.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37213.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39302.log 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37886.log 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37886.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39302.log.gz 2024-09-16T04:28:29.825 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34561.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37886.log.gz 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39275.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36973.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36775.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34561.log.gz 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39248.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39275.log.gz 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36775.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36973.log.gz 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35728.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39248.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36775.log.gz 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41901.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34889.log 2024-09-16T04:28:29.826 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39248.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41901.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35728.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34889.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41901.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35680.log 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34889.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39947.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36122.log 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35680.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34962.log 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35680.log.gz 2024-09-16T04:28:29.827 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40548.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36122.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34962.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38226.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.41997.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34962.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38226.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41740.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38226.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.foo.41997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37141.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40933.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.foo.41997.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37141.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41740.log.gz 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37714.log 2024-09-16T04:28:29.828 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40933.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37141.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38030.log 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36465.log 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40933.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38030.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37714.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35950.log 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36465.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38030.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42041.log 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34611.log 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36465.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.foo.42041.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35950.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42382.log 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34611.log: 88.3% -- replaced with /var/log/ceph/ceph-client.foo.42041.log.gz 2024-09-16T04:28:29.829 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40765.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34611.log.gz 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39635.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42382.log.gz 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31436.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40765.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.31436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36655.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40765.log.gz 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41764.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31436.log.gz 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38078.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36655.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36245.log 2024-09-16T04:28:29.830 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36147.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36655.log.gz 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36245.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38078.log.gz 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36147.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.36245.log.gz /var/log/ceph/ceph-client.admin.40813.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38906.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36147.log.gz 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40813.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38780.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38906.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40813.log.gz 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-09-16T04:28:29.831 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38906.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41850.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38780.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40741.log: /var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39104.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39482.log 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38468.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35331.log 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37764.log 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-09-16T04:28:29.832 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37570.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39482.log.gz 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35331.log.gz 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37764.log.gz 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36631.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37570.log.gz 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36631.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36631.log.gz 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:28:29.833 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42287.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31596.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39377.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42287.log.gz 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.31596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35900.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31596.log.gz 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36441.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35900.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39377.log.gz/var/log/ceph/ceph-client.admin.36441.log: 2024-09-16T04:28:29.834 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34587.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35900.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36441.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39128.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42148.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34587.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37546.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34587.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bar.42148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39128.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.bar.42148.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37546.log.gz 2024-09-16T04:28:29.835 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39683.log 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40356.log 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40092.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35012.log 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40356.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39683.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40140.log 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40356.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35012.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40140.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35012.log.gz 2024-09-16T04:28:29.836 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39557.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40140.log.gz 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38518.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38856.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39557.log.gz 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37932.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 86.6% --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38856.log: -- replaced with /var/log/ceph/ceph-client.admin.38518.log.gz 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38154.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38856.log.gz 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-09-16T04:28:29.837 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37932.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39827.log: /var/log/ceph/ceph-client.admin.38154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7%/var/log/ceph/ceph-client.admin.40668.log: -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38154.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36583.log 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39923.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37167.log 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35475.log 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37167.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36583.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35776.log 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35475.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37167.log.gz 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-09-16T04:28:29.838 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35875.log 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35475.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35776.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36269.log 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35875.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36269.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35875.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37355.log 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36269.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37355.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38178.log 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41788.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37355.log.gz 2024-09-16T04:28:29.839 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35355.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40212.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37958.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38178.log.gz 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35355.log.gz 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40885.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38930.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40212.log.gz 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37958.log.gz 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38930.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40885.log.gz 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35246.log 2024-09-16T04:28:29.840 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38930.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36487.log 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39404.log 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36487.log: gzip -5 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35246.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39972.log 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39404.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36487.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37281.log 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39404.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37281.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39972.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr: -- /var/log/ceph/ceph-client.admin.34797.log 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39056.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37281.log.gz 2024-09-16T04:28:29.841 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39458.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41437.log: gzip -5 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34797.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37910.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39458.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38276.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37910.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39458.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38276.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37910.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42432.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41077.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38276.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37788.log 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-09-16T04:28:29.842 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.42432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39611.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37788.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42432.log.gz 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr: -- /var/log/ceph/ceph-client.admin.37305.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39611.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37788.log.gz 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37093.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35976.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39611.log.gz 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35536.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37305.log.gz 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39530.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37093.log.gz 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34936.log 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35976.log.gz 2024-09-16T04:28:29.843 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35536.log: /var/log/ceph/ceph-client.admin.39530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38006.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39731.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39530.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35536.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38006.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34936.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36607.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38006.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36751.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39731.log: /var/log/ceph/ceph-client.admin.36607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41365.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36751.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36607.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40260.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36751.log.gz 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36511.log 2024-09-16T04:28:29.844 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41365.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37380.log 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36511.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40260.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36511.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39584.log 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35584.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37380.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38613.log 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.39584.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36295.log 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39584.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38613.log.gz 2024-09-16T04:28:29.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39329.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41413.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35800.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36295.log.gz 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz/var/log/ceph/ceph-client.admin.39329.log: 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35800.log: /var/log/ceph/ceph-client.admin.34707.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39329.log.gz 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41245.log: gzip -5 87.4% --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35800.log.gz 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.34707.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36679.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37618.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34707.log.gz 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-09-16T04:28:29.846 INFO:teuthology.orchestra.run.smithi171.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36679.log: /var/log/ceph/ceph-client.admin.37618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35310.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37618.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36679.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36877.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41317.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35224.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35310.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41389.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2024-09-16T04:28:29.847 INFO:teuthology.orchestra.run.smithi171.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35224.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37257.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38564.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35224.log.gz 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38300.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37257.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.42063.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37498.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38300.log: /var/log/ceph/ceph-client.foo.42063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.42194.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37498.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38300.log.gz 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37498.log.gz 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39779.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36903.log 2024-09-16T04:28:29.848 INFO:teuthology.orchestra.run.smithi171.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.foo.42063.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.bar.42194.log: /var/log/ceph/ceph-client.admin.39779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36393.log 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.bar.42194.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36903.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39779.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36172.log 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36903.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.36393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35180.log 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr: 87.7%/var/log/ceph/ceph-client.admin.36172.log: -- replaced with /var/log/ceph/ceph-client.admin.36393.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35094.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36172.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.35180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.38420.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35180.log.gz 2024-09-16T04:28:29.849 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37235.log 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37666.log 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.41005.log: /var/log/ceph/ceph-client.admin.37235.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37235.log.gz 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36727.log 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/ceph-client.admin.37666.log: /var/log/ceph/ceph-client.admin.36727.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37666.log.gz 2024-09-16T04:28:29.850 INFO:teuthology.orchestra.run.smithi171.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36727.log.gz 2024-09-16T04:28:29.897 INFO:teuthology.orchestra.run.smithi171.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:28:30.045 INFO:teuthology.orchestra.run.smithi148.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:28:30.256 INFO:teuthology.orchestra.run.smithi148.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:28:30.398 INFO:teuthology.orchestra.run.smithi171.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:28:30.565 INFO:teuthology.orchestra.run.smithi171.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:28:30.578 INFO:teuthology.orchestra.run.smithi148.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:28:30.610 INFO:teuthology.orchestra.run.smithi148.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:28:30.716 INFO:teuthology.orchestra.run.smithi171.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:28:30.835 INFO:teuthology.orchestra.run.smithi171.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:28:30.837 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-09-16T04:28:30.837 INFO:teuthology.orchestra.run.smithi171.stderr:real 0m1.263s 2024-09-16T04:28:30.838 INFO:teuthology.orchestra.run.smithi171.stderr:user 0m4.140s 2024-09-16T04:28:30.838 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m0.382s 2024-09-16T04:28:30.954 INFO:teuthology.orchestra.run.smithi148.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:28:30.956 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-16T04:28:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:real 0m1.417s 2024-09-16T04:28:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:user 0m4.540s 2024-09-16T04:28:30.956 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m0.308s 2024-09-16T04:28:30.957 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:28:30.957 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/remote/smithi148/log 2024-09-16T04:28:30.958 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:28:31.873 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/remote/smithi171/log 2024-09-16T04:28:31.873 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:28:35.224 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:28:35.235 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:28:35.235 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:28:35.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:28:35.320 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:28:35.330 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:28:35.330 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:28:35.333 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:28:35.348 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:28:35.350 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-09-16T04:28:35.353 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:28:35.353 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-09-16T04:28:35.353 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -224us[ -211us] +/- 57ms 2024-09-16T04:28:35.353 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 +1148us[+1148us] +/- 62ms 2024-09-16T04:28:35.353 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -1100us[-1100us] +/- 71ms 2024-09-16T04:28:35.353 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:28:35.354 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T04:28:35.355 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-09-16T04:28:35.355 INFO:teuthology.orchestra.run.smithi171.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 -223us[ -225us] +/- 57ms 2024-09-16T04:28:35.355 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 +1158us[+1158us] +/- 62ms 2024-09-16T04:28:35.355 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -1079us[-1079us] +/- 71ms 2024-09-16T04:28:35.355 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T04:28:35.356 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:28:35.364 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:28:35.365 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:28:35.394 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:28:35.465 DEBUG:teuthology.orchestra.run.smithi171:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T04:28:35.533 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:28:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:28:35.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T04:28:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:28:35.592 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:28:35.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:28:35.634 INFO:teuthology.task.internal:Duration was 1730.613481 seconds 2024-09-16T04:28:35.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:28:35.667 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:28:35.668 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:28:35.670 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:28:35.708 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:28:35.711 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T04:28:36.007 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:28:36.007 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-09-16T04:28:36.007 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:28:36.053 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-09-16T04:28:36.053 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:28:36.079 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:28:36.079 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:28:36.096 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:28:36.147 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:28:36.148 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:28:36.163 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:28:36.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:28:36.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:28:36.301 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:28:36.327 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:28:36.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:28:36.365 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:28:36.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:28:36.390 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-09-16T04:28:36.417 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-09-16T04:28:36.428 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:28:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:28:36.472 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:28:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:28:36.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:28:36.507 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:28:36.508 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/remote/smithi148 2024-09-16T04:28:36.508 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:28:36.547 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905963/remote/smithi171 2024-09-16T04:28:36.547 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:28:36.581 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:28:36.581 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:28:36.583 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:28:36.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:28:36.644 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:28:36.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:28:36.684 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:28:36.685 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:28:36.688 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:28:36.699 INFO:teuthology.orchestra.run.smithi148.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:28 /home/ubuntu/cephtest 2024-09-16T04:28:36.702 INFO:teuthology.orchestra.run.smithi171.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:28 /home/ubuntu/cephtest 2024-09-16T04:28:36.704 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:28:36.716 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:28:36.785 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_mon_workunits} duration: 1730.6134810447693 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:28:36.786 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:28:36.873 INFO:teuthology.run:pass